2024-04-25T16:14:58.335 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T16:14:58.335 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/orig.config.yaml 2024-04-25T16:14:58.387 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T16:14:58.486 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} email: pdonnell@redhat.com first_in_suite: false job_id: '7673254' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 kclient: client.0: mount_subvol_num: 1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbpKmuJ8XytYJkBUjL8NiCFnnUjKlfTN87sG+UVhfJQCodHf6YHJ8hBC6Us584YzPhINaGSRUeZ8h3mB+o038Y= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFLgQk0metQO+1/T7DgZU4TXXsmE+JGSAbvvJUAGX4M8t0fK7Tr5YY7HBPydPNUMbozvUCQhOEhbzEnlvBq+rg4= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVxOtrt9fVPNBp+Qd2NuaBQF7bnlNYrx4MqFBf+vottMr/B4o54q4upy4rRMciDt7RiQtEr6tXHjcsFuBpQFfM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - kernel_untar_build.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T16:14:58.487 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T16:14:58.487 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T16:14:58.487 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T16:14:58.489 INFO:teuthology.task.internal:Checking packages... 2024-04-25T16:14:58.512 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T16:14:58.512 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T16:14:58.512 INFO:teuthology.packaging:ref: None 2024-04-25T16:14:58.512 INFO:teuthology.packaging:tag: None 2024-04-25T16:14:58.512 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T16:14:58.512 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:14:58.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T16:14:58.651 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T16:14:58.652 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T16:14:58.658 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T16:14:58.658 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T16:14:58.692 INFO:teuthology.task.internal:Saving configuration 2024-04-25T16:14:58.712 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T16:14:58.725 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T16:14:58.750 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:09:53.127783', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbpKmuJ8XytYJkBUjL8NiCFnnUjKlfTN87sG+UVhfJQCodHf6YHJ8hBC6Us584YzPhINaGSRUeZ8h3mB+o038Y='} 2024-04-25T16:14:58.776 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:09:53.129061', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFLgQk0metQO+1/T7DgZU4TXXsmE+JGSAbvvJUAGX4M8t0fK7Tr5YY7HBPydPNUMbozvUCQhOEhbzEnlvBq+rg4='} 2024-04-25T16:14:58.796 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:09:53.130185', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVxOtrt9fVPNBp+Qd2NuaBQF7bnlNYrx4MqFBf+vottMr/B4o54q4upy4rRMciDt7RiQtEr6tXHjcsFuBpQFfM='} 2024-04-25T16:14:58.796 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T16:14:58.802 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T16:14:58.802 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T16:14:58.802 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T16:14:58.803 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T16:14:58.921 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7c82606f70>, signals=[15]) 2024-04-25T16:14:58.922 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T16:14:58.928 INFO:teuthology.task.internal:Opening connections... 2024-04-25T16:14:58.928 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T16:14:58.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:14:59.006 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T16:14:59.007 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:14:59.079 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T16:14:59.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:14:59.154 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T16:14:59.161 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-04-25T16:14:59.179 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-04-25T16:14:59.180 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-04-25T16:14:59.236 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T16:14:59.237 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T16:14:59.238 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-04-25T16:14:59.513 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-04-25T16:14:59.532 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-04-25T16:14:59.532 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-04-25T16:14:59.589 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T16:14:59.590 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T16:14:59.590 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T16:14:59.590 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-04-25T16:14:59.611 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-04-25T16:14:59.629 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-04-25T16:14:59.629 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-04-25T16:14:59.685 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-04-25T16:14:59.686 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T16:14:59.687 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T16:14:59.687 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T16:14:59.687 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-04-25T16:14:59.711 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T16:14:59.722 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T16:14:59.763 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T16:14:59.763 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-04-25T16:14:59.766 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-04-25T16:14:59.769 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-04-25T16:14:59.785 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T16:14:59.791 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T16:14:59.791 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-04-25T16:14:59.823 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-04-25T16:14:59.826 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-04-25T16:14:59.862 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T16:14:59.939 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T16:14:59.953 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T16:14:59.953 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T16:14:59.953 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T16:14:59.954 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T16:14:59.954 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T16:14:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:14:59.970 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-04-25T16:14:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:14:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:14:59.972 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-25T16:14:59.973 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-04-25T16:15:00.027 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-437.el9.x86_64 2024-04-25T16:15:00.028 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-437.el9.x86_64 2024-04-25T16:15:00.028 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-04-25T16:15:00.031 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-437.el9.x86_64 2024-04-25T16:15:00.031 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-437.el9.x86_64 2024-04-25T16:15:00.031 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-437.el9.x86_64 2024-04-25T16:15:00.031 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-04-25T16:15:00.032 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-437.el9.x86_64 2024-04-25T16:15:00.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-04-25T16:15:02.580 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.0 MB 00:00 2024-04-25T16:15:03.075 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-04-25T16:15:03.203 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-04-25T16:15:04.447 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-25T16:15:05.251 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-25T16:15:05.338 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-25T16:15:08.615 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T16:15:09.363 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-25T16:15:09.486 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-25T16:15:10.180 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-04-25T16:15:11.117 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-25T16:15:11.312 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-04-25T16:15:11.615 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-25T16:15:13.571 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-04-25T16:15:16.366 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 4.2 MB/s | 21 MB 00:04 2024-04-25T16:15:16.748 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-04-25T16:15:18.601 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-25T16:15:18.638 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T16:15:18.638 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T16:15:18.729 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-25T16:15:18.730 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-25T16:15:18.730 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-25T16:15:18.824 DEBUG:teuthology.orchestra.run.smithi203:> echo no | sudo yum reinstall kernel || true 2024-04-25T16:15:19.327 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 04:15:16 PM UTC. 2024-04-25T16:15:19.446 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-25T16:15:19.459 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-25T16:15:19.459 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-04-25T16:15:19.459 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-25T16:15:19.459 INFO:teuthology.orchestra.run.smithi203.stdout:Reinstalling: 2024-04-25T16:15:19.459 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T16:15:19.459 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:15:19.459 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-25T16:15:19.459 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-25T16:15:19.460 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:15:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 255 k 2024-04-25T16:15:19.460 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 0 2024-04-25T16:15:19.460 INFO:teuthology.orchestra.run.smithi203.stderr:Operation aborted. 2024-04-25T16:15:19.516 INFO:teuthology.orchestra.run.smithi203.stdout:Is this ok [y/N]: 2024-04-25T16:15:19.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum reinstall -y kernel || true 2024-04-25T16:15:20.012 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 04:15:16 PM UTC. 2024-04-25T16:15:20.129 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout:Reinstalling: 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 255 k 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 0 2024-04-25T16:15:20.130 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-25T16:15:20.409 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.6 MB/s | 255 kB 00:00 2024-04-25T16:15:20.409 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:15:20.409 INFO:teuthology.orchestra.run.smithi203.stdout:Total 911 kB/s | 255 kB 00:00 2024-04-25T16:15:20.431 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-25T16:15:20.549 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T16:15:20.549 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T16:15:20.591 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-25T16:15:20.591 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-25T16:15:20.596 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-25T16:15:20.596 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-25T16:15:20.640 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-25T16:15:20.641 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-04-25T16:15:20.641 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-04-25T16:15:20.721 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-25T16:15:20.771 DEBUG:teuthology.orchestra.run.smithi173:> echo no | sudo yum reinstall kernel || true 2024-04-25T16:15:20.801 INFO:teuthology.orchestra.run.smithi203.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T16:15:20.802 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:21.311 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 04:15:18 PM UTC. 2024-04-25T16:15:21.421 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-25T16:15:21.460 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-25T16:15:21.467 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T16:15:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-04-25T16:15:21.467 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T16:15:21.467 INFO:teuthology.orchestra.run.smithi173.stdout:Reinstalling: 2024-04-25T16:15:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T16:15:21.467 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:15:21.467 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-04-25T16:15:21.467 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T16:15:21.468 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:15:21.468 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 255 k 2024-04-25T16:15:21.468 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 0 2024-04-25T16:15:21.468 INFO:teuthology.orchestra.run.smithi173.stderr:Operation aborted. 2024-04-25T16:15:21.480 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:21.480 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T16:15:21.532 INFO:teuthology.orchestra.run.smithi173.stdout:Is this ok [y/N]: 2024-04-25T16:15:21.532 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum reinstall -y kernel || true 2024-04-25T16:15:22.020 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 04:15:18 PM UTC. 2024-04-25T16:15:22.136 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-25T16:15:22.136 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T16:15:22.136 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-04-25T16:15:22.136 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout:Reinstalling: 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 255 k 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 0 2024-04-25T16:15:22.137 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-04-25T16:15:22.208 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:22.208 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:15:22.208 INFO:teuthology.orchestra.run.smithi203.stdout:Reinstalled: 2024-04-25T16:15:22.208 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T16:15:22.209 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:15:22.209 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-25T16:15:22.355 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.7 MB/s | 255 kB 00:00 2024-04-25T16:15:22.355 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:15:22.355 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.1 MB/s | 255 kB 00:00 2024-04-25T16:15:22.374 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-04-25T16:15:22.414 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T16:15:22.461 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T16:15:22.461 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T16:15:22.461 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T16:15:22.461 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-04-25T16:15:22.462 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T16:15:22.493 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-04-25T16:15:22.522 DEBUG:teuthology.parallel:result is None 2024-04-25T16:15:22.571 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-04-25T16:15:22.571 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-04-25T16:15:22.576 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-04-25T16:15:22.576 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-04-25T16:15:22.696 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-04-25T16:15:22.770 INFO:teuthology.orchestra.run.smithi173.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T16:15:22.771 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:23.310 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T16:15:23.311 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T16:15:23.403 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-25T16:15:23.404 INFO:teuthology.orchestra.run.smithi098.stdout:Nothing to do. 2024-04-25T16:15:23.404 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-25T16:15:23.454 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:23.454 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T16:15:23.503 DEBUG:teuthology.orchestra.run.smithi098:> echo no | sudo yum reinstall kernel || true 2024-04-25T16:15:23.999 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:02 ago on Thu 25 Apr 2024 04:15:21 PM UTC. 2024-04-25T16:15:24.117 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-25T16:15:24.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T16:15:24.134 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-25T16:15:24.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T16:15:24.134 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalling: 2024-04-25T16:15:24.134 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T16:15:24.134 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:15:24.134 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-25T16:15:24.134 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T16:15:24.135 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:15:24.135 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 255 k 2024-04-25T16:15:24.135 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 0 2024-04-25T16:15:24.135 INFO:teuthology.orchestra.run.smithi098.stderr:Operation aborted. 2024-04-25T16:15:24.137 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:24.137 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:15:24.137 INFO:teuthology.orchestra.run.smithi173.stdout:Reinstalled: 2024-04-25T16:15:24.137 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T16:15:24.137 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:15:24.137 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-04-25T16:15:24.189 INFO:teuthology.orchestra.run.smithi098.stdout:Is this ok [y/N]: 2024-04-25T16:15:24.190 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum reinstall -y kernel || true 2024-04-25T16:15:24.316 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T16:15:24.363 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T16:15:24.364 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T16:15:24.364 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T16:15:24.364 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-25T16:15:24.364 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T16:15:24.400 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-04-25T16:15:24.415 DEBUG:teuthology.parallel:result is None 2024-04-25T16:15:24.682 INFO:teuthology.orchestra.run.smithi098.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 04:15:21 PM UTC. 2024-04-25T16:15:24.798 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-25T16:15:24.802 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T16:15:24.802 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-04-25T16:15:24.802 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T16:15:24.802 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalling: 2024-04-25T16:15:24.802 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T16:15:24.802 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:15:24.802 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-25T16:15:24.802 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-04-25T16:15:24.803 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:15:24.803 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 255 k 2024-04-25T16:15:24.803 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 0 2024-04-25T16:15:24.803 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-25T16:15:25.548 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-437.el9.x86_64.rpm 538 kB/s | 255 kB 00:00 2024-04-25T16:15:25.548 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:15:25.548 INFO:teuthology.orchestra.run.smithi098.stdout:Total 340 kB/s | 255 kB 00:00 2024-04-25T16:15:25.569 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-25T16:15:25.752 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-25T16:15:25.752 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-25T16:15:25.757 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-25T16:15:25.757 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-25T16:15:25.889 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-25T16:15:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T16:15:25.974 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:26.701 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:26.701 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T16:15:27.388 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T16:15:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:15:27.389 INFO:teuthology.orchestra.run.smithi098.stdout:Reinstalled: 2024-04-25T16:15:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T16:15:27.389 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:15:27.389 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-25T16:15:27.568 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T16:15:27.616 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T16:15:27.616 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T16:15:27.616 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T16:15:27.616 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T16:15:27.616 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T16:15:27.650 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-04-25T16:15:27.675 DEBUG:teuthology.parallel:result is None 2024-04-25T16:15:27.676 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T16:15:27.685 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T16:15:27.685 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T16:15:27.718 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T16:15:27.720 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T16:15:27.738 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T16:15:27.746 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T16:15:27.782 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T16:15:27.782 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T16:15:27.784 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T16:15:27.786 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T16:15:27.833 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T16:15:27.850 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T16:15:27.851 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T16:15:27.856 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T16:15:27.867 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T16:15:27.897 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:15:27.905 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:15:27.908 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:15:27.914 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:15:27.914 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:15:27.925 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T16:15:27.926 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T16:15:27.945 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T16:15:27.946 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T16:15:27.950 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T16:15:27.958 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T16:15:27.998 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T16:15:28.025 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T16:15:28.026 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T16:15:28.035 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T16:15:28.039 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T16:15:28.067 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:15:28.135 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:15:28.199 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:15:28.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:15:28.323 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:15:28.323 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T16:15:28.387 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:15:28.433 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:15:28.501 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:15:28.543 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:15:28.585 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:15:28.585 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T16:15:28.651 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:15:28.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T16:15:28.757 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:15:28.801 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T16:15:28.843 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:15:28.843 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T16:15:28.909 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-04-25T16:15:28.912 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-04-25T16:15:28.914 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-04-25T16:15:28.945 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:15:28.951 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:15:28.987 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:15:29.303 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T16:15:29.314 INFO:teuthology.task.internal:Starting timer... 2024-04-25T16:15:29.314 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T16:15:29.363 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T16:15:29.395 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T16:15:29.395 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-04-25T16:15:29.449 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-04-25T16:15:29.452 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-04-25T16:15:29.509 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-04-25T16:15:29.511 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-04-25T16:15:29.565 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-04-25T16:15:29.567 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T16:15:29.567 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-04-25T16:15:29.586 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-04-25T16:15:29.587 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-04-25T16:15:29.609 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-04-25T16:15:29.609 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-04-25T16:15:29.629 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-04-25T16:15:29.629 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-04-25T16:15:29.630 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:15:29.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:15:29.654 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:15:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:15:29.679 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:15:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:15:29.704 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T16:15:29.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-04-25T16:15:29.727 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-04-25T16:15:29.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-04-25T16:15:29.775 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T16:15:29.786 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T16:15:29.806 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T16:15:29.807 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi173.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T16:19:00.190 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-04-25T16:19:00.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-25T16:19:00.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:19:00.267 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-04-25T16:19:00.341 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-04-25T16:19:00.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T16:19:00.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:19:00.415 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-04-25T16:19:00.448 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-04-25T16:19:00.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T16:19:00.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T16:19:00.524 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-25T16:19:00.606 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-25T16:19:00.607 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T16:19:00.619 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T16:19:00.619 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T16:19:00.619 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:19:00.622 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T16:19:00.622 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:19:00.624 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T16:19:00.624 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:19:00.648 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T16:19:00.650 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T16:19:00.661 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T16:19:00.664 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T16:19:00.676 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T16:19:00.684 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-04-25T16:19:00.687 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-04-25T16:19:00.690 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T16:19:00.695 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-04-25T16:19:00.697 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-04-25T16:19:00.708 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T16:19:00.711 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T16:19:00.714 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-04-25T16:19:00.721 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T16:19:00.724 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T16:19:00.725 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-04-25T16:19:00.738 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T16:19:00.751 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T16:19:00.779 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:19:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:19:00.782 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:19:00.782 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-25T16:19:00.782 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.783 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.783 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.783 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.784 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:19:00.784 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-04-25T16:19:00.784 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.784 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.784 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.785 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.813 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:19:00.816 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:19:00.816 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-25T16:19:00.816 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.817 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.817 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.817 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:19:00.818 INFO:teuthology.run_tasks:Running task install... 2024-04-25T16:19:00.826 DEBUG:teuthology.task.install:project ceph 2024-04-25T16:19:00.826 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T16:19:00.826 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T16:19:00.826 INFO:teuthology.task.install:Using flavor: default 2024-04-25T16:19:00.834 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T16:19:00.834 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T16:19:00.834 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T16:19:00.835 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:00.836 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T16:19:00.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:00.838 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T16:19:00.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:19:01.191 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T16:19:01.191 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T16:19:01.195 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T16:19:01.195 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T16:19:01.197 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T16:19:01.198 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T16:19:01.381 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T16:19:01.382 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:19:01.382 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T16:19:01.386 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T16:19:01.387 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:19:01.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T16:19:01.398 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T16:19:01.398 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:19:01.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T16:19:01.409 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T16:19:01.409 DEBUG:teuthology.orchestra.run.smithi098:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T16:19:01.413 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T16:19:01.413 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T16:19:01.423 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T16:19:01.424 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T16:19:01.475 DEBUG:teuthology.orchestra.run.smithi098:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T16:19:01.480 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T16:19:01.489 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T16:19:01.551 DEBUG:teuthology.orchestra.run.smithi098:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T16:19:01.553 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T16:19:01.562 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T16:19:01.616 INFO:teuthology.orchestra.run.smithi098.stdout:check_obsoletes = 1 2024-04-25T16:19:01.617 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-04-25T16:19:01.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum clean all 2024-04-25T16:19:01.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-04-25T16:19:01.625 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-04-25T16:19:01.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-04-25T16:19:01.844 INFO:teuthology.orchestra.run.smithi098.stdout:48 files removed 2024-04-25T16:19:01.847 INFO:teuthology.orchestra.run.smithi203.stdout:48 files removed 2024-04-25T16:19:01.852 INFO:teuthology.orchestra.run.smithi173.stdout:48 files removed 2024-04-25T16:19:01.870 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T16:19:01.874 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T16:19:01.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T16:19:02.552 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 204 kB/s | 84 kB 00:00 2024-04-25T16:19:02.631 INFO:teuthology.orchestra.run.smithi098.stdout:ceph packages for x86_64 166 kB/s | 84 kB 00:00 2024-04-25T16:19:02.636 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 167 kB/s | 84 kB 00:00 2024-04-25T16:19:02.886 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 40 kB/s | 12 kB 00:00 2024-04-25T16:19:03.093 INFO:teuthology.orchestra.run.smithi098.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-04-25T16:19:03.096 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-04-25T16:19:03.262 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-25T16:19:03.435 INFO:teuthology.orchestra.run.smithi098.stdout:ceph source packages 6.2 kB/s | 1.9 kB 00:00 2024-04-25T16:19:03.513 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-04-25T16:19:04.238 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-25T16:19:04.290 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 8.0 MB 00:00 2024-04-25T16:19:04.332 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-25T16:19:06.164 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-25T16:19:06.174 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-04-25T16:19:06.525 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-25T16:19:10.307 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T16:19:10.442 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.2 MB 00:00 2024-04-25T16:19:10.643 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T16:19:11.813 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-04-25T16:19:11.865 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-04-25T16:19:12.175 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-25T16:19:13.226 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-25T16:19:13.802 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-25T16:19:14.408 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 21 MB 00:02 2024-04-25T16:19:18.265 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-25T16:19:18.919 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-04-25T16:19:19.416 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-25T16:19:20.408 INFO:teuthology.orchestra.run.smithi098.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:19:20.408 INFO:teuthology.orchestra.run.smithi098.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:19:20.419 INFO:teuthology.orchestra.run.smithi098.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T16:19:20.421 INFO:teuthology.orchestra.run.smithi098.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T16:19:20.421 INFO:teuthology.orchestra.run.smithi098.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T16:19:20.422 INFO:teuthology.orchestra.run.smithi098.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T16:19:20.424 INFO:teuthology.orchestra.run.smithi098.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T16:19:20.426 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T16:19:20.427 INFO:teuthology.orchestra.run.smithi098.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T16:19:20.488 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: Package Arch Version Repository Size 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T16:19:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T16:19:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T16:19:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrading: 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T16:19:20.500 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T16:19:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T16:19:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T16:19:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T16:19:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T16:19:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T16:19:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T16:19:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout:Installing weak dependencies: 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:19:20.508 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-04-25T16:19:20.509 INFO:teuthology.orchestra.run.smithi098.stdout:======================================================================================= 2024-04-25T16:19:20.509 INFO:teuthology.orchestra.run.smithi098.stdout:Install 148 Packages 2024-04-25T16:19:20.509 INFO:teuthology.orchestra.run.smithi098.stdout:Upgrade 4 Packages 2024-04-25T16:19:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:19:20.509 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 214 M 2024-04-25T16:19:20.509 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-04-25T16:19:21.046 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:19:21.046 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:19:21.057 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T16:19:21.058 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T16:19:21.059 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T16:19:21.060 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T16:19:21.061 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T16:19:21.064 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T16:19:21.065 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T16:19:21.074 INFO:teuthology.orchestra.run.smithi098.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-25T16:19:21.126 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-25T16:19:21.135 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T16:19:21.136 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T16:19:21.137 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-04-25T16:19:21.138 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T16:19:21.139 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T16:19:21.140 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T16:19:21.141 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T16:19:21.142 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T16:19:21.143 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T16:19:21.144 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T16:19:21.145 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T16:19:21.146 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================= 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout:Install 148 Packages 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 4 Packages 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:19:21.147 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 214 M 2024-04-25T16:19:21.148 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-25T16:19:21.316 INFO:teuthology.orchestra.run.smithi098.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 3.5 MB/s | 863 kB 00:00 2024-04-25T16:19:21.383 INFO:teuthology.orchestra.run.smithi098.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 2.2 MB/s | 147 kB 00:00 2024-04-25T16:19:21.567 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:19:21.568 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T16:19:21.578 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T16:19:21.579 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T16:19:21.580 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T16:19:21.581 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T16:19:21.582 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T16:19:21.584 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T16:19:21.585 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T16:19:21.644 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-04-25T16:19:21.653 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-25T16:19:21.653 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T16:19:21.654 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T16:19:21.655 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T16:19:21.656 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T16:19:21.657 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T16:19:21.658 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T16:19:21.659 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T16:19:21.660 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T16:19:21.661 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T16:19:21.662 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T16:19:21.663 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T16:19:21.664 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================= 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout:Install 148 Packages 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 4 Packages 2024-04-25T16:19:21.665 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:19:21.666 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 214 M 2024-04-25T16:19:21.666 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-04-25T16:19:21.675 INFO:teuthology.orchestra.run.smithi098.stdout:(4/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 6.9 MB/s | 5.4 MB 00:00 2024-04-25T16:19:21.767 INFO:teuthology.orchestra.run.smithi098.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 5.9 MB/s | 2.3 MB 00:00 2024-04-25T16:19:21.876 INFO:teuthology.orchestra.run.smithi098.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 7.5 MB/s | 1.5 MB 00:00 2024-04-25T16:19:22.229 INFO:teuthology.orchestra.run.smithi203.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-04-25T16:19:22.426 INFO:teuthology.orchestra.run.smithi173.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-25T16:19:22.759 INFO:teuthology.orchestra.run.smithi098.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 4.9 MB/s | 4.9 MB 00:00 2024-04-25T16:19:23.043 INFO:teuthology.orchestra.run.smithi098.stdout:(8/152): ceph-common-19.0.0-3291.g994e2f20.el9. 9.1 MB/s | 20 MB 00:02 2024-04-25T16:19:23.070 INFO:teuthology.orchestra.run.smithi203.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 1.0 MB/s | 863 kB 00:00 2024-04-25T16:19:23.110 INFO:teuthology.orchestra.run.smithi098.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 379 kB/s | 25 kB 00:00 2024-04-25T16:19:23.229 INFO:teuthology.orchestra.run.smithi203.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 928 kB/s | 147 kB 00:00 2024-04-25T16:19:23.971 INFO:teuthology.orchestra.run.smithi203.stdout:(4/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 2.5 MB/s | 5.4 MB 00:02 2024-04-25T16:19:24.529 INFO:teuthology.orchestra.run.smithi203.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 2.7 MB/s | 1.5 MB 00:00 2024-04-25T16:19:24.946 INFO:teuthology.orchestra.run.smithi203.stdout:(6/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 1.3 MB/s | 2.3 MB 00:01 2024-04-25T16:19:24.951 INFO:teuthology.orchestra.run.smithi173.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 342 kB/s | 863 kB 00:02 2024-04-25T16:19:24.960 INFO:teuthology.orchestra.run.smithi098.stdout:(10/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 5.8 MB/s | 18 MB 00:03 2024-04-25T16:19:25.102 INFO:teuthology.orchestra.run.smithi098.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.6 MB/s | 224 kB 00:00 2024-04-25T16:19:25.134 INFO:teuthology.orchestra.run.smithi173.stdout:(3/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-25T16:19:25.169 INFO:teuthology.orchestra.run.smithi098.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 406 kB/s | 27 kB 00:00 2024-04-25T16:19:25.244 INFO:teuthology.orchestra.run.smithi098.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 422 kB/s | 32 kB 00:00 2024-04-25T16:19:25.268 INFO:teuthology.orchestra.run.smithi173.stdout:(4/152): ceph-immutable-object-cache-19.0.0-329 464 kB/s | 147 kB 00:00 2024-04-25T16:19:25.445 INFO:teuthology.orchestra.run.smithi098.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 3.7 MB/s | 756 kB 00:00 2024-04-25T16:19:25.528 INFO:teuthology.orchestra.run.smithi098.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 1.9 MB/s | 158 kB 00:00 2024-04-25T16:19:25.679 INFO:teuthology.orchestra.run.smithi098.stdout:(16/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 3.5 MB/s | 10 MB 00:02 2024-04-25T16:19:25.680 INFO:teuthology.orchestra.run.smithi203.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 4.2 MB/s | 4.9 MB 00:01 2024-04-25T16:19:25.704 INFO:teuthology.orchestra.run.smithi098.stdout:(17/152): librados-devel-19.0.0-3291.g994e2f20. 723 kB/s | 127 kB 00:00 2024-04-25T16:19:25.854 INFO:teuthology.orchestra.run.smithi098.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 2.8 MB/s | 497 kB 00:00 2024-04-25T16:19:25.938 INFO:teuthology.orchestra.run.smithi098.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 540 kB/s | 45 kB 00:00 2024-04-25T16:19:25.993 INFO:teuthology.orchestra.run.smithi173.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 2.6 MB/s | 2.3 MB 00:00 2024-04-25T16:19:26.029 INFO:teuthology.orchestra.run.smithi098.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.4 MB/s | 135 kB 00:00 2024-04-25T16:19:26.130 INFO:teuthology.orchestra.run.smithi098.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 1.6 MB/s | 161 kB 00:00 2024-04-25T16:19:26.288 INFO:teuthology.orchestra.run.smithi098.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 2.0 MB/s | 321 kB 00:00 2024-04-25T16:19:26.439 INFO:teuthology.orchestra.run.smithi098.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 1.9 MB/s | 298 kB 00:00 2024-04-25T16:19:26.522 INFO:teuthology.orchestra.run.smithi098.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.2 MB/s | 100 kB 00:00 2024-04-25T16:19:26.606 INFO:teuthology.orchestra.run.smithi098.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.0 MB/s | 86 kB 00:00 2024-04-25T16:19:26.723 INFO:teuthology.orchestra.run.smithi098.stdout:(26/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 5.1 MB/s | 5.2 MB 00:01 2024-04-25T16:19:26.823 INFO:teuthology.orchestra.run.smithi098.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 1.7 MB/s | 169 kB 00:00 2024-04-25T16:19:26.907 INFO:teuthology.orchestra.run.smithi098.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 410 kB/s | 34 kB 00:00 2024-04-25T16:19:26.982 INFO:teuthology.orchestra.run.smithi098.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.9 MB/s | 142 kB 00:00 2024-04-25T16:19:27.197 INFO:teuthology.orchestra.run.smithi203.stdout:(8/152): ceph-common-19.0.0-3291.g994e2f20.el9. 3.6 MB/s | 20 MB 00:05 2024-04-25T16:19:27.280 INFO:teuthology.orchestra.run.smithi203.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 303 kB/s | 25 kB 00:00 2024-04-25T16:19:27.493 INFO:teuthology.orchestra.run.smithi173.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 692 kB/s | 1.5 MB 00:02 2024-04-25T16:19:27.525 INFO:teuthology.orchestra.run.smithi098.stdout:(30/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 3.3 MB/s | 3.0 MB 00:00 2024-04-25T16:19:27.642 INFO:teuthology.orchestra.run.smithi098.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 5.4 MB/s | 3.5 MB 00:00 2024-04-25T16:19:27.742 INFO:teuthology.orchestra.run.smithi098.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.4 MB/s | 249 kB 00:00 2024-04-25T16:19:27.802 INFO:teuthology.orchestra.run.smithi173.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 2.7 MB/s | 4.9 MB 00:01 2024-04-25T16:19:27.809 INFO:teuthology.orchestra.run.smithi098.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 741 kB/s | 49 kB 00:00 2024-04-25T16:19:27.884 INFO:teuthology.orchestra.run.smithi098.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 219 kB/s | 16 kB 00:00 2024-04-25T16:19:27.976 INFO:teuthology.orchestra.run.smithi098.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 2.9 MB/s | 273 kB 00:00 2024-04-25T16:19:28.160 INFO:teuthology.orchestra.run.smithi098.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 4.1 MB/s | 762 kB 00:00 2024-04-25T16:19:28.244 INFO:teuthology.orchestra.run.smithi098.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 218 kB/s | 18 kB 00:00 2024-04-25T16:19:28.352 INFO:teuthology.orchestra.run.smithi098.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 743 kB/s | 80 kB 00:00 2024-04-25T16:19:28.419 INFO:teuthology.orchestra.run.smithi098.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 12 MB/s | 803 kB 00:00 2024-04-25T16:19:28.453 INFO:teuthology.orchestra.run.smithi098.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 5.8 MB/s | 195 kB 00:00 2024-04-25T16:19:28.495 INFO:teuthology.orchestra.run.smithi098.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.9 MB/s | 253 kB 00:00 2024-04-25T16:19:28.553 INFO:teuthology.orchestra.run.smithi098.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 21 MB/s | 1.2 MB 00:00 2024-04-25T16:19:28.587 INFO:teuthology.orchestra.run.smithi098.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 106 kB 00:00 2024-04-25T16:19:28.620 INFO:teuthology.orchestra.run.smithi098.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 4.0 MB/s | 135 kB 00:00 2024-04-25T16:19:28.662 INFO:teuthology.orchestra.run.smithi098.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T16:19:28.689 INFO:teuthology.orchestra.run.smithi203.stdout:(10/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 3.4 MB/s | 10 MB 00:03 2024-04-25T16:19:28.713 INFO:teuthology.orchestra.run.smithi098.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T16:19:28.780 INFO:teuthology.orchestra.run.smithi098.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 10 MB/s | 689 kB 00:00 2024-04-25T16:19:28.831 INFO:teuthology.orchestra.run.smithi203.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.6 MB/s | 224 kB 00:00 2024-04-25T16:19:28.863 INFO:teuthology.orchestra.run.smithi098.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 943 kB 00:00 2024-04-25T16:19:28.906 INFO:teuthology.orchestra.run.smithi203.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 360 kB/s | 27 kB 00:00 2024-04-25T16:19:28.922 INFO:teuthology.orchestra.run.smithi098.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T16:19:28.964 INFO:teuthology.orchestra.run.smithi098.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 516 kB/s | 21 kB 00:00 2024-04-25T16:19:28.973 INFO:teuthology.orchestra.run.smithi203.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 476 kB/s | 32 kB 00:00 2024-04-25T16:19:29.006 INFO:teuthology.orchestra.run.smithi098.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 313 kB 00:00 2024-04-25T16:19:29.165 INFO:teuthology.orchestra.run.smithi203.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 3.8 MB/s | 756 kB 00:00 2024-04-25T16:19:29.224 INFO:teuthology.orchestra.run.smithi203.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.6 MB/s | 158 kB 00:00 2024-04-25T16:19:29.291 INFO:teuthology.orchestra.run.smithi203.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.9 MB/s | 127 kB 00:00 2024-04-25T16:19:29.398 INFO:teuthology.orchestra.run.smithi098.stdout:(52/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 6.9 MB/s | 43 MB 00:06 2024-04-25T16:19:29.423 INFO:teuthology.orchestra.run.smithi098.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 78 kB/s | 33 kB 00:00 2024-04-25T16:19:29.441 INFO:teuthology.orchestra.run.smithi203.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 3.2 MB/s | 497 kB 00:00 2024-04-25T16:19:29.532 INFO:teuthology.orchestra.run.smithi098.stdout:(54/152): ceph-mgr-diskprediction-local-19.0.0- 3.7 MB/s | 7.4 MB 00:02 2024-04-25T16:19:29.591 INFO:teuthology.orchestra.run.smithi098.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-04-25T16:19:29.616 INFO:teuthology.orchestra.run.smithi098.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 95 kB/s | 18 kB 00:00 2024-04-25T16:19:29.641 INFO:teuthology.orchestra.run.smithi098.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 105 kB/s | 11 kB 00:00 2024-04-25T16:19:29.666 INFO:teuthology.orchestra.run.smithi098.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T16:19:29.692 INFO:teuthology.orchestra.run.smithi098.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 601 kB/s | 45 kB 00:00 2024-04-25T16:19:29.717 INFO:teuthology.orchestra.run.smithi098.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-25T16:19:29.742 INFO:teuthology.orchestra.run.smithi098.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 582 kB 00:00 2024-04-25T16:19:29.759 INFO:teuthology.orchestra.run.smithi098.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 388 kB/s | 25 kB 00:00 2024-04-25T16:19:29.776 INFO:teuthology.orchestra.run.smithi098.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 827 kB/s | 47 kB 00:00 2024-04-25T16:19:29.801 INFO:teuthology.orchestra.run.smithi098.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-04-25T16:19:29.826 INFO:teuthology.orchestra.run.smithi098.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-25T16:19:29.843 INFO:teuthology.orchestra.run.smithi098.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 534 kB/s | 35 kB 00:00 2024-04-25T16:19:30.010 INFO:teuthology.orchestra.run.smithi098.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-04-25T16:19:30.077 INFO:teuthology.orchestra.run.smithi098.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-25T16:19:30.119 INFO:teuthology.orchestra.run.smithi098.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-04-25T16:19:30.208 INFO:teuthology.orchestra.run.smithi098.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 4.6 MB/s | 911 kB 00:00 2024-04-25T16:19:30.236 INFO:teuthology.orchestra.run.smithi098.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 849 kB/s | 133 kB 00:00 2024-04-25T16:19:30.283 INFO:teuthology.orchestra.run.smithi203.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 6.1 MB/s | 5.2 MB 00:00 2024-04-25T16:19:30.295 INFO:teuthology.orchestra.run.smithi098.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-25T16:19:30.350 INFO:teuthology.orchestra.run.smithi203.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 680 kB/s | 45 kB 00:00 2024-04-25T16:19:30.370 INFO:teuthology.orchestra.run.smithi098.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 36 MB/s | 5.8 MB 00:00 2024-04-25T16:19:30.395 INFO:teuthology.orchestra.run.smithi098.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 559 kB/s | 87 kB 00:00 2024-04-25T16:19:30.408 INFO:teuthology.orchestra.run.smithi203.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 2.3 MB/s | 135 kB 00:00 2024-04-25T16:19:30.468 INFO:teuthology.orchestra.run.smithi173.stdout:(8/152): ceph-common-19.0.0-3291.g994e2f20.el9. 2.3 MB/s | 20 MB 00:08 2024-04-25T16:19:30.475 INFO:teuthology.orchestra.run.smithi203.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.4 MB/s | 161 kB 00:00 2024-04-25T16:19:30.504 INFO:teuthology.orchestra.run.smithi098.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 6.0 MB 00:00 2024-04-25T16:19:30.529 INFO:teuthology.orchestra.run.smithi098.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2024-04-25T16:19:30.535 INFO:teuthology.orchestra.run.smithi173.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 380 kB/s | 25 kB 00:00 2024-04-25T16:19:30.554 INFO:teuthology.orchestra.run.smithi098.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.5 MB/s | 248 kB 00:00 2024-04-25T16:19:30.580 INFO:teuthology.orchestra.run.smithi098.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 343 kB/s | 26 kB 00:00 2024-04-25T16:19:30.592 INFO:teuthology.orchestra.run.smithi203.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 2.7 MB/s | 321 kB 00:00 2024-04-25T16:19:30.605 INFO:teuthology.orchestra.run.smithi098.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-25T16:19:30.630 INFO:teuthology.orchestra.run.smithi098.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T16:19:30.644 INFO:teuthology.orchestra.run.smithi173.stdout:(10/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 3.6 MB/s | 10 MB 00:02 2024-04-25T16:19:30.655 INFO:teuthology.orchestra.run.smithi098.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 463 kB/s | 35 kB 00:00 2024-04-25T16:19:30.697 INFO:teuthology.orchestra.run.smithi098.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-25T16:19:30.734 INFO:teuthology.orchestra.run.smithi203.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 2.1 MB/s | 298 kB 00:00 2024-04-25T16:19:30.744 INFO:teuthology.orchestra.run.smithi173.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.2 MB/s | 224 kB 00:00 2024-04-25T16:19:30.764 INFO:teuthology.orchestra.run.smithi098.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 5.1 MB 00:00 2024-04-25T16:19:30.789 INFO:teuthology.orchestra.run.smithi098.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-04-25T16:19:30.814 INFO:teuthology.orchestra.run.smithi098.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 672 kB/s | 77 kB 00:00 2024-04-25T16:19:30.818 INFO:teuthology.orchestra.run.smithi203.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.2 MB/s | 100 kB 00:00 2024-04-25T16:19:30.840 INFO:teuthology.orchestra.run.smithi098.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-04-25T16:19:30.844 INFO:teuthology.orchestra.run.smithi173.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 271 kB/s | 27 kB 00:00 2024-04-25T16:19:30.865 INFO:teuthology.orchestra.run.smithi098.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-25T16:19:30.890 INFO:teuthology.orchestra.run.smithi098.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-25T16:19:30.910 INFO:teuthology.orchestra.run.smithi203.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 938 kB/s | 86 kB 00:00 2024-04-25T16:19:30.916 INFO:teuthology.orchestra.run.smithi098.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T16:19:30.941 INFO:teuthology.orchestra.run.smithi098.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 681 kB/s | 51 kB 00:00 2024-04-25T16:19:30.944 INFO:teuthology.orchestra.run.smithi173.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 317 kB/s | 32 kB 00:00 2024-04-25T16:19:30.966 INFO:teuthology.orchestra.run.smithi098.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 718 kB/s | 54 kB 00:00 2024-04-25T16:19:30.992 INFO:teuthology.orchestra.run.smithi098.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 827 kB/s | 42 kB 00:00 2024-04-25T16:19:31.092 INFO:teuthology.orchestra.run.smithi098.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 359 kB/s | 44 kB 00:00 2024-04-25T16:19:31.293 INFO:teuthology.orchestra.run.smithi098.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-04-25T16:19:31.318 INFO:teuthology.orchestra.run.smithi098.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 683 kB/s | 221 kB 00:00 2024-04-25T16:19:31.328 INFO:teuthology.orchestra.run.smithi173.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 1.9 MB/s | 756 kB 00:00 2024-04-25T16:19:31.343 INFO:teuthology.orchestra.run.smithi098.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 144 kB/s | 36 kB 00:00 2024-04-25T16:19:31.368 INFO:teuthology.orchestra.run.smithi098.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 635 kB/s | 47 kB 00:00 2024-04-25T16:19:31.393 INFO:teuthology.orchestra.run.smithi098.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-25T16:19:31.418 INFO:teuthology.orchestra.run.smithi098.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 770 kB/s | 58 kB 00:00 2024-04-25T16:19:31.443 INFO:teuthology.orchestra.run.smithi098.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-25T16:19:31.453 INFO:teuthology.orchestra.run.smithi173.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 1.2 MB/s | 158 kB 00:00 2024-04-25T16:19:31.469 INFO:teuthology.orchestra.run.smithi098.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-25T16:19:31.494 INFO:teuthology.orchestra.run.smithi098.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 599 kB/s | 45 kB 00:00 2024-04-25T16:19:31.519 INFO:teuthology.orchestra.run.smithi098.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-04-25T16:19:31.537 INFO:teuthology.orchestra.run.smithi173.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.5 MB/s | 127 kB 00:00 2024-04-25T16:19:31.553 INFO:teuthology.orchestra.run.smithi098.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-04-25T16:19:31.645 INFO:teuthology.orchestra.run.smithi098.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-04-25T16:19:31.694 INFO:teuthology.orchestra.run.smithi203.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 3.9 MB/s | 3.0 MB 00:00 2024-04-25T16:19:31.704 INFO:teuthology.orchestra.run.smithi098.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.7 MB/s | 309 kB 00:00 2024-04-25T16:19:31.754 INFO:teuthology.orchestra.run.smithi098.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.1 MB/s | 548 kB 00:00 2024-04-25T16:19:31.770 INFO:teuthology.orchestra.run.smithi173.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 2.1 MB/s | 497 kB 00:00 2024-04-25T16:19:31.777 INFO:teuthology.orchestra.run.smithi203.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.0 MB/s | 169 kB 00:00 2024-04-25T16:19:31.779 INFO:teuthology.orchestra.run.smithi098.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-04-25T16:19:31.805 INFO:teuthology.orchestra.run.smithi098.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-04-25T16:19:31.830 INFO:teuthology.orchestra.run.smithi098.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-25T16:19:31.847 INFO:teuthology.orchestra.run.smithi098.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-04-25T16:19:31.877 INFO:teuthology.orchestra.run.smithi203.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 342 kB/s | 34 kB 00:00 2024-04-25T16:19:31.914 INFO:teuthology.orchestra.run.smithi098.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 8.0 MB/s | 548 kB 00:00 2024-04-25T16:19:31.947 INFO:teuthology.orchestra.run.smithi098.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-25T16:19:31.964 INFO:teuthology.orchestra.run.smithi098.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 867 kB/s | 43 kB 00:00 2024-04-25T16:19:31.969 INFO:teuthology.orchestra.run.smithi203.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.5 MB/s | 142 kB 00:00 2024-04-25T16:19:31.990 INFO:teuthology.orchestra.run.smithi098.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 766 kB/s | 32 kB 00:00 2024-04-25T16:19:32.006 INFO:teuthology.orchestra.run.smithi098.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 357 kB/s | 15 kB 00:00 2024-04-25T16:19:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 4.0 MB/s | 172 kB 00:00 2024-04-25T16:19:32.082 INFO:teuthology.orchestra.run.smithi098.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-04-25T16:19:32.107 INFO:teuthology.orchestra.run.smithi098.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-25T16:19:32.124 INFO:teuthology.orchestra.run.smithi098.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 838 kB/s | 35 kB 00:00 2024-04-25T16:19:32.191 INFO:teuthology.orchestra.run.smithi098.stdout:(121/152): libarrow-9.0.0-10.el9.x86_64.rpm 8.1 MB/s | 4.4 MB 00:00 2024-04-25T16:19:32.217 INFO:teuthology.orchestra.run.smithi098.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.8 MB/s | 206 kB 00:00 2024-04-25T16:19:32.242 INFO:teuthology.orchestra.run.smithi098.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-25T16:19:32.267 INFO:teuthology.orchestra.run.smithi098.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-04-25T16:19:32.279 INFO:teuthology.orchestra.run.smithi203.stdout:(30/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 2.4 MB/s | 18 MB 00:07 2024-04-25T16:19:32.293 INFO:teuthology.orchestra.run.smithi098.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-04-25T16:19:32.318 INFO:teuthology.orchestra.run.smithi098.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 461 kB/s | 23 kB 00:00 2024-04-25T16:19:32.344 INFO:teuthology.orchestra.run.smithi098.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-04-25T16:19:32.368 INFO:teuthology.orchestra.run.smithi098.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-04-25T16:19:32.419 INFO:teuthology.orchestra.run.smithi098.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 6.9 MB/s | 2.0 MB 00:00 2024-04-25T16:19:32.444 INFO:teuthology.orchestra.run.smithi098.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 616 kB/s | 46 kB 00:00 2024-04-25T16:19:32.461 INFO:teuthology.orchestra.run.smithi098.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-04-25T16:19:32.495 INFO:teuthology.orchestra.run.smithi098.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-04-25T16:19:32.520 INFO:teuthology.orchestra.run.smithi098.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-04-25T16:19:32.546 INFO:teuthology.orchestra.run.smithi098.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-04-25T16:19:32.570 INFO:teuthology.orchestra.run.smithi098.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.7 MB/s | 90 kB 00:00 2024-04-25T16:19:32.596 INFO:teuthology.orchestra.run.smithi098.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 565 kB/s | 28 kB 00:00 2024-04-25T16:19:32.621 INFO:teuthology.orchestra.run.smithi098.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-04-25T16:19:32.647 INFO:teuthology.orchestra.run.smithi098.stdout:(138/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-04-25T16:19:32.680 INFO:teuthology.orchestra.run.smithi098.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-25T16:19:32.705 INFO:teuthology.orchestra.run.smithi098.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 613 kB/s | 36 kB 00:00 2024-04-25T16:19:32.722 INFO:teuthology.orchestra.run.smithi098.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-25T16:19:32.751 INFO:teuthology.orchestra.run.smithi098.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 761 kB/s | 51 kB 00:00 2024-04-25T16:19:32.773 INFO:teuthology.orchestra.run.smithi098.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-25T16:19:32.806 INFO:teuthology.orchestra.run.smithi098.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-25T16:19:32.831 INFO:teuthology.orchestra.run.smithi098.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 342 kB/s | 20 kB 00:00 2024-04-25T16:19:32.857 INFO:teuthology.orchestra.run.smithi098.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-25T16:19:32.882 INFO:teuthology.orchestra.run.smithi098.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-25T16:19:32.958 INFO:teuthology.orchestra.run.smithi098.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-25T16:19:32.962 INFO:teuthology.orchestra.run.smithi203.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 3.6 MB/s | 3.5 MB 00:00 2024-04-25T16:19:33.016 INFO:teuthology.orchestra.run.smithi098.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T16:19:33.054 INFO:teuthology.orchestra.run.smithi203.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.7 MB/s | 249 kB 00:00 2024-04-25T16:19:33.091 INFO:teuthology.orchestra.run.smithi098.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 32 MB/s | 2.4 MB 00:00 2024-04-25T16:19:33.146 INFO:teuthology.orchestra.run.smithi203.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 538 kB/s | 49 kB 00:00 2024-04-25T16:19:33.247 INFO:teuthology.orchestra.run.smithi203.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 164 kB/s | 16 kB 00:00 2024-04-25T16:19:33.365 INFO:teuthology.orchestra.run.smithi203.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 2.3 MB/s | 273 kB 00:00 2024-04-25T16:19:33.614 INFO:teuthology.orchestra.run.smithi203.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 3.0 MB/s | 762 kB 00:00 2024-04-25T16:19:33.620 INFO:teuthology.orchestra.run.smithi173.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 2.8 MB/s | 5.2 MB 00:01 2024-04-25T16:19:33.722 INFO:teuthology.orchestra.run.smithi203.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 169 kB/s | 18 kB 00:00 2024-04-25T16:19:33.737 INFO:teuthology.orchestra.run.smithi173.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 388 kB/s | 45 kB 00:00 2024-04-25T16:19:33.784 INFO:teuthology.orchestra.run.smithi098.stdout:(151/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 3.4 MB/s | 3.1 MB 00:00 2024-04-25T16:19:33.831 INFO:teuthology.orchestra.run.smithi203.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 742 kB/s | 80 kB 00:00 2024-04-25T16:19:33.846 INFO:teuthology.orchestra.run.smithi173.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.2 MB/s | 135 kB 00:00 2024-04-25T16:19:33.906 INFO:teuthology.orchestra.run.smithi203.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 803 kB 00:00 2024-04-25T16:19:33.963 INFO:teuthology.orchestra.run.smithi173.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 1.4 MB/s | 161 kB 00:00 2024-04-25T16:19:33.990 INFO:teuthology.orchestra.run.smithi203.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-25T16:19:34.040 INFO:teuthology.orchestra.run.smithi203.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-25T16:19:34.099 INFO:teuthology.orchestra.run.smithi203.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 21 MB/s | 1.2 MB 00:00 2024-04-25T16:19:34.109 INFO:teuthology.orchestra.run.smithi098.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 2.7 MB/s | 3.3 MB 00:01 2024-04-25T16:19:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:19:34.113 INFO:teuthology.orchestra.run.smithi098.stdout:Total 16 MB/s | 214 MB 00:13 2024-04-25T16:19:34.132 INFO:teuthology.orchestra.run.smithi203.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 106 kB 00:00 2024-04-25T16:19:34.163 INFO:teuthology.orchestra.run.smithi173.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 1.6 MB/s | 321 kB 00:00 2024-04-25T16:19:34.175 INFO:teuthology.orchestra.run.smithi203.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T16:19:34.200 INFO:teuthology.orchestra.run.smithi203.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 4.9 MB/s | 125 kB 00:00 2024-04-25T16:19:34.234 INFO:teuthology.orchestra.run.smithi203.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 6.3 MB/s | 215 kB 00:00 2024-04-25T16:19:34.300 INFO:teuthology.orchestra.run.smithi203.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 10 MB/s | 689 kB 00:00 2024-04-25T16:19:34.351 INFO:teuthology.orchestra.run.smithi203.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 943 kB 00:00 2024-04-25T16:19:34.364 INFO:teuthology.orchestra.run.smithi173.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 1.5 MB/s | 298 kB 00:00 2024-04-25T16:19:34.385 INFO:teuthology.orchestra.run.smithi203.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 107 kB 00:00 2024-04-25T16:19:34.410 INFO:teuthology.orchestra.run.smithi203.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 865 kB/s | 21 kB 00:00 2024-04-25T16:19:34.452 INFO:teuthology.orchestra.run.smithi203.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 313 kB 00:00 2024-04-25T16:19:34.472 INFO:teuthology.orchestra.run.smithi173.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 921 kB/s | 100 kB 00:00 2024-04-25T16:19:34.494 INFO:teuthology.orchestra.run.smithi203.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 781 kB/s | 33 kB 00:00 2024-04-25T16:19:34.589 INFO:teuthology.orchestra.run.smithi173.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 737 kB/s | 86 kB 00:00 2024-04-25T16:19:34.603 INFO:teuthology.orchestra.run.smithi203.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-04-25T16:19:34.653 INFO:teuthology.orchestra.run.smithi203.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-04-25T16:19:34.704 INFO:teuthology.orchestra.run.smithi203.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-25T16:19:34.754 INFO:teuthology.orchestra.run.smithi203.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T16:19:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-04-25T16:19:34.880 INFO:teuthology.orchestra.run.smithi203.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 662 kB 00:00 2024-04-25T16:19:34.947 INFO:teuthology.orchestra.run.smithi203.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-04-25T16:19:35.039 INFO:teuthology.orchestra.run.smithi203.stdout:(60/152): ceph-mgr-diskprediction-local-19.0.0- 2.7 MB/s | 7.4 MB 00:02 2024-04-25T16:19:35.064 INFO:teuthology.orchestra.run.smithi203.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 218 kB/s | 25 kB 00:00 2024-04-25T16:19:35.089 INFO:teuthology.orchestra.run.smithi203.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-04-25T16:19:35.123 INFO:teuthology.orchestra.run.smithi203.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-25T16:19:35.148 INFO:teuthology.orchestra.run.smithi203.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-04-25T16:19:35.173 INFO:teuthology.orchestra.run.smithi203.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 706 kB/s | 35 kB 00:00 2024-04-25T16:19:35.216 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-04-25T16:19:35.358 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-04-25T16:19:35.358 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-04-25T16:19:35.407 INFO:teuthology.orchestra.run.smithi203.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-04-25T16:19:35.466 INFO:teuthology.orchestra.run.smithi203.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-04-25T16:19:35.533 INFO:teuthology.orchestra.run.smithi203.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-25T16:19:35.566 INFO:teuthology.orchestra.run.smithi203.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 9.1 MB/s | 911 kB 00:00 2024-04-25T16:19:35.592 INFO:teuthology.orchestra.run.smithi203.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-25T16:19:35.623 INFO:teuthology.orchestra.run.smithi173.stdout:(26/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 2.2 MB/s | 18 MB 00:08 2024-04-25T16:19:35.940 INFO:teuthology.orchestra.run.smithi173.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 535 kB/s | 169 kB 00:00 2024-04-25T16:19:35.992 INFO:teuthology.orchestra.run.smithi203.stdout:(71/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 5.0 MB/s | 43 MB 00:08 2024-04-25T16:19:36.034 INFO:teuthology.orchestra.run.smithi203.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 2.3 MB/s | 1.0 MB 00:00 2024-04-25T16:19:36.048 INFO:teuthology.orchestra.run.smithi173.stdout:(28/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 2.1 MB/s | 3.0 MB 00:01 2024-04-25T16:19:36.074 INFO:teuthology.orchestra.run.smithi173.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3291.g 256 kB/s | 34 kB 00:00 2024-04-25T16:19:36.109 INFO:teuthology.orchestra.run.smithi203.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-04-25T16:19:36.126 INFO:teuthology.orchestra.run.smithi203.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 664 kB/s | 87 kB 00:00 2024-04-25T16:19:36.158 INFO:teuthology.orchestra.run.smithi173.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.3 MB/s | 142 kB 00:00 2024-04-25T16:19:36.194 INFO:teuthology.orchestra.run.smithi203.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.8 MB/s | 248 kB 00:00 2024-04-25T16:19:36.269 INFO:teuthology.orchestra.run.smithi203.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 6.0 MB 00:00 2024-04-25T16:19:36.294 INFO:teuthology.orchestra.run.smithi203.stdout:(77/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2024-04-25T16:19:36.311 INFO:teuthology.orchestra.run.smithi203.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 219 kB/s | 26 kB 00:00 2024-04-25T16:19:36.328 INFO:teuthology.orchestra.run.smithi203.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 571 kB/s | 19 kB 00:00 2024-04-25T16:19:36.353 INFO:teuthology.orchestra.run.smithi203.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-25T16:19:36.370 INFO:teuthology.orchestra.run.smithi203.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 593 kB/s | 35 kB 00:00 2024-04-25T16:19:36.395 INFO:teuthology.orchestra.run.smithi203.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.9 MB/s | 250 kB 00:00 2024-04-25T16:19:36.420 INFO:teuthology.orchestra.run.smithi203.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.3 MB/s | 222 kB 00:00 2024-04-25T16:19:36.446 INFO:teuthology.orchestra.run.smithi203.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.6 MB/s | 77 kB 00:00 2024-04-25T16:19:36.513 INFO:teuthology.orchestra.run.smithi203.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-04-25T16:19:36.538 INFO:teuthology.orchestra.run.smithi203.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 369 kB/s | 42 kB 00:00 2024-04-25T16:19:36.563 INFO:teuthology.orchestra.run.smithi203.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.3 MB/s | 269 kB 00:00 2024-04-25T16:19:36.588 INFO:teuthology.orchestra.run.smithi203.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T16:19:36.614 INFO:teuthology.orchestra.run.smithi203.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 279 kB 00:00 2024-04-25T16:19:36.639 INFO:teuthology.orchestra.run.smithi203.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-04-25T16:19:36.664 INFO:teuthology.orchestra.run.smithi203.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T16:19:36.689 INFO:teuthology.orchestra.run.smithi203.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-04-25T16:19:36.715 INFO:teuthology.orchestra.run.smithi203.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 901 kB/s | 44 kB 00:00 2024-04-25T16:19:36.740 INFO:teuthology.orchestra.run.smithi203.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.6 MB/s | 221 kB 00:00 2024-04-25T16:19:36.766 INFO:teuthology.orchestra.run.smithi203.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 765 kB/s | 36 kB 00:00 2024-04-25T16:19:36.799 INFO:teuthology.orchestra.run.smithi203.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 855 kB/s | 47 kB 00:00 2024-04-25T16:19:36.833 INFO:teuthology.orchestra.run.smithi203.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.7 MB/s | 305 kB 00:00 2024-04-25T16:19:36.858 INFO:teuthology.orchestra.run.smithi203.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 58 kB 00:00 2024-04-25T16:19:37.042 INFO:teuthology.orchestra.run.smithi203.stdout:(99/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-04-25T16:19:37.059 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-04-25T16:19:37.059 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-04-25T16:19:37.067 INFO:teuthology.orchestra.run.smithi203.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 406 kB/s | 94 kB 00:00 2024-04-25T16:19:37.092 INFO:teuthology.orchestra.run.smithi203.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 276 kB/s | 64 kB 00:00 2024-04-25T16:19:37.118 INFO:teuthology.orchestra.run.smithi203.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-04-25T16:19:37.143 INFO:teuthology.orchestra.run.smithi203.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 593 kB/s | 45 kB 00:00 2024-04-25T16:19:37.169 INFO:teuthology.orchestra.run.smithi203.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-25T16:19:37.286 INFO:teuthology.orchestra.run.smithi203.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-04-25T16:19:37.377 INFO:teuthology.orchestra.run.smithi203.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.4 MB/s | 309 kB 00:00 2024-04-25T16:19:37.402 INFO:teuthology.orchestra.run.smithi203.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-04-25T16:19:37.428 INFO:teuthology.orchestra.run.smithi203.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 516 kB/s | 26 kB 00:00 2024-04-25T16:19:37.453 INFO:teuthology.orchestra.run.smithi203.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 966 kB/s | 49 kB 00:00 2024-04-25T16:19:37.495 INFO:teuthology.orchestra.run.smithi203.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-25T16:19:37.520 INFO:teuthology.orchestra.run.smithi203.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-25T16:19:37.612 INFO:teuthology.orchestra.run.smithi203.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-25T16:19:37.646 INFO:teuthology.orchestra.run.smithi203.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 838 kB 00:00 2024-04-25T16:19:37.671 INFO:teuthology.orchestra.run.smithi203.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 741 kB/s | 43 kB 00:00 2024-04-25T16:19:37.697 INFO:teuthology.orchestra.run.smithi203.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 3.0 MB/s | 548 kB 00:00 2024-04-25T16:19:37.722 INFO:teuthology.orchestra.run.smithi203.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-25T16:19:37.747 INFO:teuthology.orchestra.run.smithi203.stdout:(117/152): python3-cachetools-4.2.4-1.el9.noarc 319 kB/s | 32 kB 00:00 2024-04-25T16:19:37.773 INFO:teuthology.orchestra.run.smithi203.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-25T16:19:37.798 INFO:teuthology.orchestra.run.smithi203.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-04-25T16:19:37.823 INFO:teuthology.orchestra.run.smithi203.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-25T16:19:37.840 INFO:teuthology.orchestra.run.smithi203.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 526 kB/s | 35 kB 00:00 2024-04-25T16:19:37.865 INFO:teuthology.orchestra.run.smithi203.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-25T16:19:37.891 INFO:teuthology.orchestra.run.smithi203.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-25T16:19:37.916 INFO:teuthology.orchestra.run.smithi203.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-04-25T16:19:37.941 INFO:teuthology.orchestra.run.smithi203.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-04-25T16:19:37.967 INFO:teuthology.orchestra.run.smithi203.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 459 kB/s | 23 kB 00:00 2024-04-25T16:19:37.992 INFO:teuthology.orchestra.run.smithi203.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 387 kB/s | 19 kB 00:00 2024-04-25T16:19:38.017 INFO:teuthology.orchestra.run.smithi203.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 391 kB/s | 20 kB 00:00 2024-04-25T16:19:38.076 INFO:teuthology.orchestra.run.smithi203.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-04-25T16:19:38.102 INFO:teuthology.orchestra.run.smithi203.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-04-25T16:19:38.135 INFO:teuthology.orchestra.run.smithi203.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-25T16:19:38.160 INFO:teuthology.orchestra.run.smithi203.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 983 kB/s | 58 kB 00:00 2024-04-25T16:19:38.186 INFO:teuthology.orchestra.run.smithi203.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-25T16:19:38.211 INFO:teuthology.orchestra.run.smithi203.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-04-25T16:19:38.236 INFO:teuthology.orchestra.run.smithi203.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-25T16:19:38.262 INFO:teuthology.orchestra.run.smithi203.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-04-25T16:19:38.287 INFO:teuthology.orchestra.run.smithi203.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-25T16:19:38.312 INFO:teuthology.orchestra.run.smithi203.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.7 MB/s | 188 kB 00:00 2024-04-25T16:19:38.337 INFO:teuthology.orchestra.run.smithi203.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-04-25T16:19:38.363 INFO:teuthology.orchestra.run.smithi203.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2024-04-25T16:19:38.388 INFO:teuthology.orchestra.run.smithi203.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T16:19:38.422 INFO:teuthology.orchestra.run.smithi203.stdout:(142/152): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-04-25T16:19:38.447 INFO:teuthology.orchestra.run.smithi203.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-25T16:19:38.473 INFO:teuthology.orchestra.run.smithi203.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-25T16:19:38.498 INFO:teuthology.orchestra.run.smithi203.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-04-25T16:19:38.523 INFO:teuthology.orchestra.run.smithi203.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-25T16:19:38.548 INFO:teuthology.orchestra.run.smithi203.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-04-25T16:19:38.599 INFO:teuthology.orchestra.run.smithi203.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T16:19:38.665 INFO:teuthology.orchestra.run.smithi203.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 20 MB/s | 1.3 MB 00:00 2024-04-25T16:19:38.758 INFO:teuthology.orchestra.run.smithi203.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-04-25T16:19:39.233 INFO:teuthology.orchestra.run.smithi173.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 1.1 MB/s | 3.5 MB 00:03 2024-04-25T16:19:39.458 INFO:teuthology.orchestra.run.smithi173.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 1.1 MB/s | 249 kB 00:00 2024-04-25T16:19:39.558 INFO:teuthology.orchestra.run.smithi173.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 493 kB/s | 49 kB 00:00 2024-04-25T16:19:39.658 INFO:teuthology.orchestra.run.smithi173.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 164 kB/s | 16 kB 00:00 2024-04-25T16:19:39.959 INFO:teuthology.orchestra.run.smithi173.stdout:(35/152): ceph-mgr-diskprediction-local-19.0.0- 1.9 MB/s | 7.4 MB 00:03 2024-04-25T16:19:39.984 INFO:teuthology.orchestra.run.smithi173.stdout:(36/152): ceph-volume-19.0.0-3291.g994e2f20.el9 840 kB/s | 273 kB 00:00 2024-04-25T16:19:40.092 INFO:teuthology.orchestra.run.smithi173.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 169 kB/s | 18 kB 00:00 2024-04-25T16:19:40.193 INFO:teuthology.orchestra.run.smithi173.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 804 kB/s | 80 kB 00:00 2024-04-25T16:19:40.268 INFO:teuthology.orchestra.run.smithi173.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 803 kB 00:00 2024-04-25T16:19:40.310 INFO:teuthology.orchestra.run.smithi173.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-25T16:19:40.352 INFO:teuthology.orchestra.run.smithi173.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.0 MB/s | 253 kB 00:00 2024-04-25T16:19:40.441 INFO:teuthology.orchestra.run.smithi203.stdout:(151/152): librados2-19.0.0-3291.g994e2f20.el9. 1.7 MB/s | 3.3 MB 00:01 2024-04-25T16:19:40.485 INFO:teuthology.orchestra.run.smithi173.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-25T16:19:40.602 INFO:teuthology.orchestra.run.smithi173.stdout:(43/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 1.2 MB/s | 762 kB 00:00 2024-04-25T16:19:40.628 INFO:teuthology.orchestra.run.smithi173.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 750 kB/s | 106 kB 00:00 2024-04-25T16:19:40.644 INFO:teuthology.orchestra.run.smithi173.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-04-25T16:19:40.670 INFO:teuthology.orchestra.run.smithi173.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.9 MB/s | 125 kB 00:00 2024-04-25T16:19:40.695 INFO:teuthology.orchestra.run.smithi173.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T16:19:40.721 INFO:teuthology.orchestra.run.smithi173.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 14 MB/s | 689 kB 00:00 2024-04-25T16:19:40.754 INFO:teuthology.orchestra.run.smithi173.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T16:19:40.781 INFO:teuthology.orchestra.run.smithi173.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T16:19:40.807 INFO:teuthology.orchestra.run.smithi173.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 413 kB/s | 21 kB 00:00 2024-04-25T16:19:40.832 INFO:teuthology.orchestra.run.smithi173.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T16:19:40.857 INFO:teuthology.orchestra.run.smithi173.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 660 kB/s | 33 kB 00:00 2024-04-25T16:19:40.933 INFO:teuthology.orchestra.run.smithi173.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-04-25T16:19:40.958 INFO:teuthology.orchestra.run.smithi173.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 187 kB/s | 18 kB 00:00 2024-04-25T16:19:40.983 INFO:teuthology.orchestra.run.smithi173.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-25T16:19:41.008 INFO:teuthology.orchestra.run.smithi173.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T16:19:41.034 INFO:teuthology.orchestra.run.smithi173.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-04-25T16:19:41.067 INFO:teuthology.orchestra.run.smithi173.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T16:19:41.101 INFO:teuthology.orchestra.run.smithi173.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.7 MB/s | 582 kB 00:00 2024-04-25T16:19:41.126 INFO:teuthology.orchestra.run.smithi173.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 443 kB/s | 25 kB 00:00 2024-04-25T16:19:41.151 INFO:teuthology.orchestra.run.smithi173.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-25T16:19:41.177 INFO:teuthology.orchestra.run.smithi173.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T16:19:41.202 INFO:teuthology.orchestra.run.smithi173.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T16:19:41.227 INFO:teuthology.orchestra.run.smithi173.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 707 kB/s | 35 kB 00:00 2024-04-25T16:19:41.433 INFO:teuthology.orchestra.run.smithi203.stdout:(152/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 1.1 MB/s | 3.1 MB 00:02 2024-04-25T16:19:41.437 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:19:41.437 INFO:teuthology.orchestra.run.smithi203.stdout:Total 11 MB/s | 214 MB 00:20 2024-04-25T16:19:41.678 INFO:teuthology.orchestra.run.smithi173.stdout:(66/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 3.9 MB/s | 43 MB 00:11 2024-04-25T16:19:41.761 INFO:teuthology.orchestra.run.smithi173.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 8.5 MB/s | 4.7 MB 00:00 2024-04-25T16:19:41.836 INFO:teuthology.orchestra.run.smithi173.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.3 MB/s | 4.4 MB 00:00 2024-04-25T16:19:41.870 INFO:teuthology.orchestra.run.smithi173.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 8.4 MB/s | 1.6 MB 00:00 2024-04-25T16:19:41.895 INFO:teuthology.orchestra.run.smithi173.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 6.8 MB/s | 911 kB 00:00 2024-04-25T16:19:41.920 INFO:teuthology.orchestra.run.smithi173.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-25T16:19:41.996 INFO:teuthology.orchestra.run.smithi173.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-25T16:19:42.071 INFO:teuthology.orchestra.run.smithi173.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 33 MB/s | 5.8 MB 00:00 2024-04-25T16:19:42.088 INFO:teuthology.orchestra.run.smithi173.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 530 kB/s | 87 kB 00:00 2024-04-25T16:19:42.188 INFO:teuthology.orchestra.run.smithi173.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 31 MB/s | 6.0 MB 00:00 2024-04-25T16:19:42.213 INFO:teuthology.orchestra.run.smithi173.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.9 MB/s | 244 kB 00:00 2024-04-25T16:19:42.238 INFO:teuthology.orchestra.run.smithi173.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.6 MB/s | 248 kB 00:00 2024-04-25T16:19:42.263 INFO:teuthology.orchestra.run.smithi173.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 345 kB/s | 26 kB 00:00 2024-04-25T16:19:42.289 INFO:teuthology.orchestra.run.smithi173.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-25T16:19:42.314 INFO:teuthology.orchestra.run.smithi173.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T16:19:42.339 INFO:teuthology.orchestra.run.smithi173.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 462 kB/s | 35 kB 00:00 2024-04-25T16:19:42.381 INFO:teuthology.orchestra.run.smithi173.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.6 MB/s | 250 kB 00:00 2024-04-25T16:19:42.448 INFO:teuthology.orchestra.run.smithi173.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 5.1 MB 00:00 2024-04-25T16:19:42.473 INFO:teuthology.orchestra.run.smithi173.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-04-25T16:19:42.499 INFO:teuthology.orchestra.run.smithi173.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 672 kB/s | 77 kB 00:00 2024-04-25T16:19:42.524 INFO:teuthology.orchestra.run.smithi173.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 845 kB/s | 42 kB 00:00 2024-04-25T16:19:42.549 INFO:teuthology.orchestra.run.smithi173.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-25T16:19:42.554 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-25T16:19:42.574 INFO:teuthology.orchestra.run.smithi173.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-04-25T16:19:42.599 INFO:teuthology.orchestra.run.smithi173.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T16:19:42.616 INFO:teuthology.orchestra.run.smithi173.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 766 kB/s | 51 kB 00:00 2024-04-25T16:19:42.641 INFO:teuthology.orchestra.run.smithi173.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 805 kB/s | 54 kB 00:00 2024-04-25T16:19:42.675 INFO:teuthology.orchestra.run.smithi173.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 715 kB/s | 42 kB 00:00 2024-04-25T16:19:42.697 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-25T16:19:42.697 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-25T16:19:42.801 INFO:teuthology.orchestra.run.smithi173.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 282 kB/s | 44 kB 00:00 2024-04-25T16:19:42.976 INFO:teuthology.orchestra.run.smithi173.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-04-25T16:19:43.001 INFO:teuthology.orchestra.run.smithi173.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 683 kB/s | 221 kB 00:00 2024-04-25T16:19:43.026 INFO:teuthology.orchestra.run.smithi173.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 161 kB/s | 36 kB 00:00 2024-04-25T16:19:43.051 INFO:teuthology.orchestra.run.smithi173.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-04-25T16:19:43.077 INFO:teuthology.orchestra.run.smithi173.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-04-25T16:19:43.093 INFO:teuthology.orchestra.run.smithi173.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 871 kB/s | 58 kB 00:00 2024-04-25T16:19:43.106 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-04-25T16:19:43.119 INFO:teuthology.orchestra.run.smithi173.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-04-25T16:19:43.135 INFO:teuthology.orchestra.run.smithi173.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T16:19:43.152 INFO:teuthology.orchestra.run.smithi173.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 768 kB/s | 45 kB 00:00 2024-04-25T16:19:43.170 INFO:teuthology.orchestra.run.smithi173.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 667 kB/s | 22 kB 00:00 2024-04-25T16:19:43.187 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T16:19:43.194 INFO:teuthology.orchestra.run.smithi173.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-04-25T16:19:43.226 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T16:19:43.279 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T16:19:43.321 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T16:19:43.347 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T16:19:43.370 INFO:teuthology.orchestra.run.smithi173.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 111 kB/s | 19 kB 00:00 2024-04-25T16:19:43.381 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T16:19:43.529 INFO:teuthology.orchestra.run.smithi173.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 861 kB/s | 309 kB 00:00 2024-04-25T16:19:43.587 INFO:teuthology.orchestra.run.smithi173.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-04-25T16:19:43.604 INFO:teuthology.orchestra.run.smithi173.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-04-25T16:19:43.663 INFO:teuthology.orchestra.run.smithi173.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-04-25T16:19:43.688 INFO:teuthology.orchestra.run.smithi173.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 806 kB/s | 67 kB 00:00 2024-04-25T16:19:43.716 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T16:19:43.738 INFO:teuthology.orchestra.run.smithi173.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-04-25T16:19:43.832 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:19:43.855 INFO:teuthology.orchestra.run.smithi173.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 9.1 MB/s | 4.4 MB 00:00 2024-04-25T16:19:43.889 INFO:teuthology.orchestra.run.smithi173.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-04-25T16:19:43.914 INFO:teuthology.orchestra.run.smithi173.stdout:(114/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-04-25T16:19:43.939 INFO:teuthology.orchestra.run.smithi173.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-04-25T16:19:43.957 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:19:43.965 INFO:teuthology.orchestra.run.smithi173.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 425 kB/s | 32 kB 00:00 2024-04-25T16:19:43.991 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:19:43.998 INFO:teuthology.orchestra.run.smithi173.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 178 kB/s | 15 kB 00:00 2024-04-25T16:19:44.023 INFO:teuthology.orchestra.run.smithi173.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-25T16:19:44.049 INFO:teuthology.orchestra.run.smithi173.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-25T16:19:44.082 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:19:44.099 INFO:teuthology.orchestra.run.smithi173.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-25T16:19:44.120 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:19:44.124 INFO:teuthology.orchestra.run.smithi173.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-04-25T16:19:44.150 INFO:teuthology.orchestra.run.smithi173.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-25T16:19:44.217 INFO:teuthology.orchestra.run.smithi173.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-25T16:19:44.258 INFO:teuthology.orchestra.run.smithi173.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-04-25T16:19:44.266 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T16:19:44.275 INFO:teuthology.orchestra.run.smithi173.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 303 kB/s | 18 kB 00:00 2024-04-25T16:19:44.326 INFO:teuthology.orchestra.run.smithi173.stdout:(126/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-04-25T16:19:44.326 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T16:19:44.342 INFO:teuthology.orchestra.run.smithi173.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 276 kB/s | 23 kB 00:00 2024-04-25T16:19:44.346 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:19:44.359 INFO:teuthology.orchestra.run.smithi173.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 232 kB/s | 19 kB 00:00 2024-04-25T16:19:44.384 INFO:teuthology.orchestra.run.smithi173.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-04-25T16:19:44.410 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-25T16:19:44.411 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-25T16:19:44.418 INFO:teuthology.orchestra.run.smithi173.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 793 kB/s | 46 kB 00:00 2024-04-25T16:19:44.450 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:19:44.480 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:19:44.485 INFO:teuthology.orchestra.run.smithi173.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 787 kB/s | 79 kB 00:00 2024-04-25T16:19:44.510 INFO:teuthology.orchestra.run.smithi173.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 627 kB/s | 58 kB 00:00 2024-04-25T16:19:44.551 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:19:44.573 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:19:44.577 INFO:teuthology.orchestra.run.smithi173.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-04-25T16:19:44.597 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:19:44.644 INFO:teuthology.orchestra.run.smithi173.stdout:(134/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-04-25T16:19:44.678 INFO:teuthology.orchestra.run.smithi173.stdout:(135/152): python3-kubernetes-26.1.0-2.el9.noar 3.2 MB/s | 1.0 MB 00:00 2024-04-25T16:19:44.703 INFO:teuthology.orchestra.run.smithi173.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-04-25T16:19:44.762 INFO:teuthology.orchestra.run.smithi173.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-04-25T16:19:44.787 INFO:teuthology.orchestra.run.smithi173.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-04-25T16:19:44.832 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:19:44.837 INFO:teuthology.orchestra.run.smithi173.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-04-25T16:19:44.862 INFO:teuthology.orchestra.run.smithi173.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-04-25T16:19:44.865 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:19:44.913 INFO:teuthology.orchestra.run.smithi173.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2024-04-25T16:19:44.938 INFO:teuthology.orchestra.run.smithi173.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-04-25T16:19:44.955 INFO:teuthology.orchestra.run.smithi173.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-25T16:19:44.976 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T16:19:45.022 INFO:teuthology.orchestra.run.smithi173.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 300 kB/s | 20 kB 00:00 2024-04-25T16:19:45.047 INFO:teuthology.orchestra.run.smithi173.stdout:(145/152): python3-websocket-client-1.2.3-2.el9 670 kB/s | 90 kB 00:00 2024-04-25T16:19:45.072 INFO:teuthology.orchestra.run.smithi173.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-25T16:19:45.086 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T16:19:45.106 INFO:teuthology.orchestra.run.smithi173.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-25T16:19:45.131 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T16:19:45.166 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T16:19:45.181 INFO:teuthology.orchestra.run.smithi173.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T16:19:45.240 INFO:teuthology.orchestra.run.smithi173.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T16:19:45.241 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T16:19:45.281 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T16:19:45.315 INFO:teuthology.orchestra.run.smithi173.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 32 MB/s | 2.4 MB 00:00 2024-04-25T16:19:45.324 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T16:19:45.357 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T16:19:45.385 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T16:19:45.454 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T16:19:45.507 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T16:19:45.538 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T16:19:45.582 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T16:19:45.656 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T16:19:45.693 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T16:19:45.736 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T16:19:45.765 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T16:19:45.826 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T16:19:45.879 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T16:19:45.953 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T16:19:46.085 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T16:19:46.140 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T16:19:46.666 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T16:19:46.752 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T16:19:46.782 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T16:19:46.816 INFO:teuthology.orchestra.run.smithi173.stdout:(151/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 1.8 MB/s | 3.1 MB 00:01 2024-04-25T16:19:46.833 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T16:19:46.871 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T16:19:46.913 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T16:19:47.070 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T16:19:47.132 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T16:19:47.195 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T16:19:47.257 INFO:teuthology.orchestra.run.smithi173.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 1.5 MB/s | 3.3 MB 00:02 2024-04-25T16:19:47.260 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T16:19:47.261 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:19:47.262 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.4 MB/s | 214 MB 00:25 2024-04-25T16:19:47.291 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T16:19:47.328 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T16:19:47.374 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T16:19:47.419 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T16:19:47.462 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T16:19:47.505 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T16:19:47.639 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T16:19:47.684 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T16:19:47.715 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T16:19:47.756 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T16:19:47.850 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T16:19:47.890 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T16:19:47.922 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T16:19:47.947 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T16:19:47.972 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T16:19:48.046 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T16:19:48.173 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T16:19:48.205 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T16:19:48.233 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T16:19:48.267 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T16:19:48.306 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T16:19:48.338 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T16:19:48.370 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-04-25T16:19:48.376 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T16:19:48.411 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T16:19:48.444 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T16:19:48.485 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T16:19:48.515 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-04-25T16:19:48.515 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-04-25T16:19:48.520 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T16:19:48.606 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T16:19:48.838 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T16:19:49.066 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T16:19:49.223 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T16:19:49.322 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T16:19:49.376 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T16:19:50.090 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T16:19:50.147 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T16:19:50.181 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T16:19:50.259 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-04-25T16:19:50.259 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-04-25T16:19:50.280 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T16:19:50.339 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T16:19:50.420 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-25T16:19:50.427 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T16:19:50.467 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T16:19:50.511 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T16:19:50.559 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T16:19:50.612 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T16:19:50.644 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T16:19:50.646 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T16:19:50.672 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T16:19:50.673 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T16:19:50.705 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T16:19:50.750 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T16:19:51.048 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T16:19:51.181 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:19:51.306 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:19:51.332 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:19:51.390 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T16:19:51.423 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:19:51.453 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:19:51.505 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T16:19:51.607 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T16:19:51.650 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T16:19:51.678 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:19:51.767 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:19:51.796 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:19:51.842 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:19:51.872 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:19:51.894 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:19:52.089 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:19:52.131 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:19:52.239 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T16:19:52.339 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T16:19:52.398 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T16:19:52.433 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T16:19:52.507 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T16:19:52.557 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T16:19:52.601 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T16:19:52.633 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T16:19:52.661 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T16:19:52.719 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T16:19:52.724 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T16:19:52.801 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T16:19:52.811 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T16:19:52.831 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T16:19:52.839 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T16:19:52.872 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T16:19:52.875 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T16:19:52.970 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T16:19:53.002 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T16:19:53.046 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T16:19:53.074 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T16:19:53.145 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T16:19:53.192 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T16:19:53.197 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T16:19:53.227 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:19:53.273 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T16:19:53.326 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:19:53.357 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T16:19:53.393 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T16:19:53.410 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T16:19:53.467 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T16:19:53.881 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T16:19:53.905 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:19:53.960 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:19:53.981 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T16:19:54.006 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T16:19:54.248 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T16:19:54.283 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T16:19:54.326 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T16:19:54.364 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T16:19:54.406 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T16:19:54.555 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T16:19:54.601 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T16:19:54.657 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T16:19:54.730 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T16:19:54.760 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T16:19:54.797 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T16:19:54.844 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T16:19:54.880 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T16:19:54.923 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T16:19:54.965 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T16:19:55.103 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T16:19:55.163 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T16:19:55.193 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T16:19:55.235 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T16:19:55.330 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T16:19:55.368 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T16:19:55.408 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T16:19:55.434 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T16:19:55.458 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T16:19:55.535 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T16:19:55.663 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T16:19:55.700 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T16:19:55.728 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T16:19:55.762 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T16:19:55.802 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T16:19:55.833 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T16:19:55.863 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T16:19:55.898 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T16:19:55.931 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T16:19:55.981 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T16:19:56.016 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T16:19:56.104 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T16:19:56.146 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-04-25T16:19:56.147 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T16:19:56.178 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T16:19:56.217 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T16:19:56.241 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:19:56.244 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T16:19:56.279 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:19:56.282 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T16:19:56.316 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T16:19:56.344 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T16:19:56.372 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T16:19:56.377 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T16:19:56.403 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T16:19:56.446 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T16:19:56.772 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T16:19:56.835 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:19:56.891 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:19:56.896 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:19:56.971 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:19:56.972 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T16:19:56.972 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T16:19:56.972 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:19:56.992 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:19:57.022 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:19:57.055 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:19:57.110 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T16:19:57.147 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:19:57.160 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T16:19:57.184 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:19:57.194 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T16:19:57.296 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T16:19:57.330 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T16:19:57.361 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T16:19:57.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T16:19:57.401 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:19:57.451 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T16:19:57.497 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T16:19:57.531 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:19:57.560 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:19:57.615 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T16:19:57.628 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:19:57.648 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T16:19:57.676 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T16:19:57.703 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T16:19:57.782 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T16:19:57.837 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:19:57.870 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T16:19:57.981 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T16:19:58.091 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T16:19:58.153 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T16:19:58.187 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T16:19:58.260 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T16:19:58.294 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T16:19:58.337 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T16:19:58.369 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T16:19:58.398 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T16:19:58.439 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T16:19:58.466 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T16:19:58.519 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T16:19:58.550 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T16:19:58.554 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T16:19:58.594 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T16:19:58.669 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T16:19:58.705 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T16:19:58.748 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T16:19:58.777 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T16:19:58.846 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T16:19:58.900 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T16:19:58.974 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T16:19:59.107 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T16:19:59.160 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T16:19:59.671 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T16:19:59.755 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T16:19:59.785 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T16:19:59.800 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T16:19:59.827 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T16:19:59.866 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T16:19:59.893 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T16:19:59.907 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T16:19:59.921 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T16:19:59.954 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T16:20:00.055 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T16:20:00.102 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T16:20:00.156 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T16:20:00.229 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T16:20:00.261 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T16:20:00.277 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T16:20:00.297 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T16:20:00.344 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T16:20:00.380 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T16:20:00.384 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:20:00.422 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T16:20:00.464 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T16:20:00.475 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:20:00.506 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T16:20:00.542 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T16:20:00.609 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T16:20:00.662 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T16:20:00.692 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T16:20:00.733 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T16:20:00.827 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T16:20:00.867 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T16:20:00.907 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T16:20:00.932 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T16:20:00.957 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T16:20:01.029 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T16:20:01.032 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T16:20:01.054 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:20:01.125 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:20:01.156 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T16:20:01.159 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T16:20:01.198 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T16:20:01.225 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T16:20:01.259 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T16:20:01.298 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T16:20:01.339 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T16:20:01.368 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T16:20:01.412 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T16:20:01.445 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T16:20:01.486 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T16:20:01.521 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T16:20:01.606 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T16:20:01.651 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T16:20:01.682 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T16:20:01.729 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T16:20:01.827 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T16:20:01.874 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T16:20:02.583 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T16:20:02.622 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T16:20:02.656 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T16:20:02.753 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T16:20:02.805 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T16:20:02.891 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T16:20:02.958 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T16:20:03.136 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T16:20:03.172 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T16:20:03.250 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T16:20:03.437 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:20:03.470 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:20:03.880 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T16:20:03.986 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T16:20:04.045 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:20:04.082 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:20:04.173 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:20:04.173 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T16:20:04.173 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T16:20:04.174 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:04.208 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:20:05.216 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T16:20:05.300 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T16:20:05.328 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T16:20:05.361 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T16:20:05.681 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T16:20:05.724 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:20:05.815 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T16:20:05.846 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T16:20:05.882 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T16:20:06.377 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T16:20:06.410 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:20:06.465 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T16:20:06.487 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T16:20:08.652 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:20:08.652 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /sys 2024-04-25T16:20:08.652 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /proc 2024-04-25T16:20:08.652 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /mnt 2024-04-25T16:20:08.652 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /var/tmp 2024-04-25T16:20:08.652 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /home 2024-04-25T16:20:08.652 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /root 2024-04-25T16:20:08.652 INFO:teuthology.orchestra.run.smithi098.stdout:skipping the directory /tmp 2024-04-25T16:20:08.653 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:08.728 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:20:08.758 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:20:08.822 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:20:08.837 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:20:08.838 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:08.838 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T16:20:08.838 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:09.320 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T16:20:09.362 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:20:09.438 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T16:20:09.438 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T16:20:09.438 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T16:20:09.438 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:09.471 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:20:10.425 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:20:10.451 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:20:10.451 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:10.451 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T16:20:10.451 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:20:10.451 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:20:10.452 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:10.685 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:20:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:20:10.708 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:10.708 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T16:20:10.708 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:20:10.708 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:20:10.708 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:11.169 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:20:11.191 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:20:11.191 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:11.192 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T16:20:11.192 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:20:11.192 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:20:11.192 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:11.291 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T16:20:11.339 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T16:20:11.395 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T16:20:11.578 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T16:20:11.628 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T16:20:11.705 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T16:20:11.795 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:20:11.925 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:20:11.962 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T16:20:11.994 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T16:20:12.021 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T16:20:12.424 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:20:12.454 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:20:12.486 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T16:20:12.548 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T16:20:12.585 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:20:13.619 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:20:13.654 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:20:14.616 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:20:14.646 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:20:14.748 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:20:14.911 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T16:20:14.939 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:20:14.962 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:20:14.962 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:14.963 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T16:20:14.963 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:20:14.963 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:20:14.963 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:15.007 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:20:15.018 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:20:15.116 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T16:20:15.163 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T16:20:15.917 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:20:15.917 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-04-25T16:20:15.917 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-04-25T16:20:15.917 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-04-25T16:20:15.917 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-04-25T16:20:15.917 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-04-25T16:20:15.917 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-04-25T16:20:15.917 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-04-25T16:20:15.918 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:16.090 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:20:16.099 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T16:20:16.106 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:20:16.106 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:16.106 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T16:20:16.106 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:16.138 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:20:16.157 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:20:16.157 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:16.157 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T16:20:16.157 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:20:16.157 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:20:16.157 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:16.202 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:20:16.224 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:20:16.224 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:16.224 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T16:20:16.224 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:16.269 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:20:16.292 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:20:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T16:20:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:20:16.292 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:20:16.292 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:16.612 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:20:16.634 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:20:16.634 INFO:teuthology.orchestra.run.smithi098.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:16.634 INFO:teuthology.orchestra.run.smithi098.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T16:20:16.634 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:20:16.634 INFO:teuthology.orchestra.run.smithi098.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:20:16.634 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:17.679 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:20:17.705 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:20:17.705 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:17.705 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T16:20:17.705 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:20:17.705 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:20:17.705 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:17.930 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:20:17.952 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:20:17.952 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:17.952 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T16:20:17.952 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:20:17.952 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:20:17.953 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:18.413 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:20:18.436 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:20:18.436 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:18.436 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T16:20:18.436 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:20:18.436 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:20:18.436 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:18.561 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T16:20:18.618 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T16:20:18.674 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T16:20:18.865 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T16:20:18.915 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T16:20:18.995 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T16:20:19.092 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:20:19.234 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:20:19.274 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T16:20:19.307 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T16:20:19.334 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T16:20:19.745 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:20:19.775 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:20:19.807 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T16:20:19.869 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T16:20:19.906 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:20:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T16:20:20.956 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:20:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T16:20:20.991 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:20:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T16:20:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T16:20:21.074 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-04-25T16:20:21.133 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T16:20:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T16:20:21.298 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:20:21.314 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T16:20:21.314 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:21.314 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T16:20:21.314 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T16:20:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:21.962 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:20:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:21.992 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:20:22.096 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:20:22.261 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T16:20:22.303 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:20:22.326 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:20:22.326 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:22.326 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T16:20:22.326 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:20:22.326 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:20:22.327 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:22.378 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:20:22.389 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:20:22.489 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T16:20:22.544 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T16:20:22.818 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T16:20:22.884 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:20:22.905 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T16:20:22.910 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T16:20:22.910 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:22.910 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T16:20:22.910 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:20:22.910 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T16:20:22.910 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:22.946 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T16:20:23.036 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T16:20:23.076 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T16:20:23.077 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:20:23.144 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:20:23.161 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:20:23.169 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T16:20:23.169 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:23.169 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T16:20:23.170 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:20:23.170 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T16:20:23.170 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:23.193 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:20:23.194 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:20:23.281 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:20:23.350 INFO:teuthology.orchestra.run.smithi098.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:23.479 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T16:20:23.506 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:20:23.529 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:20:23.529 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:23.529 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T16:20:23.529 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:20:23.529 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:20:23.530 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:23.573 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:20:23.596 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:20:23.596 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:23.596 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T16:20:23.596 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:23.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:20:23.641 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:20:23.659 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T16:20:23.659 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:23.659 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T16:20:23.660 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:20:23.660 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T16:20:23.660 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:23.663 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:20:23.663 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:23.663 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T16:20:23.663 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:20:23.663 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:20:23.664 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:23.767 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T16:20:23.823 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T16:20:23.880 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T16:20:23.983 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:20:24.006 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:20:24.006 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:24.006 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T16:20:24.006 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:20:24.006 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:20:24.007 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:24.064 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T16:20:24.112 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T16:20:24.190 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T16:20:24.278 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:20:24.409 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:20:24.446 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T16:20:24.477 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T16:20:24.505 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T16:20:24.899 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:20:24.928 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T16:20:24.961 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T16:20:25.039 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T16:20:25.077 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:20:26.111 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T16:20:26.145 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:20:27.104 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T16:20:27.129 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:20:27.230 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T16:20:27.386 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T16:20:27.431 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:20:27.455 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T16:20:27.455 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:27.455 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T16:20:27.455 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:20:27.455 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T16:20:27.455 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:27.498 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:20:27.509 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T16:20:27.608 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T16:20:27.663 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T16:20:28.280 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T16:20:28.321 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T16:20:28.362 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T16:20:28.394 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T16:20:28.421 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T16:20:28.599 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T16:20:28.611 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T16:20:28.626 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:20:28.649 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T16:20:28.649 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:28.649 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T16:20:28.649 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:20:28.650 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T16:20:28.650 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:28.669 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T16:20:28.693 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:20:28.699 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T16:20:28.715 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T16:20:28.715 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:28.715 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T16:20:28.715 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:28.760 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:20:28.783 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T16:20:28.783 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:28.783 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T16:20:28.783 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:20:28.783 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T16:20:28.783 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:29.094 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:20:29.117 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T16:20:29.117 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T16:20:29.117 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T16:20:29.117 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:20:29.117 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T16:20:29.117 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:29.278 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:29.322 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:29.342 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:20:29.602 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T16:20:29.603 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T16:20:29.604 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T16:20:29.604 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T16:20:29.604 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T16:20:29.604 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T16:20:29.616 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T16:20:29.617 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T16:20:29.618 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T16:20:29.619 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T16:20:29.620 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T16:20:29.621 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:20:29.622 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T16:20:29.623 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T16:20:29.624 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:20:29.625 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T16:20:30.190 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T16:20:30.285 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T16:20:30.326 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T16:20:30.418 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T16:20:30.455 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T16:20:30.455 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:20:30.539 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:20:30.572 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:20:30.572 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:20:30.660 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:20:30.729 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:31.698 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout:Upgraded: 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T16:20:31.699 INFO:teuthology.orchestra.run.smithi098.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.700 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:31.701 INFO:teuthology.orchestra.run.smithi098.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.702 INFO:teuthology.orchestra.run.smithi098.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.703 INFO:teuthology.orchestra.run.smithi098.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T16:20:31.704 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T16:20:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T16:20:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T16:20:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T16:20:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T16:20:31.705 INFO:teuthology.orchestra.run.smithi098.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T16:20:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T16:20:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T16:20:31.706 INFO:teuthology.orchestra.run.smithi098.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T16:20:31.707 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T16:20:31.708 INFO:teuthology.orchestra.run.smithi098.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T16:20:31.709 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T16:20:31.710 INFO:teuthology.orchestra.run.smithi098.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.711 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T16:20:31.712 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:20:31.713 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-04-25T16:20:32.077 DEBUG:teuthology.parallel:result is None 2024-04-25T16:20:33.382 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T16:20:33.423 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T16:20:33.463 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T16:20:33.496 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T16:20:33.523 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T16:20:33.704 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T16:20:33.752 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T16:20:33.782 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T16:20:34.360 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:34.397 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:34.416 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T16:20:35.233 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T16:20:35.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T16:20:35.375 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T16:20:35.455 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T16:20:35.488 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T16:20:35.488 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:20:35.555 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T16:20:35.596 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:20:35.596 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:20:35.700 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T16:20:35.770 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:36.837 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:36.837 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T16:20:36.837 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T16:20:36.838 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T16:20:36.839 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T16:20:36.851 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T16:20:36.851 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T16:20:36.851 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T16:20:36.851 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T16:20:36.852 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T16:20:36.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T16:20:36.854 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T16:20:36.855 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T16:20:36.856 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T16:20:36.857 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T16:20:36.858 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T16:20:36.859 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T16:20:36.860 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:20:36.861 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T16:20:38.940 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:38.940 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:38.940 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.941 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.942 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T16:20:38.943 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T16:20:38.944 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T16:20:38.945 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T16:20:38.946 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T16:20:38.947 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T16:20:38.948 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T16:20:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T16:20:38.950 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.951 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:20:38.952 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-25T16:20:39.352 DEBUG:teuthology.parallel:result is None 2024-04-25T16:20:41.623 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:41.623 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T16:20:41.623 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T16:20:41.623 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T16:20:41.623 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T16:20:41.623 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T16:20:41.623 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T16:20:41.623 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T16:20:41.624 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T16:20:41.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T16:20:41.636 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T16:20:41.663 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T16:20:41.664 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T16:20:41.665 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T16:20:41.666 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T16:20:41.667 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T16:20:41.668 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T16:20:41.669 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T16:20:41.670 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T16:20:41.671 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T16:20:41.672 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T16:20:41.672 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T16:20:41.672 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T16:20:41.672 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T16:20:41.672 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T16:20:43.712 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T16:20:43.713 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.714 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T16:20:43.715 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T16:20:43.716 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T16:20:43.717 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T16:20:43.718 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T16:20:43.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T16:20:43.720 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T16:20:43.721 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T16:20:43.722 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.723 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-04-25T16:20:43.724 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-04-25T16:20:44.073 DEBUG:teuthology.parallel:result is None 2024-04-25T16:20:44.074 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:20:44.217 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T16:20:44.244 INFO:teuthology.orchestra.run.smithi098.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T16:20:44.244 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T16:20:44.244 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T16:20:44.246 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:20:44.462 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T16:20:44.483 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T16:20:44.483 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T16:20:44.483 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T16:20:44.485 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:20:44.742 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T16:20:44.767 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T16:20:44.767 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T16:20:44.767 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T16:20:44.769 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T16:20:44.769 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:20:44.769 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T16:20:44.799 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:20:44.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T16:20:44.827 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:20:44.827 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T16:20:44.857 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T16:20:44.858 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:20:44.858 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T16:20:44.885 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T16:20:44.952 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:20:44.952 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T16:20:44.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T16:20:45.042 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:20:45.042 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T16:20:45.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T16:20:45.138 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T16:20:45.138 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:20:45.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T16:20:45.165 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T16:20:45.229 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:20:45.230 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T16:20:45.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T16:20:45.318 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:20:45.319 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T16:20:45.348 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T16:20:45.414 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T16:20:45.415 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:20:45.415 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T16:20:45.439 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T16:20:45.504 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:20:45.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T16:20:45.528 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T16:20:45.592 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:20:45.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T16:20:45.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T16:20:45.685 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T16:20:45.790 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-25T16:20:45.790 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:20:45.791 INFO:tasks.cephadm:Cluster fsid is c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:20:45.791 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T16:20:45.791 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.b': '172.21.15.173', 'mon.c': '172.21.15.203'} 2024-04-25T16:20:45.791 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-04-25T16:20:45.791 INFO:tasks.cephadm:First mgr is x 2024-04-25T16:20:45.792 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T16:20:45.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-04-25T16:20:45.817 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-04-25T16:20:45.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-04-25T16:20:45.868 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T16:20:45.868 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:20:46.010 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78872/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3291-g994e2f20', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3291.g994e2f20'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-25 04:02:18.990013', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240425.015853-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-25T16:20:46.226 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-pdonnell-testing-20240425.015853-debug, sha1 994e2f20102a20fa9a27966c14b625cc01884df8 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:20:46.228 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm 2024-04-25T16:20:46.228 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:47.321 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 16:20 /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:47.321 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:48.134 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 16:20 /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:48.134 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:48.911 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 16:20 /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:48.912 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:48.931 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:48.951 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T16:20:48.985 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 on all hosts... 2024-04-25T16:20:48.986 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T16:20:48.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T16:20:48.993 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T16:20:50.491 INFO:teuthology.orchestra.run.smithi203.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:20:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:20:50.640 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:21:30.558 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T16:21:30.558 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T16:21:30.558 INFO:teuthology.orchestra.run.smithi203.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T16:21:30.559 INFO:teuthology.orchestra.run.smithi203.stdout: "repo_digests": [ 2024-04-25T16:21:30.559 INFO:teuthology.orchestra.run.smithi203.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T16:21:30.559 INFO:teuthology.orchestra.run.smithi203.stdout: ] 2024-04-25T16:21:30.559 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T16:21:58.818 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:21:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T16:21:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T16:21:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-04-25T16:21:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T16:21:58.819 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:21:58.819 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:22:00.958 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T16:22:00.958 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T16:22:00.958 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T16:22:00.958 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-04-25T16:22:00.958 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T16:22:00.958 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-04-25T16:22:00.958 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T16:22:00.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-04-25T16:22:01.014 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-04-25T16:22:01.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-04-25T16:22:01.073 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-04-25T16:22:01.101 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-04-25T16:22:01.125 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-04-25T16:22:01.152 INFO:tasks.cephadm:Writing seed config... 2024-04-25T16:22:01.153 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T16:22:01.153 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T16:22:01.153 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T16:22:01.154 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T16:22:01.155 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T16:22:01.156 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-04-25T16:22:01.157 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:22:01.157 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T16:22:01.174 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c520eaa8-031f-11ef-bc93-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T16:22:01.175 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T16:22:01.218 DEBUG:teuthology.orchestra.run.smithi098:mgr.x> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T16:22:01.261 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T16:22:01.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 -v bootstrap --fsid c520eaa8-031f-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:22:01.454 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-04-25T16:22:01.454 INFO:teuthology.orchestra.run.smithi098.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8', '-v', 'bootstrap', '--fsid', 'c520eaa8-031f-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-04-25T16:22:01.454 INFO:teuthology.orchestra.run.smithi098.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T16:22:01.455 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying podman|docker is present... 2024-04-25T16:22:01.481 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T16:22:01.481 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying lvm2 is present... 2024-04-25T16:22:01.481 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying time synchronization is in place... 2024-04-25T16:22:01.488 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T16:22:01.488 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T16:22:01.494 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T16:22:01.494 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T16:22:01.501 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-04-25T16:22:01.506 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-04-25T16:22:01.507 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-04-25T16:22:01.507 INFO:teuthology.orchestra.run.smithi098.stdout:Repeating the final host check... 2024-04-25T16:22:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T16:22:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T16:22:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl is present 2024-04-25T16:22:01.531 INFO:teuthology.orchestra.run.smithi098.stdout:lvcreate is present 2024-04-25T16:22:01.537 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T16:22:01.537 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T16:22:01.543 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T16:22:01.543 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T16:22:01.549 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout enabled 2024-04-25T16:22:01.555 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout active 2024-04-25T16:22:01.555 INFO:teuthology.orchestra.run.smithi098.stdout:Unit chronyd.service is enabled and running 2024-04-25T16:22:01.555 INFO:teuthology.orchestra.run.smithi098.stdout:Host looks OK 2024-04-25T16:22:01.555 INFO:teuthology.orchestra.run.smithi098.stdout:Cluster fsid: c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:01.555 INFO:teuthology.orchestra.run.smithi098.stdout:Acquiring lock 140033596243104 on /run/cephadm/c520eaa8-031f-11ef-bc93-c7b262605968.lock 2024-04-25T16:22:01.555 INFO:teuthology.orchestra.run.smithi098.stdout:Lock 140033596243104 acquired on /run/cephadm/c520eaa8-031f-11ef-bc93-c7b262605968.lock 2024-04-25T16:22:01.556 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 3300 ... 2024-04-25T16:22:01.556 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying IP 172.21.15.98 port 6789 ... 2024-04-25T16:22:01.556 INFO:teuthology.orchestra.run.smithi098.stdout:Base mon IP(s) is [172.21.15.98:3300, 172.21.15.98:6789], mon addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-04-25T16:22:01.559 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.98 metric 100 2024-04-25T16:22:01.559 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-04-25T16:22:01.562 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T16:22:01.562 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-25T16:22:01.562 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-25T16:22:01.564 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T16:22:01.564 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T16:22:01.564 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T16:22:01.564 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-25T16:22:01.564 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout inet6 fe80::f29f:9dff:ddbc:4615/64 scope link noprefixroute 2024-04-25T16:22:01.564 INFO:teuthology.orchestra.run.smithi098.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T16:22:01.565 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-04-25T16:22:01.565 INFO:teuthology.orchestra.run.smithi098.stdout:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-04-25T16:22:01.565 INFO:teuthology.orchestra.run.smithi098.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T16:22:01.565 INFO:teuthology.orchestra.run.smithi098.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T16:22:01.566 INFO:teuthology.orchestra.run.smithi098.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:22:02.012 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stdout d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T16:22:02.012 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T16:22:02.012 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T16:22:02.012 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:665339d96aad80d083ba9988a62422bcef0a42210fb2b28340e33ab98589f7b8 2024-04-25T16:22:02.012 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T16:22:02.012 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Copying config sha256:d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T16:22:02.012 INFO:teuthology.orchestra.run.smithi098.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T16:22:02.916 INFO:teuthology.orchestra.run.smithi098.stdout:ceph: stdout ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:22:02.916 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph version: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:22:02.917 INFO:teuthology.orchestra.run.smithi098.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T16:22:03.764 INFO:teuthology.orchestra.run.smithi098.stdout:stat: stdout 167 167 2024-04-25T16:22:03.765 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial keys... 2024-04-25T16:22:04.592 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQAsgypmRsX/AhAA5dwugXb05NX9WjxqsmZP3w== 2024-04-25T16:22:05.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQAsgypmKEPTNRAAltPsP28G2WbH7HQ8oheJ8A== 2024-04-25T16:22:06.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph-authtool: stdout AQAtgypmwLl2LBAAO0RbaZK9eh27gu7jzWSWag== 2024-04-25T16:22:06.306 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial monmap... 2024-04-25T16:22:07.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T16:22:07.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T16:22:07.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:07.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T16:22:07.161 INFO:teuthology.orchestra.run.smithi098.stdout:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T16:22:07.161 INFO:teuthology.orchestra.run.smithi098.stdout:setting min_mon_release = quincy 2024-04-25T16:22:07.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: set fsid to c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:07.161 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T16:22:07.162 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:22:07.162 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mon... 2024-04-25T16:22:08.010 INFO:teuthology.orchestra.run.smithi098.stdout:create mon.a on 2024-04-25T16:22:08.223 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T16:22:08.367 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T16:22:08.548 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c520eaa8-031f-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-c520eaa8-031f-11ef-bc93-c7b262605968.target. 2024-04-25T16:22:08.548 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c520eaa8-031f-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-c520eaa8-031f-11ef-bc93-c7b262605968.target. 2024-04-25T16:22:08.761 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.a 2024-04-25T16:22:08.762 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T16:22:08.926 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c520eaa8-031f-11ef-bc93-c7b262605968.target.wants/ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-c520eaa8-031f-11ef-bc93-c7b262605968@.service. 2024-04-25T16:22:09.492 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T16:22:09.492 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-25T16:22:09.498 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T16:22:09.498 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T16:22:09.499 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-25T16:22:09.499 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T16:22:09.499 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon to start... 2024-04-25T16:22:09.499 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mon... 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout id: c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout services: 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.521885s) 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:10.503 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout data: 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.890+0000 7fd36a880700 1 Processor -- start 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.890+0000 7fd36a880700 1 -- start start 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.890+0000 7fd36a880700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd364108c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.890+0000 7fd36a880700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd364109220 con 0x7fd364108820 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.891+0000 7fd363fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd364108c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.891+0000 7fd363fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd364108c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46482/0 (socket says 172.21.15.98:46482) 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.891+0000 7fd363fff700 1 -- 172.21.15.98:0/3843965464 learned_addr learned my addr 172.21.15.98:0/3843965464 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:10.504 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.891+0000 7fd363fff700 1 -- 172.21.15.98:0/3843965464 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd364109950 con 0x7fd364108820 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.891+0000 7fd363fff700 1 --2- 172.21.15.98:0/3843965464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd364108c30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd3540098a0 tx=0x7fd354009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1de59feb257094c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd362ffd700 1 -- 172.21.15.98:0/3843965464 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd354004160 con 0x7fd364108820 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd362ffd700 1 -- 172.21.15.98:0/3843965464 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd354004300 con 0x7fd364108820 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd362ffd700 1 -- 172.21.15.98:0/3843965464 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3540036a0 con 0x7fd364108820 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd36a880700 1 -- 172.21.15.98:0/3843965464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 msgr2=0x7fd364108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd36a880700 1 --2- 172.21.15.98:0/3843965464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd364108c30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd3540098a0 tx=0x7fd354009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd36a880700 1 -- 172.21.15.98:0/3843965464 shutdown_connections 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd36a880700 1 --2- 172.21.15.98:0/3843965464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd364108c30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd36a880700 1 -- 172.21.15.98:0/3843965464 >> 172.21.15.98:0/3843965464 conn(0x7fd364103f20 msgr2=0x7fd364106370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd36a880700 1 -- 172.21.15.98:0/3843965464 shutdown_connections 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.892+0000 7fd36a880700 1 -- 172.21.15.98:0/3843965464 wait complete. 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.893+0000 7fd36a880700 1 Processor -- start 2024-04-25T16:22:10.505 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.893+0000 7fd36a880700 1 -- start start 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.893+0000 7fd36a880700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd36419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.893+0000 7fd36a880700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd364109e80 con 0x7fd364108820 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.893+0000 7fd363fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd36419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.893+0000 7fd363fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd36419d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46496/0 (socket says 172.21.15.98:46496) 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.893+0000 7fd363fff700 1 -- 172.21.15.98:0/3432152529 learned_addr learned my addr 172.21.15.98:0/3432152529 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd363fff700 1 -- 172.21.15.98:0/3432152529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36419d880 con 0x7fd364108820 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd363fff700 1 --2- 172.21.15.98:0/3432152529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd36419d340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd354000c00 tx=0x7fd354004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd3617fa700 1 -- 172.21.15.98:0/3432152529 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd354003db0 con 0x7fd364108820 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd3617fa700 1 -- 172.21.15.98:0/3432152529 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd354011630 con 0x7fd364108820 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36419db10 con 0x7fd364108820 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd3617fa700 1 -- 172.21.15.98:0/3432152529 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd354011950 con 0x7fd364108820 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3641a0720 con 0x7fd364108820 2024-04-25T16:22:10.506 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd3617fa700 1 -- 172.21.15.98:0/3432152529 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd35401a460 con 0x7fd364108820 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.894+0000 7fd3617fa700 1 -- 172.21.15.98:0/3432152529 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd35401b780 con 0x7fd364108820 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.895+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd364064cf0 con 0x7fd364108820 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.898+0000 7fd3617fa700 1 -- 172.21.15.98:0/3432152529 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd354031020 con 0x7fd364108820 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.967+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fd36402e780 con 0x7fd364108820 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.968+0000 7fd3617fa700 1 -- 172.21.15.98:0/3432152529 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fd35401b9d0 con 0x7fd364108820 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.968+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 msgr2=0x7fd36419d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.968+0000 7fd36a880700 1 --2- 172.21.15.98:0/3432152529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd36419d340 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd354000c00 tx=0x7fd354004750 comp rx=0 tx=0).stop 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.969+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 shutdown_connections 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.969+0000 7fd36a880700 1 --2- 172.21.15.98:0/3432152529 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd364108820 0x7fd36419d340 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.969+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 >> 172.21.15.98:0/3432152529 conn(0x7fd364103f20 msgr2=0x7fd364193aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.969+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 shutdown_connections 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:09.969+0000 7fd36a880700 1 -- 172.21.15.98:0/3432152529 wait complete. 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:mon is available 2024-04-25T16:22:10.507 INFO:teuthology.orchestra.run.smithi098.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T16:22:11.515 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T16:22:11.516 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.923+0000 7fed79174700 1 Processor -- start 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.923+0000 7fed79174700 1 -- start start 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.923+0000 7fed79174700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7410c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.923+0000 7fed79174700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed7410cba0 con 0x7fed7407d4e0 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.923+0000 7fed72d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7410c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.923+0000 7fed72d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7410c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46508/0 (socket says 172.21.15.98:46508) 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.923+0000 7fed72d9d700 1 -- 172.21.15.98:0/896671019 learned_addr learned my addr 172.21.15.98:0/896671019 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.923+0000 7fed72d9d700 1 -- 172.21.15.98:0/896671019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7410cd20 con 0x7fed7407d4e0 2024-04-25T16:22:11.517 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.924+0000 7fed72d9d700 1 --2- 172.21.15.98:0/896671019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7410c5d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fed640098a0 tx=0x7fed64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c01229b2d22d1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.924+0000 7fed71d9b700 1 -- 172.21.15.98:0/896671019 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed64004160 con 0x7fed7407d4e0 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.924+0000 7fed71d9b700 1 -- 172.21.15.98:0/896671019 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed64004300 con 0x7fed7407d4e0 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.924+0000 7fed71d9b700 1 -- 172.21.15.98:0/896671019 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed640036a0 con 0x7fed7407d4e0 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.924+0000 7fed79174700 1 -- 172.21.15.98:0/896671019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 msgr2=0x7fed7410c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.924+0000 7fed79174700 1 --2- 172.21.15.98:0/896671019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7410c5d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fed640098a0 tx=0x7fed64009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.925+0000 7fed79174700 1 -- 172.21.15.98:0/896671019 shutdown_connections 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.925+0000 7fed79174700 1 --2- 172.21.15.98:0/896671019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7410c5d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.925+0000 7fed79174700 1 -- 172.21.15.98:0/896671019 >> 172.21.15.98:0/896671019 conn(0x7fed74103fe0 msgr2=0x7fed74106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.925+0000 7fed79174700 1 -- 172.21.15.98:0/896671019 shutdown_connections 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.925+0000 7fed79174700 1 -- 172.21.15.98:0/896671019 wait complete. 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.925+0000 7fed79174700 1 Processor -- start 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed79174700 1 -- start start 2024-04-25T16:22:11.518 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed79174700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7419d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed79174700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed7410d1f0 con 0x7fed7407d4e0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed72d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7419d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed72d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7419d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46522/0 (socket says 172.21.15.98:46522) 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed72d9d700 1 -- 172.21.15.98:0/2580080006 learned_addr learned my addr 172.21.15.98:0/2580080006 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed72d9d700 1 -- 172.21.15.98:0/2580080006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7419d8b0 con 0x7fed7407d4e0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed72d9d700 1 --2- 172.21.15.98:0/2580080006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7419d370 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fed64000c00 tx=0x7fed64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed5bfff700 1 -- 172.21.15.98:0/2580080006 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed64003db0 con 0x7fed7407d4e0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed5bfff700 1 -- 172.21.15.98:0/2580080006 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed64011630 con 0x7fed7407d4e0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed5bfff700 1 -- 172.21.15.98:0/2580080006 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed64011950 con 0x7fed7407d4e0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.926+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed7419db40 con 0x7fed7407d4e0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.927+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed741a0750 con 0x7fed7407d4e0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.927+0000 7fed5bfff700 1 -- 172.21.15.98:0/2580080006 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fed6401a460 con 0x7fed7407d4e0 2024-04-25T16:22:11.519 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.927+0000 7fed5bfff700 1 -- 172.21.15.98:0/2580080006 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fed6401b780 con 0x7fed7407d4e0 2024-04-25T16:22:11.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.928+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed74064cf0 con 0x7fed7407d4e0 2024-04-25T16:22:11.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.930+0000 7fed5bfff700 1 -- 172.21.15.98:0/2580080006 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fed64031020 con 0x7fed7407d4e0 2024-04-25T16:22:11.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:10.997+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fed7402e7e0 con 0x7fed7407d4e0 2024-04-25T16:22:11.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.001+0000 7fed5bfff700 1 -- 172.21.15.98:0/2580080006 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7fed64004260 con 0x7fed7407d4e0 2024-04-25T16:22:11.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.002+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 msgr2=0x7fed7419d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:11.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.002+0000 7fed79174700 1 --2- 172.21.15.98:0/2580080006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7419d370 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fed64000c00 tx=0x7fed64004750 comp rx=0 tx=0).stop 2024-04-25T16:22:11.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.002+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 shutdown_connections 2024-04-25T16:22:11.520 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.002+0000 7fed79174700 1 --2- 172.21.15.98:0/2580080006 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed7407d4e0 0x7fed7419d370 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:11.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.002+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 >> 172.21.15.98:0/2580080006 conn(0x7fed74103fe0 msgr2=0x7fed74105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:11.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.002+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 shutdown_connections 2024-04-25T16:22:11.521 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.002+0000 7fed79174700 1 -- 172.21.15.98:0/2580080006 wait complete. 2024-04-25T16:22:11.521 INFO:teuthology.orchestra.run.smithi098.stdout:Generating new minimal ceph.conf... 2024-04-25T16:22:12.655 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.958+0000 7f1e08878700 1 Processor -- start 2024-04-25T16:22:12.655 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.958+0000 7f1e08878700 1 -- start start 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.958+0000 7f1e08878700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0410c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.958+0000 7f1e08878700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0410cb40 con 0x7f1e0407d4e0 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.958+0000 7f1e0259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0410c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.958+0000 7f1e0259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0410c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46528/0 (socket says 172.21.15.98:46528) 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.958+0000 7f1e0259c700 1 -- 172.21.15.98:0/2990944911 learned_addr learned my addr 172.21.15.98:0/2990944911 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.958+0000 7f1e0259c700 1 -- 172.21.15.98:0/2990944911 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0410ccc0 con 0x7f1e0407d4e0 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.959+0000 7f1e0259c700 1 --2- 172.21.15.98:0/2990944911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0410c570 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1df40098a0 tx=0x7f1df4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a9dcd6911d575d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.959+0000 7f1e0159a700 1 -- 172.21.15.98:0/2990944911 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1df4004160 con 0x7f1e0407d4e0 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.959+0000 7f1e0159a700 1 -- 172.21.15.98:0/2990944911 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f1df4004300 con 0x7f1e0407d4e0 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.959+0000 7f1e0159a700 1 -- 172.21.15.98:0/2990944911 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1df40119e0 con 0x7f1e0407d4e0 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.959+0000 7f1e08878700 1 -- 172.21.15.98:0/2990944911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 msgr2=0x7f1e0410c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:12.656 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.959+0000 7f1e08878700 1 --2- 172.21.15.98:0/2990944911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0410c570 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1df40098a0 tx=0x7f1df4009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.960+0000 7f1e08878700 1 -- 172.21.15.98:0/2990944911 shutdown_connections 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.960+0000 7f1e08878700 1 --2- 172.21.15.98:0/2990944911 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0410c570 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.960+0000 7f1e08878700 1 -- 172.21.15.98:0/2990944911 >> 172.21.15.98:0/2990944911 conn(0x7f1e04103f40 msgr2=0x7f1e04106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.960+0000 7f1e08878700 1 -- 172.21.15.98:0/2990944911 shutdown_connections 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.960+0000 7f1e08878700 1 -- 172.21.15.98:0/2990944911 wait complete. 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.960+0000 7f1e08878700 1 Processor -- start 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e08878700 1 -- start start 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e08878700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0419db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e08878700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0410dc10 con 0x7f1e0407d4e0 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e0259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0419db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e0259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0419db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46536/0 (socket says 172.21.15.98:46536) 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e0259c700 1 -- 172.21.15.98:0/3934386406 learned_addr learned my addr 172.21.15.98:0/3934386406 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:12.657 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e0259c700 1 -- 172.21.15.98:0/3934386406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e0419e0c0 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e0259c700 1 --2- 172.21.15.98:0/3934386406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0419db80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1df40119b0 tx=0x7f1df4011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1df37fe700 1 -- 172.21.15.98:0/3934386406 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1df4003af0 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1df37fe700 1 -- 172.21.15.98:0/3934386406 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f1df401a8a0 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.961+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e0419e350 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.962+0000 7f1df37fe700 1 -- 172.21.15.98:0/3934386406 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1df4024830 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.962+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e041a0f60 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.962+0000 7f1df37fe700 1 -- 172.21.15.98:0/3934386406 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1df4022020 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.962+0000 7f1df37fe700 1 -- 172.21.15.98:0/3934386406 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1df40193c0 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.963+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e04064cf0 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:11.965+0000 7f1df37fe700 1 -- 172.21.15.98:0/3934386406 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1df402ddf0 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.034+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f1e041a1130 con 0x7f1e0407d4e0 2024-04-25T16:22:12.658 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.035+0000 7f1df37fe700 1 -- 172.21.15.98:0/3934386406 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f1df4032030 con 0x7f1e0407d4e0 2024-04-25T16:22:12.659 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.035+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 msgr2=0x7f1e0419db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:12.659 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.035+0000 7f1e08878700 1 --2- 172.21.15.98:0/3934386406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0419db80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1df40119b0 tx=0x7f1df4011a90 comp rx=0 tx=0).stop 2024-04-25T16:22:12.659 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.035+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 shutdown_connections 2024-04-25T16:22:12.659 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.035+0000 7f1e08878700 1 --2- 172.21.15.98:0/3934386406 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1e0407d4e0 0x7f1e0419db80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:12.659 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.035+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 >> 172.21.15.98:0/3934386406 conn(0x7f1e04103f40 msgr2=0x7f1e041047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:12.659 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.036+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 shutdown_connections 2024-04-25T16:22:12.659 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:12.036+0000 7f1e08878700 1 -- 172.21.15.98:0/3934386406 wait complete. 2024-04-25T16:22:12.659 INFO:teuthology.orchestra.run.smithi098.stdout:Restarting the monitor... 2024-04-25T16:22:12.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:12 smithi098 systemd[1]: Stopping Ceph mon.a for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:22:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:12 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a[28781]: 2024-04-25T16:22:12.915+0000 7f6b6c7bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:22:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:12 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a[28781]: 2024-04-25T16:22:12.915+0000 7f6b6c7bb700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T16:22:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:13 smithi098 podman[29032]: 2024-04-25 16:22:13.141319171 +0000 UTC m=+0.326702649 container died 3043e3a79c283e3035708deb93980cfcdcf03f0fff72018164b117336a9a4a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:22:13.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:13 smithi098 podman[29032]: 2024-04-25 16:22:13.457540778 +0000 UTC m=+0.642924254 container cleanup 3043e3a79c283e3035708deb93980cfcdcf03f0fff72018164b117336a9a4a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T16:22:13.676 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:13 smithi098 bash[29032]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a 2024-04-25T16:22:14.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:13 smithi098 podman[29046]: 2024-04-25 16:22:13.675359867 +0000 UTC m=+0.529434734 container remove 3043e3a79c283e3035708deb93980cfcdcf03f0fff72018164b117336a9a4a82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5) 2024-04-25T16:22:14.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:13 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T16:22:14.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:13 smithi098 systemd[1]: Stopped Ceph mon.a for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:22:14.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:13 smithi098 systemd[1]: Starting Ceph mon.a for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:22:14.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 podman[29143]: 2024-04-25 16:22:13.905585074 +0000 UTC m=+0.029275709 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:22:14.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 podman[29143]: 2024-04-25 16:22:14.009643571 +0000 UTC m=+0.133334192 container create 14b5f450ea62fdb680835601015e2c12ecb96813a73cae46fe511f2f48e4e3bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T16:22:14.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 podman[29143]: 2024-04-25 16:22:14.187166333 +0000 UTC m=+0.310856963 container init 14b5f450ea62fdb680835601015e2c12ecb96813a73cae46fe511f2f48e4e3bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-25T16:22:14.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 podman[29143]: 2024-04-25 16:22:14.190232105 +0000 UTC m=+0.313922728 container start 14b5f450ea62fdb680835601015e2c12ecb96813a73cae46fe511f2f48e4e3bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: pidfile_write: ignore empty --pid-file 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: load: jerasure load: lrc 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: RocksDB version: 7.9.2 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Git sha 0 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: DB SUMMARY 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: DB Session ID: 2MRS19CY54OKXBFT4R5I 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: CURRENT file: CURRENT 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T16:22:14.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T16:22:14.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103546 ; 2024-04-25T16:22:14.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.error_if_exists: 0 2024-04-25T16:22:14.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.create_if_missing: 0 2024-04-25T16:22:14.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.paranoid_checks: 1 2024-04-25T16:22:14.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T16:22:14.257 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.env: 0x5642056a4500 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.info_log: 0x564206589fc0 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.statistics: (nil) 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.use_fsync: 0 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_log_file_size: 0 2024-04-25T16:22:14.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T16:22:14.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T16:22:14.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T16:22:14.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T16:22:14.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.allow_fallocate: 1 2024-04-25T16:22:14.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T16:22:14.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T16:22:14.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.use_direct_reads: 0 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.db_log_dir: 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.wal_dir: 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T16:22:14.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.write_buffer_manager: 0x56420655a8c0 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.unordered_write: 0 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T16:22:14.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.row_cache: None 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.wal_filter: None 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.two_write_queues: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.wal_compression: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.atomic_flush: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.log_readahead_size: 0 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T16:22:14.262 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_background_jobs: 2 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_background_compactions: -1 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_subcompactions: 1 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T16:22:14.263 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_open_files: -1 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_background_flushes: -1 2024-04-25T16:22:14.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Compression algorithms supported: 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: kZSTD supported: 0 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: kXpressCompression supported: 0 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: kZlibCompression supported: 1 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: kSnappyCompression supported: 1 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: kLZ4Compression supported: 1 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: kBZip2Compression supported: 0 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.merge_operator: 2024-04-25T16:22:14.265 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_filter: None 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5642066901a0) 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-04-25T16:22:14.266 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x56420658d1f0 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-04-25T16:22:14.267 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-04-25T16:22:14.268 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression: NoCompression 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.num_levels: 7 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T16:22:14.269 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T16:22:14.270 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T16:22:14.271 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T16:22:14.272 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.inplace_update_support: 0 2024-04-25T16:22:14.273 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T16:22:14.274 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T16:22:14.274 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T16:22:14.274 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T16:22:14.274 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.bloom_locality: 0 2024-04-25T16:22:14.274 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.max_successive_merges: 0 2024-04-25T16:22:14.274 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T16:22:14.274 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T16:22:14.274 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.ttl: 2592000 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.enable_blob_files: false 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.min_blob_size: 0 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T16:22:14.275 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 22b58246-068c-4993-9977-d18027e7a4b8 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062134230119, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062134231172, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98834, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96787, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13792, "raw_average_key_size": 49, "raw_value_size": 89229, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714062134, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "22b58246-068c-4993-9977-d18027e7a4b8", "db_session_id": "2MRS19CY54OKXBFT4R5I", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062134231255, "job": 1, "event": "recovery_finished"} 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564206571c00 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: DB pointer 0x56420b364000 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-04-25T16:22:14.276 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 98.38 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 98.38 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T16:22:14.277 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T16:22:14.278 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 29.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 29.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x56420658d1f0#2 capacity: 512.00 MB usage: 1.41 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.91 KB,0.000172853%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:14.279 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a@-1(???) e1 preinit fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a@-1(???).mds e1 new map 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a@-1(???).mds e1 print_map 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-04-25T16:22:14.280 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T16:22:14.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:22:14.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:22:14.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:22:14.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T16:22:14.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 bash[29143]: 14b5f450ea62fdb680835601015e2c12ecb96813a73cae46fe511f2f48e4e3bd 2024-04-25T16:22:14.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 systemd[1]: Started Ceph mon.a for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:22:14.282 INFO:teuthology.orchestra.run.smithi098.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T16:22:14.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T16:22:14.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: monmap epoch 1 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: last_changed 2024-04-25T16:22:06.623046+0000 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: created 2024-04-25T16:22:06.623046+0000 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: min_mon_release 19 (squid) 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: election_strategy: 1 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: fsmap 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T16:22:14.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:14 smithi098 ceph-mon[29162]: mgrmap e1: no daemons active 2024-04-25T16:22:15.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.675+0000 7f37668fa700 1 Processor -- start 2024-04-25T16:22:15.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.675+0000 7f37668fa700 1 -- start start 2024-04-25T16:22:15.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.675+0000 7f37668fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f3760109d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:15.285 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.675+0000 7f37668fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f376010a370 con 0x7f3760109970 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.675+0000 7f375ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f3760109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.675+0000 7f375ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f3760109d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46550/0 (socket says 172.21.15.98:46550) 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.675+0000 7f375ffff700 1 -- 172.21.15.98:0/1616162334 learned_addr learned my addr 172.21.15.98:0/1616162334 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.676+0000 7f375ffff700 1 -- 172.21.15.98:0/1616162334 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376010aaa0 con 0x7f3760109970 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.676+0000 7f375ffff700 1 --2- 172.21.15.98:0/1616162334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f3760109d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f37500098a0 tx=0x7f3750009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26c6031088fa55f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.676+0000 7f375effd700 1 -- 172.21.15.98:0/1616162334 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3750004160 con 0x7f3760109970 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.676+0000 7f375effd700 1 -- 172.21.15.98:0/1616162334 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f3750004300 con 0x7f3760109970 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.676+0000 7f375effd700 1 -- 172.21.15.98:0/1616162334 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37500119e0 con 0x7f3760109970 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.677+0000 7f37668fa700 1 -- 172.21.15.98:0/1616162334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 msgr2=0x7f3760109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.677+0000 7f37668fa700 1 --2- 172.21.15.98:0/1616162334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f3760109d80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f37500098a0 tx=0x7f3750009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.677+0000 7f37668fa700 1 -- 172.21.15.98:0/1616162334 shutdown_connections 2024-04-25T16:22:15.286 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.677+0000 7f37668fa700 1 --2- 172.21.15.98:0/1616162334 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f3760109d80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.677+0000 7f37668fa700 1 -- 172.21.15.98:0/1616162334 >> 172.21.15.98:0/1616162334 conn(0x7f3760105010 msgr2=0x7f3760107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.677+0000 7f37668fa700 1 -- 172.21.15.98:0/1616162334 shutdown_connections 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.677+0000 7f37668fa700 1 -- 172.21.15.98:0/1616162334 wait complete. 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.677+0000 7f37668fa700 1 Processor -- start 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.678+0000 7f37668fa700 1 -- start start 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.678+0000 7f37668fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f376019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.678+0000 7f37668fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f376010dcd0 con 0x7f3760109970 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.678+0000 7f375ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f376019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:15.287 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.678+0000 7f375ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f376019db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46554/0 (socket says 172.21.15.98:46554) 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.678+0000 7f375ffff700 1 -- 172.21.15.98:0/3643895662 learned_addr learned my addr 172.21.15.98:0/3643895662 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.678+0000 7f375ffff700 1 -- 172.21.15.98:0/3643895662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376019e0d0 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.678+0000 7f375ffff700 1 --2- 172.21.15.98:0/3643895662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f376019db90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f37500119b0 tx=0x7f3750011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.679+0000 7f375d7fa700 1 -- 172.21.15.98:0/3643895662 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3750003970 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.679+0000 7f375d7fa700 1 -- 172.21.15.98:0/3643895662 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f3750003b10 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.679+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376019e360 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.679+0000 7f375d7fa700 1 -- 172.21.15.98:0/3643895662 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3750024610 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.679+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37601a0f70 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.679+0000 7f375d7fa700 1 -- 172.21.15.98:0/3643895662 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3750024c60 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.679+0000 7f375d7fa700 1 -- 172.21.15.98:0/3643895662 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3750019700 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.680+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3760064cf0 con 0x7f3760109970 2024-04-25T16:22:15.288 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.682+0000 7f375d7fa700 1 -- 172.21.15.98:0/3643895662 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f375001aae0 con 0x7f3760109970 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.751+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f37601a1140 con 0x7f3760109970 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.753+0000 7f375d7fa700 1 -- 172.21.15.98:0/3643895662 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f375001a430 con 0x7f3760109970 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.754+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 msgr2=0x7f376019db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.754+0000 7f37668fa700 1 --2- 172.21.15.98:0/3643895662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f376019db90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f37500119b0 tx=0x7f3750011a90 comp rx=0 tx=0).stop 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.754+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 shutdown_connections 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.754+0000 7f37668fa700 1 --2- 172.21.15.98:0/3643895662 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3760109970 0x7f376019db90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.754+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 >> 172.21.15.98:0/3643895662 conn(0x7f3760105010 msgr2=0x7f3760105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.754+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 shutdown_connections 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:14.754+0000 7f37668fa700 1 -- 172.21.15.98:0/3643895662 wait complete. 2024-04-25T16:22:15.289 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T16:22:15.302 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:22:15.302 INFO:teuthology.orchestra.run.smithi098.stdout:Creating mgr... 2024-04-25T16:22:15.302 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T16:22:15.303 INFO:teuthology.orchestra.run.smithi098.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T16:22:15.555 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.x 2024-04-25T16:22:15.556 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T16:22:15.697 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c520eaa8-031f-11ef-bc93-c7b262605968.target.wants/ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-c520eaa8-031f-11ef-bc93-c7b262605968@.service. 2024-04-25T16:22:15.921 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:15 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3643895662' entity='client.admin' 2024-04-25T16:22:16.170 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T16:22:16.170 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-25T16:22:16.178 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T16:22:16.178 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T16:22:16.178 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-25T16:22:16.178 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T16:22:16.187 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T16:22:16.187 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-25T16:22:16.195 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T16:22:16.195 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T16:22:16.195 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-25T16:22:16.195 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T16:22:16.195 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr to start... 2024-04-25T16:22:16.195 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr... 2024-04-25T16:22:16.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:16 smithi098 systemd[1]: Started Ceph mgr.x for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:22:16.535 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:16.263+0000 7fb7bc2dc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:22:16.535 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:16.453+0000 7fb7bc2dc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:22:16.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:16 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4263023568' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T16:22:16.876 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:16.556+0000 7fb7bc2dc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:22:16.876 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:16.874+0000 7fb7bc2dc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:22:17.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:16.960+0000 7fb7bc2dc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:22:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:22:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:22:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T16:22:17.304 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T16:22:17.305 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T16:22:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T16:22:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T16:22:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T16:22:17.306 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T16:22:17.314 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T16:22:17.314 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T16:22:17.314 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T16:22:17.314 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:17.314 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T16:22:17.314 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:17.315 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T16:22:09.449842+0000", 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:22:17.316 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.644+0000 7f7e70c2a700 1 Processor -- start 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.644+0000 7f7e70c2a700 1 -- start start 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.644+0000 7f7e70c2a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c10a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.644+0000 7f7e70c2a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c10a730 con 0x7f7e6c109d50 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.644+0000 7f7e6a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c10a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.645+0000 7f7e6a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c10a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54076/0 (socket says 172.21.15.98:54076) 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.645+0000 7f7e6a59c700 1 -- 172.21.15.98:0/3153856756 learned_addr learned my addr 172.21.15.98:0/3153856756 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.645+0000 7f7e6a59c700 1 -- 172.21.15.98:0/3153856756 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c1089d0 con 0x7f7e6c109d50 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.645+0000 7f7e6a59c700 1 --2- 172.21.15.98:0/3153856756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c10a160 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0098a0 tx=0x7f7e5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379978e99b3d2c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.645+0000 7f7e6959a700 1 -- 172.21.15.98:0/3153856756 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e5c004160 con 0x7f7e6c109d50 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.645+0000 7f7e6959a700 1 -- 172.21.15.98:0/3153856756 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f7e5c004300 con 0x7f7e6c109d50 2024-04-25T16:22:17.317 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.645+0000 7f7e6959a700 1 -- 172.21.15.98:0/3153856756 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e5c0119e0 con 0x7f7e6c109d50 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.646+0000 7f7e70c2a700 1 -- 172.21.15.98:0/3153856756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 msgr2=0x7f7e6c10a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.646+0000 7f7e70c2a700 1 --2- 172.21.15.98:0/3153856756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c10a160 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0098a0 tx=0x7f7e5c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.646+0000 7f7e70c2a700 1 -- 172.21.15.98:0/3153856756 shutdown_connections 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.646+0000 7f7e70c2a700 1 --2- 172.21.15.98:0/3153856756 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c10a160 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.646+0000 7f7e70c2a700 1 -- 172.21.15.98:0/3153856756 >> 172.21.15.98:0/3153856756 conn(0x7f7e6c1035f0 msgr2=0x7f7e6c105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.646+0000 7f7e70c2a700 1 -- 172.21.15.98:0/3153856756 shutdown_connections 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.646+0000 7f7e70c2a700 1 -- 172.21.15.98:0/3153856756 wait complete. 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.647+0000 7f7e70c2a700 1 Processor -- start 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.647+0000 7f7e70c2a700 1 -- start start 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.647+0000 7f7e70c2a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c1082a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.647+0000 7f7e70c2a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6c10cc10 con 0x7f7e6c109d50 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.647+0000 7f7e6a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c1082a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:17.318 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.647+0000 7f7e6a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c1082a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54078/0 (socket says 172.21.15.98:54078) 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.647+0000 7f7e6a59c700 1 -- 172.21.15.98:0/4263023568 learned_addr learned my addr 172.21.15.98:0/4263023568 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e6a59c700 1 -- 172.21.15.98:0/4263023568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6c1087e0 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e6a59c700 1 --2- 172.21.15.98:0/4263023568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c1082a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0119b0 tx=0x7f7e5c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e5b7fe700 1 -- 172.21.15.98:0/4263023568 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e5c003970 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e5b7fe700 1 -- 172.21.15.98:0/4263023568 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f7e5c003b10 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e6c1a0890 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e5b7fe700 1 -- 172.21.15.98:0/4263023568 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7e5c024610 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6c1a0d70 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e5b7fe700 1 -- 172.21.15.98:0/4263023568 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7e5c024c60 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.648+0000 7f7e5b7fe700 1 -- 172.21.15.98:0/4263023568 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7e5c019700 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.649+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e6c064cf0 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.651+0000 7f7e5b7fe700 1 -- 172.21.15.98:0/4263023568 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7e5c01aae0 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.722+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f7e6c1a1110 con 0x7f7e6c109d50 2024-04-25T16:22:17.319 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.722+0000 7f7e5b7fe700 1 -- 172.21.15.98:0/4263023568 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7e5c01a430 con 0x7f7e6c109d50 2024-04-25T16:22:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.724+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 msgr2=0x7f7e6c1082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.724+0000 7f7e70c2a700 1 --2- 172.21.15.98:0/4263023568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c1082a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c0119b0 tx=0x7f7e5c011a90 comp rx=0 tx=0).stop 2024-04-25T16:22:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.724+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 shutdown_connections 2024-04-25T16:22:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.724+0000 7f7e70c2a700 1 --2- 172.21.15.98:0/4263023568 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e6c109d50 0x7f7e6c1082a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.724+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 >> 172.21.15.98:0/4263023568 conn(0x7f7e6c1035f0 msgr2=0x7f7e6c103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.724+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 shutdown_connections 2024-04-25T16:22:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:16.724+0000 7f7e70c2a700 1 -- 172.21.15.98:0/4263023568 wait complete. 2024-04-25T16:22:17.320 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (1/15)... 2024-04-25T16:22:18.498 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:18.138+0000 7fb7bc2dc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:22:18.498 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:18.328+0000 7fb7bc2dc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:22:18.498 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:18.496+0000 7fb7bc2dc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:22:19.073 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:18.782+0000 7fb7bc2dc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:22:19.073 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:18.872+0000 7fb7bc2dc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:22:19.073 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:19 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:19.071+0000 7fb7bc2dc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:22:19.437 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:19 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:19.261+0000 7fb7bc2dc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:22:20.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:19 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:19.873+0000 7fb7bc2dc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:22:20.598 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:20 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:20.295+0000 7fb7bc2dc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:22:20.598 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:20 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:20.393+0000 7fb7bc2dc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:22:20.598 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:20 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:20.596+0000 7fb7bc2dc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:22:21.454 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:21.117+0000 7fb7bc2dc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:22:21.455 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:21.208+0000 7fb7bc2dc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:22:21.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:21.453+0000 7fb7bc2dc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:22:21.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:21.583+0000 7fb7bc2dc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:22:21.733 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:21.677+0000 7fb7bc2dc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:22:22.004 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:21.790+0000 7fb7bc2dc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:22:22.504 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:22 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:22.089+0000 7fb7bc2dc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:22:22.912 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:22 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/382339890' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T16:22:22.912 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:22 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:22.688+0000 7fb7bc2dc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:22:22.912 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:22 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:22.910+0000 7fb7bc2dc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:22:24.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: Activating manager daemon x 2024-04-25T16:22:24.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: mgrmap e2: x(active, starting, since 0.00379289s) 2024-04-25T16:22:24.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: Manager daemon x is now available 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' 2024-04-25T16:22:24.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:23 smithi098 ceph-mon[29162]: from='mgr.14100 172.21.15.98:0/3166509496' entity='mgr.x' 2024-04-25T16:22:25.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:24 smithi098 ceph-mon[29162]: mgrmap e3: x(active, since 1.00829s) 2024-04-25T16:22:26.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:25 smithi098 ceph-mon[29162]: mgrmap e4: x(active, since 2s) 2024-04-25T16:22:29.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:29.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:22:29.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:22:29.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T16:22:29.075 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T16:22:29.076 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T16:22:29.077 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T16:22:29.078 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:29.079 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T16:22:09.449842+0000", 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.694+0000 7f295c9fa700 1 Processor -- start 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.694+0000 7f295c9fa700 1 -- start start 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.694+0000 7f295c9fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f2958109d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.694+0000 7f295c9fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f295810a2f0 con 0x7f29581098f0 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.695+0000 7f295659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f2958109d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.695+0000 7f295659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f2958109d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54092/0 (socket says 172.21.15.98:54092) 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.695+0000 7f295659c700 1 -- 172.21.15.98:0/420842847 learned_addr learned my addr 172.21.15.98:0/420842847 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.695+0000 7f295659c700 1 -- 172.21.15.98:0/420842847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295810aa20 con 0x7f29581098f0 2024-04-25T16:22:29.080 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.695+0000 7f295659c700 1 --2- 172.21.15.98:0/420842847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f2958109d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f29480098a0 tx=0x7f2948009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f2eebb06662214 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.695+0000 7f295559a700 1 -- 172.21.15.98:0/420842847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2948004160 con 0x7f29581098f0 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.695+0000 7f295559a700 1 -- 172.21.15.98:0/420842847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f2948004300 con 0x7f29581098f0 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.695+0000 7f295559a700 1 -- 172.21.15.98:0/420842847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29480119e0 con 0x7f29581098f0 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.696+0000 7f295c9fa700 1 -- 172.21.15.98:0/420842847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 msgr2=0x7f2958109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.696+0000 7f295c9fa700 1 --2- 172.21.15.98:0/420842847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f2958109d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f29480098a0 tx=0x7f2948009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.696+0000 7f295c9fa700 1 -- 172.21.15.98:0/420842847 shutdown_connections 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.696+0000 7f295c9fa700 1 --2- 172.21.15.98:0/420842847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f2958109d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.696+0000 7f295c9fa700 1 -- 172.21.15.98:0/420842847 >> 172.21.15.98:0/420842847 conn(0x7f2958104f50 msgr2=0x7f29581073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.696+0000 7f295c9fa700 1 -- 172.21.15.98:0/420842847 shutdown_connections 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.696+0000 7f295c9fa700 1 -- 172.21.15.98:0/420842847 wait complete. 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.697+0000 7f295c9fa700 1 Processor -- start 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.697+0000 7f295c9fa700 1 -- start start 2024-04-25T16:22:29.081 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.697+0000 7f295c9fa700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f295819dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.697+0000 7f295c9fa700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f295810dc50 con 0x7f29581098f0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.697+0000 7f295659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f295819dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.697+0000 7f295659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f295819dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:54106/0 (socket says 172.21.15.98:54106) 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.697+0000 7f295659c700 1 -- 172.21.15.98:0/382339890 learned_addr learned my addr 172.21.15.98:0/382339890 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f295659c700 1 -- 172.21.15.98:0/382339890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295819e010 con 0x7f29581098f0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f295659c700 1 --2- 172.21.15.98:0/382339890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f295819dad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29480046e0 tx=0x7f2948011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f29477fe700 1 -- 172.21.15.98:0/382339890 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29480039e0 con 0x7f29581098f0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f29477fe700 1 -- 172.21.15.98:0/382339890 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f2948003b80 con 0x7f29581098f0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f29477fe700 1 -- 172.21.15.98:0/382339890 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2948024700 con 0x7f29581098f0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295819e2a0 con 0x7f29581098f0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29581a0eb0 con 0x7f29581098f0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f29477fe700 1 -- 172.21.15.98:0/382339890 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2948022020 con 0x7f29581098f0 2024-04-25T16:22:29.082 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.698+0000 7f29477fe700 1 -- 172.21.15.98:0/382339890 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f294802dce0 con 0x7f29581098f0 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.699+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2958064cf0 con 0x7f29581098f0 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.701+0000 7f29477fe700 1 -- 172.21.15.98:0/382339890 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f294801ab30 con 0x7f29581098f0 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.773+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f29581a1080 con 0x7f29581098f0 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.773+0000 7f29477fe700 1 -- 172.21.15.98:0/382339890 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f294802d3f0 con 0x7f29581098f0 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.774+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 msgr2=0x7f295819dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.774+0000 7f295c9fa700 1 --2- 172.21.15.98:0/382339890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f295819dad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f29480046e0 tx=0x7f2948011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.774+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 shutdown_connections 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.775+0000 7f295c9fa700 1 --2- 172.21.15.98:0/382339890 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29581098f0 0x7f295819dad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.775+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 >> 172.21.15.98:0/382339890 conn(0x7f2958104f50 msgr2=0x7f29581056a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.775+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 shutdown_connections 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:22.775+0000 7f295c9fa700 1 -- 172.21.15.98:0/382339890 wait complete. 2024-04-25T16:22:29.083 INFO:teuthology.orchestra.run.smithi098.stdout:mgr not available, waiting (2/15)... 2024-04-25T16:22:32.205 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:31 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3655114313' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T16:22:32.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:32.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:22:32.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:22:32.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T16:22:32.447 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 0 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "quorum_age": 17, 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:32.448 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T16:22:32.449 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:32.450 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ], 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T16:22:32.451 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T16:22:09.449842+0000", 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout }, 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.505+0000 7f14b6cc7700 1 Processor -- start 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.505+0000 7f14b6cc7700 1 -- start start 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.505+0000 7f14b6cc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b010c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.505+0000 7f14b6cc7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14b010ca60 con 0x7f14b007cc70 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.505+0000 7f14b4a63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b010c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.506+0000 7f14b4a63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b010c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46382/0 (socket says 172.21.15.98:46382) 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.506+0000 7f14b4a63700 1 -- 172.21.15.98:0/2902279158 learned_addr learned my addr 172.21.15.98:0/2902279158 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.506+0000 7f14b4a63700 1 -- 172.21.15.98:0/2902279158 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b010cbe0 con 0x7f14b007cc70 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.506+0000 7f14b4a63700 1 --2- 172.21.15.98:0/2902279158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b010c490 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f14a40098a0 tx=0x7f14a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d290ae0f66e7a775 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.506+0000 7f14af7fe700 1 -- 172.21.15.98:0/2902279158 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14a4004160 con 0x7f14b007cc70 2024-04-25T16:22:32.452 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.506+0000 7f14af7fe700 1 -- 172.21.15.98:0/2902279158 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f14a4004300 con 0x7f14b007cc70 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.506+0000 7f14af7fe700 1 -- 172.21.15.98:0/2902279158 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14a40119e0 con 0x7f14b007cc70 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.507+0000 7f14b6cc7700 1 -- 172.21.15.98:0/2902279158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 msgr2=0x7f14b010c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.507+0000 7f14b6cc7700 1 --2- 172.21.15.98:0/2902279158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b010c490 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f14a40098a0 tx=0x7f14a4009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.507+0000 7f14b6cc7700 1 -- 172.21.15.98:0/2902279158 shutdown_connections 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.507+0000 7f14b6cc7700 1 --2- 172.21.15.98:0/2902279158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b010c490 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.507+0000 7f14b6cc7700 1 -- 172.21.15.98:0/2902279158 >> 172.21.15.98:0/2902279158 conn(0x7f14b0103ee0 msgr2=0x7f14b0106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.507+0000 7f14b6cc7700 1 -- 172.21.15.98:0/2902279158 shutdown_connections 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.507+0000 7f14b6cc7700 1 -- 172.21.15.98:0/2902279158 wait complete. 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b6cc7700 1 Processor -- start 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b6cc7700 1 -- start start 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b6cc7700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b6cc7700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14b010db90 con 0x7f14b007cc70 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b4a63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:32.453 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b4a63700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b019db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46386/0 (socket says 172.21.15.98:46386) 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b4a63700 1 -- 172.21.15.98:0/3655114313 learned_addr learned my addr 172.21.15.98:0/3655114313 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b4a63700 1 -- 172.21.15.98:0/3655114313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b019e050 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.508+0000 7f14b4a63700 1 --2- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b019db10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f14a40116a0 tx=0x7f14a4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.509+0000 7f14adffb700 1 -- 172.21.15.98:0/3655114313 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14a40039f0 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.509+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b019e2e0 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.509+0000 7f14adffb700 1 -- 172.21.15.98:0/3655114313 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f14a4003b90 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.509+0000 7f14adffb700 1 -- 172.21.15.98:0/3655114313 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f14a401a600 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.509+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b01a0ef0 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.510+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14b0064cf0 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.510+0000 7f14adffb700 1 -- 172.21.15.98:0/3655114313 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f14a401a8b0 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.510+0000 7f14adffb700 1 --2- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f14a003d5e0 0x7f14a003faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.510+0000 7f14adffb700 1 -- 172.21.15.98:0/3655114313 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f14a40529a0 con 0x7f14b007cc70 2024-04-25T16:22:32.454 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.510+0000 7f14affff700 1 --2- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f14a003d5e0 0x7f14a003faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.511+0000 7f14affff700 1 --2- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f14a003d5e0 0x7f14a003faf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f149c006fd0 tx=0x7f149c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.516+0000 7f14adffb700 1 -- 172.21.15.98:0/3655114313 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14a400e260 con 0x7f14b007cc70 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.799+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f14b01a1110 con 0x7f14b007cc70 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.799+0000 7f14adffb700 1 -- 172.21.15.98:0/3655114313 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f14b01a1110 con 0x7f14b007cc70 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f14a003d5e0 msgr2=0x7f14a003faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 --2- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f14a003d5e0 0x7f14a003faf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f149c006fd0 tx=0x7f149c006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 msgr2=0x7f14b019db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 --2- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b019db10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f14a40116a0 tx=0x7f14a4011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 shutdown_connections 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 --2- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f14a003d5e0 0x7f14a003faf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 --2- 172.21.15.98:0/3655114313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14b007cc70 0x7f14b019db10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:32.455 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 >> 172.21.15.98:0/3655114313 conn(0x7f14b0103ee0 msgr2=0x7f14b0105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:32.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 shutdown_connections 2024-04-25T16:22:32.456 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:31.801+0000 7f14b6cc7700 1 -- 172.21.15.98:0/3655114313 wait complete. 2024-04-25T16:22:32.456 INFO:teuthology.orchestra.run.smithi098.stdout:mgr is available 2024-04-25T16:22:33.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:33 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3783227288' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T16:22:33.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:33.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T16:22:33.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout fsid = c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:33.625 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T16:22:33.626 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.852+0000 7f6a6aa7d700 1 Processor -- start 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.852+0000 7f6a6aa7d700 1 -- start start 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.852+0000 7f6a6aa7d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6410c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.852+0000 7f6a6aa7d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6410cb40 con 0x7f6a6407d4e0 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.852+0000 7f6a68819700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6410c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.852+0000 7f6a68819700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6410c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46392/0 (socket says 172.21.15.98:46392) 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.852+0000 7f6a68819700 1 -- 172.21.15.98:0/2672017800 learned_addr learned my addr 172.21.15.98:0/2672017800 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.852+0000 7f6a68819700 1 -- 172.21.15.98:0/2672017800 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6410ccc0 con 0x7f6a6407d4e0 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a68819700 1 --2- 172.21.15.98:0/2672017800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6410c570 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a580098a0 tx=0x7f6a58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68dd2d5efe286e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:33.627 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a637fe700 1 -- 172.21.15.98:0/2672017800 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a58004160 con 0x7f6a6407d4e0 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a637fe700 1 -- 172.21.15.98:0/2672017800 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6a58004300 con 0x7f6a6407d4e0 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a637fe700 1 -- 172.21.15.98:0/2672017800 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a580119e0 con 0x7f6a6407d4e0 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/2672017800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 msgr2=0x7f6a6410c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a6aa7d700 1 --2- 172.21.15.98:0/2672017800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6410c570 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a580098a0 tx=0x7f6a58009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/2672017800 shutdown_connections 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a6aa7d700 1 --2- 172.21.15.98:0/2672017800 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6410c570 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.853+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/2672017800 >> 172.21.15.98:0/2672017800 conn(0x7f6a64103f40 msgr2=0x7f6a64106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.854+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/2672017800 shutdown_connections 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.854+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/2672017800 wait complete. 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.854+0000 7f6a6aa7d700 1 Processor -- start 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.854+0000 7f6a6aa7d700 1 -- start start 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.854+0000 7f6a6aa7d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6419db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.854+0000 7f6a6aa7d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6a6410dc10 con 0x7f6a6407d4e0 2024-04-25T16:22:33.628 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a68819700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6419db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a68819700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6419db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46408/0 (socket says 172.21.15.98:46408) 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a68819700 1 -- 172.21.15.98:0/3783227288 learned_addr learned my addr 172.21.15.98:0/3783227288 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a68819700 1 -- 172.21.15.98:0/3783227288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a6419e0c0 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a68819700 1 --2- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6419db80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a580116a0 tx=0x7f6a58011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a61ffb700 1 -- 172.21.15.98:0/3783227288 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a580039f0 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a61ffb700 1 -- 172.21.15.98:0/3783227288 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6a58003b90 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a6419e350 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a61ffb700 1 -- 172.21.15.98:0/3783227288 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a5801a600 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.855+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a641a0f60 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.856+0000 7f6a61ffb700 1 -- 172.21.15.98:0/3783227288 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f6a5801a8b0 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.856+0000 7f6a61ffb700 1 --2- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f6a5403d5e0 0x7f6a5403faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.857+0000 7f6a61ffb700 1 -- 172.21.15.98:0/3783227288 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6a580529a0 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.857+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a64064cf0 con 0x7f6a6407d4e0 2024-04-25T16:22:33.629 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.857+0000 7f6a63fff700 1 --2- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f6a5403d5e0 0x7f6a5403faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.857+0000 7f6a63fff700 1 --2- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f6a5403d5e0 0x7f6a5403faf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6a50006fd0 tx=0x7f6a50006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:32.862+0000 7f6a61ffb700 1 -- 172.21.15.98:0/3783227288 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a5800e260 con 0x7f6a6407d4e0 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.059+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f6a641a1130 con 0x7f6a6407d4e0 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.060+0000 7f6a61ffb700 1 -- 172.21.15.98:0/3783227288 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f6a58018e10 con 0x7f6a6407d4e0 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f6a5403d5e0 msgr2=0x7f6a5403faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 --2- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f6a5403d5e0 0x7f6a5403faf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6a50006fd0 tx=0x7f6a50006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 msgr2=0x7f6a6419db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 --2- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6419db80 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6a580116a0 tx=0x7f6a58011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 shutdown_connections 2024-04-25T16:22:33.630 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 --2- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f6a5403d5e0 0x7f6a5403faf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 --2- 172.21.15.98:0/3783227288 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6a6407d4e0 0x7f6a6419db80 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 >> 172.21.15.98:0/3783227288 conn(0x7f6a64103f40 msgr2=0x7f6a641047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 shutdown_connections 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:33.062+0000 7f6a6aa7d700 1 -- 172.21.15.98:0/3783227288 wait complete. 2024-04-25T16:22:33.631 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling cephadm module... 2024-04-25T16:22:34.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:34 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1776484558' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T16:22:34.743 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:34 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setuser ceph since I am not root 2024-04-25T16:22:34.743 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:34 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setgroup ceph since I am not root 2024-04-25T16:22:34.743 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:34 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:34.555+0000 7f59a7d9c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:22:34.743 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:34 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:34.741+0000 7f59a7d9c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:22:35.004 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:34 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:34.843+0000 7f59a7d9c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:22:35.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:35 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1776484558' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T16:22:35.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:35 smithi098 ceph-mon[29162]: mgrmap e5: x(active, since 11s) 2024-04-25T16:22:35.504 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:35 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:35.154+0000 7f59a7d9c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:22:35.504 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:35 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:35.240+0000 7f59a7d9c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:22:35.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.044+0000 7f78d5dcb700 1 Processor -- start 2024-04-25T16:22:35.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.044+0000 7f78d5dcb700 1 -- start start 2024-04-25T16:22:35.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78d5dcb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d0109d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:35.792 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78d5dcb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78d010a370 con 0x7f78d0109970 2024-04-25T16:22:35.793 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78cf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d0109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:35.793 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78cf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d0109d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46412/0 (socket says 172.21.15.98:46412) 2024-04-25T16:22:35.793 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78cf7fe700 1 -- 172.21.15.98:0/3937112332 learned_addr learned my addr 172.21.15.98:0/3937112332 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:35.793 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78cf7fe700 1 -- 172.21.15.98:0/3937112332 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d010aaa0 con 0x7f78d0109970 2024-04-25T16:22:35.793 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78cf7fe700 1 --2- 172.21.15.98:0/3937112332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d0109d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f78c40098a0 tx=0x7f78c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c251c27a05f7f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:35.793 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78ce7fc700 1 -- 172.21.15.98:0/3937112332 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78c4004160 con 0x7f78d0109970 2024-04-25T16:22:35.793 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78ce7fc700 1 -- 172.21.15.98:0/3937112332 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f78c4004300 con 0x7f78d0109970 2024-04-25T16:22:35.794 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.045+0000 7f78ce7fc700 1 -- 172.21.15.98:0/3937112332 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78c40119e0 con 0x7f78d0109970 2024-04-25T16:22:35.794 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.046+0000 7f78d5dcb700 1 -- 172.21.15.98:0/3937112332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 msgr2=0x7f78d0109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:35.794 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.046+0000 7f78d5dcb700 1 --2- 172.21.15.98:0/3937112332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d0109d80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f78c40098a0 tx=0x7f78c4009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:35.794 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.046+0000 7f78d5dcb700 1 -- 172.21.15.98:0/3937112332 shutdown_connections 2024-04-25T16:22:35.794 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.046+0000 7f78d5dcb700 1 --2- 172.21.15.98:0/3937112332 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d0109d80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:35.794 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.046+0000 7f78d5dcb700 1 -- 172.21.15.98:0/3937112332 >> 172.21.15.98:0/3937112332 conn(0x7f78d0105010 msgr2=0x7f78d0107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:35.794 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.046+0000 7f78d5dcb700 1 -- 172.21.15.98:0/3937112332 shutdown_connections 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.046+0000 7f78d5dcb700 1 -- 172.21.15.98:0/3937112332 wait complete. 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.047+0000 7f78d5dcb700 1 Processor -- start 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.047+0000 7f78d5dcb700 1 -- start start 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.047+0000 7f78d5dcb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.047+0000 7f78d5dcb700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f78d010dcd0 con 0x7f78d0109970 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.047+0000 7f78cf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.047+0000 7f78cf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d019db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:46416/0 (socket says 172.21.15.98:46416) 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.047+0000 7f78cf7fe700 1 -- 172.21.15.98:0/1776484558 learned_addr learned my addr 172.21.15.98:0/1776484558 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.048+0000 7f78cf7fe700 1 -- 172.21.15.98:0/1776484558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d019e0d0 con 0x7f78d0109970 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.048+0000 7f78cf7fe700 1 --2- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d019db90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009fb0 tx=0x7f78c4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.048+0000 7f78ccff9700 1 -- 172.21.15.98:0/1776484558 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78c40038b0 con 0x7f78d0109970 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.048+0000 7f78ccff9700 1 -- 172.21.15.98:0/1776484558 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f78c4003a50 con 0x7f78d0109970 2024-04-25T16:22:35.795 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.048+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d019e360 con 0x7f78d0109970 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.048+0000 7f78ccff9700 1 -- 172.21.15.98:0/1776484558 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f78c4019550 con 0x7f78d0109970 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.048+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d01a0f70 con 0x7f78d0109970 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.049+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78d0064cf0 con 0x7f78d0109970 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.049+0000 7f78ccff9700 1 -- 172.21.15.98:0/1776484558 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f78c4019800 con 0x7f78d0109970 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.049+0000 7f78ccff9700 1 --2- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f78b803d5e0 0x7f78b803faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.049+0000 7f78ccff9700 1 -- 172.21.15.98:0/1776484558 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f78c4052900 con 0x7f78d0109970 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.049+0000 7f78ceffd700 1 --2- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f78b803d5e0 0x7f78b803faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.050+0000 7f78ceffd700 1 --2- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f78b803d5e0 0x7f78b803faf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f78c0006fd0 tx=0x7f78c0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.054+0000 7f78ccff9700 1 -- 172.21.15.98:0/1776484558 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78c4017e90 con 0x7f78d0109970 2024-04-25T16:22:35.796 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.309+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f78d01a1260 con 0x7f78d0109970 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.361+0000 7f78ccff9700 1 -- 172.21.15.98:0/1776484558 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f78c4023390 con 0x7f78d0109970 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.363+0000 7f78ccff9700 1 -- 172.21.15.98:0/1776484558 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f78c402d440 con 0x7f78d0109970 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f78b803d5e0 msgr2=0x7f78b803faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 --2- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f78b803d5e0 0x7f78b803faf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f78c0006fd0 tx=0x7f78c0006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 msgr2=0x7f78d019db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 --2- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d019db90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009fb0 tx=0x7f78c4011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 shutdown_connections 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 --2- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f78b803d5e0 0x7f78b803faf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 --2- 172.21.15.98:0/1776484558 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f78d0109970 0x7f78d019db90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:35.797 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 >> 172.21.15.98:0/1776484558 conn(0x7f78d0105010 msgr2=0x7f78d0105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:35.798 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 shutdown_connections 2024-04-25T16:22:35.798 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:34.364+0000 7f78d5dcb700 1 -- 172.21.15.98:0/1776484558 wait complete. 2024-04-25T16:22:36.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1342838916' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T16:22:36.754 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:36 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:36.462+0000 7f59a7d9c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:22:36.754 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:36 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:36.641+0000 7f59a7d9c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.206+0000 7f9a4b402700 1 Processor -- start 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.207+0000 7f9a4b402700 1 -- start start 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.207+0000 7f9a4b402700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a44109d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.207+0000 7f9a4b402700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4410a340 con 0x7f9a44109940 2024-04-25T16:22:37.046 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.207+0000 7f9a4919e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a44109d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.207+0000 7f9a4919e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a44109d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57216/0 (socket says 172.21.15.98:57216) 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.207+0000 7f9a4919e700 1 -- 172.21.15.98:0/2579068230 learned_addr learned my addr 172.21.15.98:0/2579068230 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.207+0000 7f9a4919e700 1 -- 172.21.15.98:0/2579068230 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4410aa70 con 0x7f9a44109940 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.208+0000 7f9a4919e700 1 --2- 172.21.15.98:0/2579068230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a44109d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9a400098a0 tx=0x7f9a40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee1d452dcf13db1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.208+0000 7f9a3bfff700 1 -- 172.21.15.98:0/2579068230 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a40004160 con 0x7f9a44109940 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.208+0000 7f9a3bfff700 1 -- 172.21.15.98:0/2579068230 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f9a40004300 con 0x7f9a44109940 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.208+0000 7f9a3bfff700 1 -- 172.21.15.98:0/2579068230 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a400119e0 con 0x7f9a44109940 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.208+0000 7f9a4b402700 1 -- 172.21.15.98:0/2579068230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 msgr2=0x7f9a44109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.208+0000 7f9a4b402700 1 --2- 172.21.15.98:0/2579068230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a44109d50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9a400098a0 tx=0x7f9a40009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.209+0000 7f9a4b402700 1 -- 172.21.15.98:0/2579068230 shutdown_connections 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.209+0000 7f9a4b402700 1 --2- 172.21.15.98:0/2579068230 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a44109d50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.209+0000 7f9a4b402700 1 -- 172.21.15.98:0/2579068230 >> 172.21.15.98:0/2579068230 conn(0x7f9a44104fa0 msgr2=0x7f9a44107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:37.047 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.209+0000 7f9a4b402700 1 -- 172.21.15.98:0/2579068230 shutdown_connections 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.209+0000 7f9a4b402700 1 -- 172.21.15.98:0/2579068230 wait complete. 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.209+0000 7f9a4b402700 1 Processor -- start 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.209+0000 7f9a4b402700 1 -- start start 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4b402700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a4419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4b402700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a4410dca0 con 0x7f9a44109940 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4919e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a4419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4919e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a4419db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57220/0 (socket says 172.21.15.98:57220) 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4919e700 1 -- 172.21.15.98:0/1342838916 learned_addr learned my addr 172.21.15.98:0/1342838916 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4919e700 1 -- 172.21.15.98:0/1342838916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a4419e080 con 0x7f9a44109940 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4919e700 1 --2- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a4419db40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009fb0 tx=0x7f9a40011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a3a7fc700 1 -- 172.21.15.98:0/1342838916 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a400038b0 con 0x7f9a44109940 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a3a7fc700 1 -- 172.21.15.98:0/1342838916 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f9a40003a50 con 0x7f9a44109940 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a4419e310 con 0x7f9a44109940 2024-04-25T16:22:37.048 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a3a7fc700 1 -- 172.21.15.98:0/1342838916 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a4001a5c0 con 0x7f9a44109940 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.210+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a441a0f20 con 0x7f9a44109940 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.211+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a44064cf0 con 0x7f9a44109940 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.212+0000 7f9a3a7fc700 1 -- 172.21.15.98:0/1342838916 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f9a4001a8c0 con 0x7f9a44109940 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.212+0000 7f9a3a7fc700 1 --2- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f9a3003d5e0 0x7f9a3003faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.212+0000 7f9a3a7fc700 1 -- 172.21.15.98:0/1342838916 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9a400529f0 con 0x7f9a44109940 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.212+0000 7f9a4899d700 1 -- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f9a3003d5e0 msgr2=0x7f9a3003faf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4161723298 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.212+0000 7f9a4899d700 1 --2- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f9a3003d5e0 0x7f9a3003faf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.217+0000 7f9a3a7fc700 1 -- 172.21.15.98:0/1342838916 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a40017e90 con 0x7f9a44109940 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.413+0000 7f9a4899d700 1 -- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f9a3003d5e0 msgr2=0x7f9a3003faf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4161723298 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.413+0000 7f9a4899d700 1 --2- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f9a3003d5e0 0x7f9a3003faf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.481+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f9a441a1140 con 0x7f9a44109940 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.482+0000 7f9a3a7fc700 1 -- 172.21.15.98:0/1342838916 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f9a441a1140 con 0x7f9a44109940 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f9a3003d5e0 msgr2=0x7f9a3003faf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:22:37.049 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 --2- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f9a3003d5e0 0x7f9a3003faf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 msgr2=0x7f9a4419db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 --2- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a4419db40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9a40009fb0 tx=0x7f9a40011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 shutdown_connections 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 --2- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7f9a3003d5e0 0x7f9a3003faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 --2- 172.21.15.98:0/1342838916 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a44109940 0x7f9a4419db40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 >> 172.21.15.98:0/1342838916 conn(0x7f9a44104fa0 msgr2=0x7f9a441057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 shutdown_connections 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:36.484+0000 7f9a4b402700 1 -- 172.21.15.98:0/1342838916 wait complete. 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-04-25T16:22:37.050 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 5... 2024-04-25T16:22:37.116 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:36 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:36.817+0000 7f59a7d9c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:22:37.116 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:37 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:37.114+0000 7f59a7d9c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:22:37.418 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:37 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:37.209+0000 7f59a7d9c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:22:37.418 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:37 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:37.417+0000 7f59a7d9c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:22:37.754 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:37 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:37.616+0000 7f59a7d9c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:22:38.504 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:38 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:38.212+0000 7f59a7d9c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:22:38.938 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:38 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:38.637+0000 7f59a7d9c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:22:38.938 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:38 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:38.734+0000 7f59a7d9c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:22:38.938 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:38 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:38.937+0000 7f59a7d9c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:22:39.754 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:39 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:39.461+0000 7f59a7d9c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:22:39.754 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:39 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:39.554+0000 7f59a7d9c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:22:40.143 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:39 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:39.797+0000 7f59a7d9c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:22:40.143 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:39 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:39.929+0000 7f59a7d9c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:22:40.143 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:40 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:40.025+0000 7f59a7d9c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:22:40.143 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:40 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:40.141+0000 7f59a7d9c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:22:40.754 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:40 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:40.445+0000 7f59a7d9c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:22:41.318 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:41.041+0000 7f59a7d9c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:22:41.318 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:22:41.263+0000 7f59a7d9c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:22:41.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: Active manager daemon x restarted 2024-04-25T16:22:41.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: Activating manager daemon x 2024-04-25T16:22:41.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T16:22:41.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: mgrmap e6: x(active, starting, since 0.00370811s) 2024-04-25T16:22:41.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:22:41.736 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: Manager daemon x is now available 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:41.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:41 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:22:42.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:42 smithi098 ceph-mon[29162]: Found migration_current of "None". Setting to last migration. 2024-04-25T16:22:42.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:42 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:22:42.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:42 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:42.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:42 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:42.609 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:42 smithi098 ceph-mon[29162]: mgrmap e7: x(active, since 1.00634s) 2024-04-25T16:22:42.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:22:42.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T16:22:42.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T16:22:42.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:22:42.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.457+0000 7fcce6ba6700 1 Processor -- start 2024-04-25T16:22:42.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.457+0000 7fcce6ba6700 1 -- start start 2024-04-25T16:22:42.888 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.457+0000 7fcce6ba6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce0109d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.457+0000 7fcce6ba6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcce010a360 con 0x7fcce0109960 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.457+0000 7fcce4942700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce0109d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.457+0000 7fcce4942700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce0109d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57228/0 (socket says 172.21.15.98:57228) 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.457+0000 7fcce4942700 1 -- 172.21.15.98:0/2933282979 learned_addr learned my addr 172.21.15.98:0/2933282979 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.457+0000 7fcce4942700 1 -- 172.21.15.98:0/2933282979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce010aa40 con 0x7fcce0109960 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.458+0000 7fcce4942700 1 --2- 172.21.15.98:0/2933282979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce0109d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fccd40098a0 tx=0x7fccd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa58ed07df036f3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.458+0000 7fccdf7fe700 1 -- 172.21.15.98:0/2933282979 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd4004160 con 0x7fcce0109960 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.458+0000 7fccdf7fe700 1 -- 172.21.15.98:0/2933282979 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fccd4004300 con 0x7fcce0109960 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.458+0000 7fccdf7fe700 1 -- 172.21.15.98:0/2933282979 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd40119e0 con 0x7fcce0109960 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 -- 172.21.15.98:0/2933282979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 msgr2=0x7fcce0109d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 --2- 172.21.15.98:0/2933282979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce0109d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fccd40098a0 tx=0x7fccd4009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 -- 172.21.15.98:0/2933282979 shutdown_connections 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 --2- 172.21.15.98:0/2933282979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce0109d70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 -- 172.21.15.98:0/2933282979 >> 172.21.15.98:0/2933282979 conn(0x7fcce0104fe0 msgr2=0x7fcce0107450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:42.889 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 -- 172.21.15.98:0/2933282979 shutdown_connections 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 -- 172.21.15.98:0/2933282979 wait complete. 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 Processor -- start 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 -- start start 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.459+0000 7fcce6ba6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce019fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fcce6ba6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcce010dc70 con 0x7fcce0109960 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fcce4942700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce019fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fcce4942700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce019fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57240/0 (socket says 172.21.15.98:57240) 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fcce4942700 1 -- 172.21.15.98:0/2463732030 learned_addr learned my addr 172.21.15.98:0/2463732030 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fcce4942700 1 -- 172.21.15.98:0/2463732030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcce01a0220 con 0x7fcce0109960 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fcce4942700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce019fce0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009fb0 tx=0x7fccd4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:42.890 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd40038b0 con 0x7fcce0109960 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fccd4003a50 con 0x7fcce0109960 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fccd401a5c0 con 0x7fcce0109960 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.460+0000 7fcce6ba6700 1 -- 172.21.15.98:0/2463732030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcce01a04b0 con 0x7fcce0109960 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.461+0000 7fcce6ba6700 1 -- 172.21.15.98:0/2463732030 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcce01a30c0 con 0x7fcce0109960 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.461+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7fccd401a870 con 0x7fcce0109960 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.462+0000 7fccddffb700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 0x7fccd003fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.462+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 --> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fccd0040250 con 0x7fccd003d630 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.462+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fccd4052ac0 con 0x7fcce0109960 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.462+0000 7fccdffff700 1 -- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 msgr2=0x7fccd003fb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4161723298 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.462+0000 7fccdffff700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 0x7fccd003fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.663+0000 7fccdffff700 1 -- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 msgr2=0x7fccd003fb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4161723298 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:37.663+0000 7fccdffff700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 0x7fccd003fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:38.064+0000 7fccdffff700 1 -- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 msgr2=0x7fccd003fb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4161723298 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:38.064+0000 7fccdffff700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 0x7fccd003fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:38.865+0000 7fccdffff700 1 -- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 msgr2=0x7fccd003fb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4161723298 2024-04-25T16:22:42.891 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:38.865+0000 7fccdffff700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 0x7fccd003fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:40.467+0000 7fccdffff700 1 -- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 msgr2=0x7fccd003fb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/4161723298 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:40.467+0000 7fccdffff700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 0x7fccd003fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:41.267+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 6) ==== 49283+0+0 (secure 0 0 0) 0x7fccd402d440 con 0x7fcce0109960 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:41.267+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 msgr2=0x7fccd003fb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:41.267+0000 7fccddffb700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 0x7fccd003fb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.268+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7fccd40188a0 con 0x7fcce0109960 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.269+0000 7fccddffb700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fccd0040f70 0x7fccd0043390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.269+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fccd401ce50 con 0x7fccd0040f70 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.269+0000 7fccdffff700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fccd0040f70 0x7fccd0043390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.270+0000 7fccdffff700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fccd0040f70 0x7fccd0043390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcccc003a10 tx=0x7fcccc0092b0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.271+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mgr.14118 v2:172.21.15.98:6800/217035597 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fccd401ce50 con 0x7fccd0040f70 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 -- 172.21.15.98:0/2463732030 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fccc0002600 con 0x7fccd0040f70 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fccddffb700 1 -- 172.21.15.98:0/2463732030 <== mgr.14118 v2:172.21.15.98:6800/217035597 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7fccc0002600 con 0x7fccd0040f70 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 -- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fccd0040f70 msgr2=0x7fccd0043390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fccd0040f70 0x7fccd0043390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fcccc003a10 tx=0x7fcccc0092b0 comp rx=0 tx=0).stop 2024-04-25T16:22:42.892 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 -- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 msgr2=0x7fcce019fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce019fce0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009fb0 tx=0x7fccd4011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 -- 172.21.15.98:0/2463732030 shutdown_connections 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fccd0040f70 0x7fccd0043390 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcce0109960 0x7fcce019fce0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 --2- 172.21.15.98:0/2463732030 >> [v2:172.21.15.98:6800/4161723298,v1:172.21.15.98:6801/4161723298] conn(0x7fccd003d630 0x7fccd003fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.279+0000 7fcccb7fe700 1 -- 172.21.15.98:0/2463732030 >> 172.21.15.98:0/2463732030 conn(0x7fcce0104fe0 msgr2=0x7fcce007d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.280+0000 7fcccb7fe700 1 -- 172.21.15.98:0/2463732030 shutdown_connections 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:42.280+0000 7fcccb7fe700 1 -- 172.21.15.98:0/2463732030 wait complete. 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 5 is available 2024-04-25T16:22:42.893 INFO:teuthology.orchestra.run.smithi098.stdout:Setting orchestrator backend to cephadm... 2024-04-25T16:22:43.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:43 smithi098 ceph-mon[29162]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T16:22:43.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:43 smithi098 ceph-mon[29162]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T16:22:43.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:43 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.288+0000 7ff1dd82a700 1 Processor -- start 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.288+0000 7ff1dd82a700 1 -- start start 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.288+0000 7ff1dd82a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d8108ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.288+0000 7ff1dd82a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d81092d0 con 0x7ff1d81088d0 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.288+0000 7ff1d6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d8108ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.288+0000 7ff1d6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d8108ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57316/0 (socket says 172.21.15.98:57316) 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.288+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/950346542 learned_addr learned my addr 172.21.15.98:0/950346542 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.288+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/950346542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d8109a00 con 0x7ff1d81088d0 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.289+0000 7ff1d6ffd700 1 --2- 172.21.15.98:0/950346542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d8108ce0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80098a0 tx=0x7ff1c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79988b95b587e055 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:44.166 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.289+0000 7ff1d5ffb700 1 -- 172.21.15.98:0/950346542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1c8004160 con 0x7ff1d81088d0 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.289+0000 7ff1d5ffb700 1 -- 172.21.15.98:0/950346542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7ff1c8004300 con 0x7ff1d81088d0 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.289+0000 7ff1d5ffb700 1 -- 172.21.15.98:0/950346542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1c80119e0 con 0x7ff1d81088d0 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.290+0000 7ff1dd82a700 1 -- 172.21.15.98:0/950346542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 msgr2=0x7ff1d8108ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.290+0000 7ff1dd82a700 1 --2- 172.21.15.98:0/950346542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d8108ce0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff1c80098a0 tx=0x7ff1c8009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.290+0000 7ff1dd82a700 1 -- 172.21.15.98:0/950346542 shutdown_connections 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.290+0000 7ff1dd82a700 1 --2- 172.21.15.98:0/950346542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d8108ce0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.290+0000 7ff1dd82a700 1 -- 172.21.15.98:0/950346542 >> 172.21.15.98:0/950346542 conn(0x7ff1d8103f90 msgr2=0x7ff1d81063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.290+0000 7ff1dd82a700 1 -- 172.21.15.98:0/950346542 shutdown_connections 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.290+0000 7ff1dd82a700 1 -- 172.21.15.98:0/950346542 wait complete. 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.290+0000 7ff1dd82a700 1 Processor -- start 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1dd82a700 1 -- start start 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1dd82a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:44.167 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1dd82a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d810cc30 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1d6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1d6ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d819dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57330/0 (socket says 172.21.15.98:57330) 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/1044474128 learned_addr learned my addr 172.21.15.98:0/1044474128 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1d6ffd700 1 -- 172.21.15.98:0/1044474128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d819e0f0 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1d6ffd700 1 --2- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d819dbb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009fb0 tx=0x7ff1c8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1dc828700 1 -- 172.21.15.98:0/1044474128 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1c80038b0 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1dc828700 1 -- 172.21.15.98:0/1044474128 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7ff1c8003a50 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d819e380 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1dc828700 1 -- 172.21.15.98:0/1044474128 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff1c8019550 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.291+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d81a0f90 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.292+0000 7ff1dc828700 1 -- 172.21.15.98:0/1044474128 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7ff1c8019850 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.292+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d8064cf0 con 0x7ff1d81088d0 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.293+0000 7ff1dc828700 1 --2- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7ff1c003d4d0 0x7ff1c003f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:44.168 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.293+0000 7ff1dc828700 1 -- 172.21.15.98:0/1044474128 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff1c8052810 con 0x7ff1d81088d0 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.293+0000 7ff1d67fc700 1 --2- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7ff1c003d4d0 0x7ff1c003f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.293+0000 7ff1d67fc700 1 --2- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7ff1c003d4d0 0x7ff1c003f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc006fd0 tx=0x7ff1cc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.298+0000 7ff1dc828700 1 -- 172.21.15.98:0/1044474128 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1c8017e90 con 0x7ff1d81088d0 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.324+0000 7ff1dc828700 1 -- 172.21.15.98:0/1044474128 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7ff1c802d440 con 0x7ff1d81088d0 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.517+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7ff1d802e880 con 0x7ff1c003d4d0 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.523+0000 7ff1dc828700 1 -- 172.21.15.98:0/1044474128 <== mgr.14118 v2:172.21.15.98:6800/217035597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff1d802e880 con 0x7ff1c003d4d0 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7ff1c003d4d0 msgr2=0x7ff1c003f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 --2- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7ff1c003d4d0 0x7ff1c003f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc006fd0 tx=0x7ff1cc006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 msgr2=0x7ff1d819dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 --2- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d819dbb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff1c8009fb0 tx=0x7ff1c8011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 shutdown_connections 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 --2- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7ff1c003d4d0 0x7ff1c003f9e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 --2- 172.21.15.98:0/1044474128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff1d81088d0 0x7ff1d819dbb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:44.169 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 >> 172.21.15.98:0/1044474128 conn(0x7ff1d8103f90 msgr2=0x7ff1d81940b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:44.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 shutdown_connections 2024-04-25T16:22:44.170 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:43.525+0000 7ff1dd82a700 1 -- 172.21.15.98:0/1044474128 wait complete. 2024-04-25T16:22:44.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:44 smithi098 ceph-mon[29162]: [25/Apr/2024:16:22:42] ENGINE Bus STARTING 2024-04-25T16:22:44.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:44 smithi098 ceph-mon[29162]: [25/Apr/2024:16:22:42] ENGINE Serving on http://172.21.15.98:8765 2024-04-25T16:22:44.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:44 smithi098 ceph-mon[29162]: [25/Apr/2024:16:22:42] ENGINE Serving on https://172.21.15.98:7150 2024-04-25T16:22:44.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:44 smithi098 ceph-mon[29162]: [25/Apr/2024:16:22:42] ENGINE Bus STARTED 2024-04-25T16:22:44.470 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:44 smithi098 ceph-mon[29162]: mgrmap e8: x(active, since 2s) 2024-04-25T16:22:44.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:44 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:44.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:44 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:45.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:45 smithi098 ceph-mon[29162]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.584+0000 7f2200577700 1 Processor -- start 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.584+0000 7f2200577700 1 -- start start 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.584+0000 7f2200577700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f8108ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.584+0000 7f2200577700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21f81092d0 con 0x7f21f81088d0 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.584+0000 7f21fe313700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f8108ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.584+0000 7f21fe313700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f8108ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57340/0 (socket says 172.21.15.98:57340) 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.584+0000 7f21fe313700 1 -- 172.21.15.98:0/522875195 learned_addr learned my addr 172.21.15.98:0/522875195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.585+0000 7f21fe313700 1 -- 172.21.15.98:0/522875195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f8109a00 con 0x7f21f81088d0 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.585+0000 7f21fe313700 1 --2- 172.21.15.98:0/522875195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f8108ce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21f40098a0 tx=0x7f21f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53c904f695a6ef4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.585+0000 7f21fd311700 1 -- 172.21.15.98:0/522875195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4004160 con 0x7f21f81088d0 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.585+0000 7f21fd311700 1 -- 172.21.15.98:0/522875195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f21f4004300 con 0x7f21f81088d0 2024-04-25T16:22:45.423 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.585+0000 7f21fd311700 1 -- 172.21.15.98:0/522875195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f40119e0 con 0x7f21f81088d0 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 -- 172.21.15.98:0/522875195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 msgr2=0x7f21f8108ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 --2- 172.21.15.98:0/522875195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f8108ce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f21f40098a0 tx=0x7f21f4009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 -- 172.21.15.98:0/522875195 shutdown_connections 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 --2- 172.21.15.98:0/522875195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f8108ce0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 -- 172.21.15.98:0/522875195 >> 172.21.15.98:0/522875195 conn(0x7f21f8103f90 msgr2=0x7f21f81063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 -- 172.21.15.98:0/522875195 shutdown_connections 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 -- 172.21.15.98:0/522875195 wait complete. 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 Processor -- start 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.586+0000 7f2200577700 1 -- start start 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f2200577700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f807d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f2200577700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21f810cc30 con 0x7f21f81088d0 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f21fe313700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f807d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f21fe313700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f807d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57348/0 (socket says 172.21.15.98:57348) 2024-04-25T16:22:45.424 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f21fe313700 1 -- 172.21.15.98:0/827463957 learned_addr learned my addr 172.21.15.98:0/827463957 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f21fe313700 1 -- 172.21.15.98:0/827463957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f807d740 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f21fe313700 1 --2- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f807d200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009fb0 tx=0x7f21f4011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f21ef7fe700 1 -- 172.21.15.98:0/827463957 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f40038b0 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f21ef7fe700 1 -- 172.21.15.98:0/827463957 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f21f4003a50 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f81a0950 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f21ef7fe700 1 -- 172.21.15.98:0/827463957 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21f4019550 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.587+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f81a0dd0 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.588+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f8064cf0 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.588+0000 7f21ef7fe700 1 -- 172.21.15.98:0/827463957 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f21f4019850 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.589+0000 7f21ef7fe700 1 --2- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f21e403d5e0 0x7f21e403faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.589+0000 7f21ef7fe700 1 -- 172.21.15.98:0/827463957 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f21f40529c0 con 0x7f21f81088d0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.589+0000 7f21fdb12700 1 --2- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f21e403d5e0 0x7f21e403faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.589+0000 7f21fdb12700 1 --2- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f21e403d5e0 0x7f21e403faf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f21e8006fd0 tx=0x7f21e8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:45.425 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.594+0000 7f21ef7fe700 1 -- 172.21.15.98:0/827463957 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21f4054e20 con 0x7f21f81088d0 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.795+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f21f8105f60 con 0x7f21e403d5e0 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.796+0000 7f21ef7fe700 1 -- 172.21.15.98:0/827463957 <== mgr.14118 v2:172.21.15.98:6800/217035597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f21f8105f60 con 0x7f21e403d5e0 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f21e403d5e0 msgr2=0x7f21e403faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 --2- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f21e403d5e0 0x7f21e403faf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f21e8006fd0 tx=0x7f21e8006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 msgr2=0x7f21f807d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 --2- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f807d200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f21f4009fb0 tx=0x7f21f4011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 shutdown_connections 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 --2- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f21e403d5e0 0x7f21e403faf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 --2- 172.21.15.98:0/827463957 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f81088d0 0x7f21f807d200 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 >> 172.21.15.98:0/827463957 conn(0x7f21f8103f90 msgr2=0x7f21f8105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 shutdown_connections 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:44.798+0000 7f2200577700 1 -- 172.21.15.98:0/827463957 wait complete. 2024-04-25T16:22:45.426 INFO:teuthology.orchestra.run.smithi098.stdout:Generating ssh key... 2024-04-25T16:22:46.326 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: Generating public/private rsa key pair. 2024-04-25T16:22:46.326 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: Your identification has been saved in /tmp/tmp5wuwxgld/key. 2024-04-25T16:22:46.326 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: Your public key has been saved in /tmp/tmp5wuwxgld/key.pub. 2024-04-25T16:22:46.326 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: The key fingerprint is: 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: SHA256:rJ7M90vLv1/j6J+BonvaKPOQsEtsXnIaaZM9vY4se9w ceph-c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: The key's randomart image is: 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: +---[RSA 3072]----+ 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | | 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | | 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | | 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | . | 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | . S | 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | . *.o . | 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | %oB.... . o.| 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | ==X*=E=o. o.+| 2024-04-25T16:22:46.327 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: | =O+=XO++=o+ | 2024-04-25T16:22:46.328 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: +----[SHA256]-----+ 2024-04-25T16:22:46.721 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-mon[29162]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:46.721 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:46.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:46 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.803+0000 7f1583629700 1 Processor -- start 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.803+0000 7f1583629700 1 -- start start 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.803+0000 7f1583629700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c108080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.803+0000 7f1583629700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f157c108670 con 0x7f157c107c70 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.803+0000 7f15813c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.803+0000 7f15813c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51666/0 (socket says 172.21.15.98:51666) 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.803+0000 7f15813c5700 1 -- 172.21.15.98:0/2976600487 learned_addr learned my addr 172.21.15.98:0/2976600487 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.804+0000 7f15813c5700 1 -- 172.21.15.98:0/2976600487 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157c1087f0 con 0x7f157c107c70 2024-04-25T16:22:46.728 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.804+0000 7f15813c5700 1 --2- 172.21.15.98:0/2976600487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c108080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f15780098a0 tx=0x7f1578009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eda060bdc20418b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.804+0000 7f1573fff700 1 -- 172.21.15.98:0/2976600487 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1578004160 con 0x7f157c107c70 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.804+0000 7f1573fff700 1 -- 172.21.15.98:0/2976600487 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f1578004300 con 0x7f157c107c70 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.804+0000 7f1573fff700 1 -- 172.21.15.98:0/2976600487 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15780119e0 con 0x7f157c107c70 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 -- 172.21.15.98:0/2976600487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 msgr2=0x7f157c108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 --2- 172.21.15.98:0/2976600487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c108080 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f15780098a0 tx=0x7f1578009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 -- 172.21.15.98:0/2976600487 shutdown_connections 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 --2- 172.21.15.98:0/2976600487 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c108080 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 -- 172.21.15.98:0/2976600487 >> 172.21.15.98:0/2976600487 conn(0x7f157c103670 msgr2=0x7f157c105ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 -- 172.21.15.98:0/2976600487 shutdown_connections 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 -- 172.21.15.98:0/2976600487 wait complete. 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 Processor -- start 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.805+0000 7f1583629700 1 -- start start 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f1583629700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c10a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:46.729 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f1583629700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f157c10dc20 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f15813c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c10a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f15813c5700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c10a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51670/0 (socket says 172.21.15.98:51670) 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f15813c5700 1 -- 172.21.15.98:0/960039666 learned_addr learned my addr 172.21.15.98:0/960039666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f15813c5700 1 -- 172.21.15.98:0/960039666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157c10a9c0 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f15813c5700 1 --2- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c10a480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f15780116a0 tx=0x7f1578011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f15727fc700 1 -- 172.21.15.98:0/960039666 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f15780038b0 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f15727fc700 1 -- 172.21.15.98:0/960039666 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f1578003a50 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f157c108c10 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f15727fc700 1 -- 172.21.15.98:0/960039666 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1578019550 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.806+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f157c1090f0 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.808+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f157c064cf0 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.808+0000 7f15727fc700 1 -- 172.21.15.98:0/960039666 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f157801a430 con 0x7f157c107c70 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.808+0000 7f15727fc700 1 --2- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f156803d5e0 0x7f156803faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:46.730 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.808+0000 7f15727fc700 1 -- 172.21.15.98:0/960039666 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f15780529f0 con 0x7f157c107c70 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.808+0000 7f1580bc4700 1 --2- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f156803d5e0 0x7f156803faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.808+0000 7f1580bc4700 1 --2- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f156803d5e0 0x7f156803faf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f156c006fd0 tx=0x7f156c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:45.813+0000 7f15727fc700 1 -- 172.21.15.98:0/960039666 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f157800b3e0 con 0x7f157c107c70 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.011+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f157c1098c0 con 0x7f156803d5e0 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.109+0000 7f15727fc700 1 -- 172.21.15.98:0/960039666 <== mgr.14118 v2:172.21.15.98:6800/217035597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f157c1098c0 con 0x7f156803d5e0 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f156803d5e0 msgr2=0x7f156803faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 --2- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f156803d5e0 0x7f156803faf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f156c006fd0 tx=0x7f156c006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 msgr2=0x7f157c10a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 --2- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c10a480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f15780116a0 tx=0x7f1578011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 shutdown_connections 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 --2- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f156803d5e0 0x7f156803faf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:46.731 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 --2- 172.21.15.98:0/960039666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f157c107c70 0x7f157c10a480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:46.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 >> 172.21.15.98:0/960039666 conn(0x7f157c103670 msgr2=0x7f157c103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:46.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 shutdown_connections 2024-04-25T16:22:46.732 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:46.112+0000 7f1583629700 1 -- 172.21.15.98:0/960039666 wait complete. 2024-04-25T16:22:47.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:47 smithi098 ceph-mon[29162]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:47.668 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:47 smithi098 ceph-mon[29162]: Generating ssh key... 2024-04-25T16:22:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAC/RVSqSaEOy9NgdSFWdOb8xovoBYy0xK5zmBzzMo2wHbDZnXXYyFOv66PhC7Mj3Jqr/Jczme8AA28cQtkAzxxAhH2ftobP0n/Ru5LSBFIlTrKMHUmxFwcCj3pvCCLJwnRINJqksu3AMteGcg2lVT2f7VHeIEfDECWCv3WiwSJVcleYK4dybRhiol2bZ/k9fPnnjxUQ/U/CpiUYnLBOsI8lvMGKefsOEb+K8U8InURtCuQzrJwS0LOU/V92w2EU9usXcwy88EL/ZG+9R59iDB/s+eQLkp9zlCb2rF54aNrnAcllWwX8B9h7p2IndASJ8W5U7eMd5qZkBCbs49Bl33ENnFLIHdYH0FWMegdAic3moGXVecCoCrecgDp7Q8aK3KJ/QxHfajATDfX65UC+hdnkyQnIZ5/RnFFcJ7NhmOPD0DtNd+kkOts/LO+zxZX1vGYxUo7MPVcHbcypjuo2n7140Vp8YTBv+pC4j0xkeAm3YezlmUNPzB7Gc5nNJDN8= ceph-c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:22:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.156+0000 7f61ae216700 1 Processor -- start 2024-04-25T16:22:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.156+0000 7f61ae216700 1 -- start start 2024-04-25T16:22:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.156+0000 7f61ae216700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a8108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.156+0000 7f61ae216700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61a81092a0 con 0x7f61a81088a0 2024-04-25T16:22:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.156+0000 7f61a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a8108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.156+0000 7f61a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a8108cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51686/0 (socket says 172.21.15.98:51686) 2024-04-25T16:22:47.945 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.156+0000 7f61a77fe700 1 -- 172.21.15.98:0/1222410468 learned_addr learned my addr 172.21.15.98:0/1222410468 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.156+0000 7f61a77fe700 1 -- 172.21.15.98:0/1222410468 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a81099d0 con 0x7f61a81088a0 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.157+0000 7f61a77fe700 1 --2- 172.21.15.98:0/1222410468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a8108cb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f61980098a0 tx=0x7f6198009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8583345cbc171f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.157+0000 7f61a67fc700 1 -- 172.21.15.98:0/1222410468 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6198004160 con 0x7f61a81088a0 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.157+0000 7f61a67fc700 1 -- 172.21.15.98:0/1222410468 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6198004300 con 0x7f61a81088a0 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.157+0000 7f61a67fc700 1 -- 172.21.15.98:0/1222410468 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61980119e0 con 0x7f61a81088a0 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 -- 172.21.15.98:0/1222410468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 msgr2=0x7f61a8108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 --2- 172.21.15.98:0/1222410468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a8108cb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f61980098a0 tx=0x7f6198009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 -- 172.21.15.98:0/1222410468 shutdown_connections 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 --2- 172.21.15.98:0/1222410468 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a8108cb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 -- 172.21.15.98:0/1222410468 >> 172.21.15.98:0/1222410468 conn(0x7f61a8103f40 msgr2=0x7f61a8106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 -- 172.21.15.98:0/1222410468 shutdown_connections 2024-04-25T16:22:47.946 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 -- 172.21.15.98:0/1222410468 wait complete. 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 Processor -- start 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.158+0000 7f61ae216700 1 -- start start 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61ae216700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a807d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61ae216700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f61a810cc00 con 0x7f61a81088a0 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a807d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61a77fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a807d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51690/0 (socket says 172.21.15.98:51690) 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61a77fe700 1 -- 172.21.15.98:0/3661142155 learned_addr learned my addr 172.21.15.98:0/3661142155 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61a77fe700 1 -- 172.21.15.98:0/3661142155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a807d760 con 0x7f61a81088a0 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61a77fe700 1 --2- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a807d220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6198009fb0 tx=0x7f6198011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61a4ff9700 1 -- 172.21.15.98:0/3661142155 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f61980038b0 con 0x7f61a81088a0 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61a4ff9700 1 -- 172.21.15.98:0/3661142155 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f6198003a50 con 0x7f61a81088a0 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61a4ff9700 1 -- 172.21.15.98:0/3661142155 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6198019550 con 0x7f61a81088a0 2024-04-25T16:22:47.947 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a81a0970 con 0x7f61a81088a0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.159+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a81a0df0 con 0x7f61a81088a0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.160+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a8064cf0 con 0x7f61a81088a0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.160+0000 7f61a4ff9700 1 -- 172.21.15.98:0/3661142155 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f6198019850 con 0x7f61a81088a0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.161+0000 7f61a4ff9700 1 --2- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f619003d5e0 0x7f619003faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.161+0000 7f61a4ff9700 1 -- 172.21.15.98:0/3661142155 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f61980529d0 con 0x7f61a81088a0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.161+0000 7f61a6ffd700 1 --2- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f619003d5e0 0x7f619003faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.161+0000 7f61a6ffd700 1 --2- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f619003d5e0 0x7f619003faf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f619c006fd0 tx=0x7f619c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.166+0000 7f61a4ff9700 1 -- 172.21.15.98:0/3661142155 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6198017e90 con 0x7f61a81088a0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.365+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f61a8104e50 con 0x7f619003d5e0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.366+0000 7f61a4ff9700 1 -- 172.21.15.98:0/3661142155 <== mgr.14118 v2:172.21.15.98:6800/217035597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f61a8104e50 con 0x7f619003d5e0 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f619003d5e0 msgr2=0x7f619003faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 --2- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f619003d5e0 0x7f619003faf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f619c006fd0 tx=0x7f619c006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:47.948 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 msgr2=0x7f61a807d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 --2- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a807d220 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6198009fb0 tx=0x7f6198011a60 comp rx=0 tx=0).stop 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 shutdown_connections 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 --2- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f619003d5e0 0x7f619003faf0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 --2- 172.21.15.98:0/3661142155 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f61a81088a0 0x7f61a807d220 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 >> 172.21.15.98:0/3661142155 conn(0x7f61a8103f40 msgr2=0x7f61a8104740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 shutdown_connections 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:47.368+0000 7f61ae216700 1 -- 172.21.15.98:0/3661142155 wait complete. 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T16:22:47.949 INFO:teuthology.orchestra.run.smithi098.stdout:Adding host smithi098... 2024-04-25T16:22:48.605 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:48 smithi098 ceph-mon[29162]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:49.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:49 smithi098 ceph-mon[29162]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:49.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:49 smithi098 ceph-mon[29162]: Deploying cephadm binary to smithi098 2024-04-25T16:22:53.235 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Added host 'smithi098' with addr '172.21.15.98' 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.390+0000 7f27d9141700 1 Processor -- start 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.390+0000 7f27d9141700 1 -- start start 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d9141700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d4109db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d9141700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27d410a3a0 con 0x7f27d41099a0 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d4109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d4109db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51704/0 (socket says 172.21.15.98:51704) 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d2d9d700 1 -- 172.21.15.98:0/4221424541 learned_addr learned my addr 172.21.15.98:0/4221424541 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d2d9d700 1 -- 172.21.15.98:0/4221424541 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d410aad0 con 0x7f27d41099a0 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d2d9d700 1 --2- 172.21.15.98:0/4221424541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d4109db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f27c80098a0 tx=0x7f27c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b4b12172e2c2db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d1d9b700 1 -- 172.21.15.98:0/4221424541 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27c8004160 con 0x7f27d41099a0 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d1d9b700 1 -- 172.21.15.98:0/4221424541 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f27c8004300 con 0x7f27d41099a0 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.391+0000 7f27d1d9b700 1 -- 172.21.15.98:0/4221424541 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27c80119e0 con 0x7f27d41099a0 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.392+0000 7f27d9141700 1 -- 172.21.15.98:0/4221424541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 msgr2=0x7f27d4109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:53.236 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.392+0000 7f27d9141700 1 --2- 172.21.15.98:0/4221424541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d4109db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f27c80098a0 tx=0x7f27c8009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.392+0000 7f27d9141700 1 -- 172.21.15.98:0/4221424541 shutdown_connections 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.392+0000 7f27d9141700 1 --2- 172.21.15.98:0/4221424541 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d4109db0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.392+0000 7f27d9141700 1 -- 172.21.15.98:0/4221424541 >> 172.21.15.98:0/4221424541 conn(0x7f27d4105040 msgr2=0x7f27d4107490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.392+0000 7f27d9141700 1 -- 172.21.15.98:0/4221424541 shutdown_connections 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.392+0000 7f27d9141700 1 -- 172.21.15.98:0/4221424541 wait complete. 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.393+0000 7f27d9141700 1 Processor -- start 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.393+0000 7f27d9141700 1 -- start start 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.393+0000 7f27d9141700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:53.237 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.393+0000 7f27d9141700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f27d410dd00 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.393+0000 7f27d2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.393+0000 7f27d2d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d419dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51720/0 (socket says 172.21.15.98:51720) 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.393+0000 7f27d2d9d700 1 -- 172.21.15.98:0/338248627 learned_addr learned my addr 172.21.15.98:0/338248627 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.394+0000 7f27d2d9d700 1 -- 172.21.15.98:0/338248627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27d419e120 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.394+0000 7f27d2d9d700 1 --2- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d419dbe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f27c80119b0 tx=0x7f27c8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.394+0000 7f27bbfff700 1 -- 172.21.15.98:0/338248627 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27c8003970 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.394+0000 7f27bbfff700 1 -- 172.21.15.98:0/338248627 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f27c8003b10 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.394+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27d419e3b0 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.394+0000 7f27bbfff700 1 -- 172.21.15.98:0/338248627 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f27c8024610 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.394+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27d41a0fc0 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.395+0000 7f27bbfff700 1 -- 172.21.15.98:0/338248627 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f27c802d3f0 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.395+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27d4064cf0 con 0x7f27d41099a0 2024-04-25T16:22:53.238 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.395+0000 7f27bbfff700 1 --2- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f27bc03d630 0x7f27bc03fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.395+0000 7f27bbfff700 1 -- 172.21.15.98:0/338248627 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f27c8052830 con 0x7f27d41099a0 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.395+0000 7f27d259c700 1 --2- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f27bc03d630 0x7f27bc03fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.396+0000 7f27d259c700 1 --2- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f27bc03d630 0x7f27bc03fb40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27c4006fd0 tx=0x7f27c4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.401+0000 7f27bbfff700 1 -- 172.21.15.98:0/338248627 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27c80094f0 con 0x7f27d41099a0 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:48.603+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}) -- 0x7f27d4105f50 con 0x7f27bc03d630 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.658+0000 7f27bbfff700 1 -- 172.21.15.98:0/338248627 <== mgr.14118 v2:172.21.15.98:6800/217035597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f27d4105f50 con 0x7f27bc03d630 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f27bc03d630 msgr2=0x7f27bc03fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 --2- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f27bc03d630 0x7f27bc03fb40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27c4006fd0 tx=0x7f27c4006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 msgr2=0x7f27d419dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 --2- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d419dbe0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f27c80119b0 tx=0x7f27c8011a90 comp rx=0 tx=0).stop 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 shutdown_connections 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 --2- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f27bc03d630 0x7f27bc03fb40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 --2- 172.21.15.98:0/338248627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f27d41099a0 0x7f27d419dbe0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:53.239 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 >> 172.21.15.98:0/338248627 conn(0x7f27d4105040 msgr2=0x7f27d4105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:53.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 shutdown_connections 2024-04-25T16:22:53.240 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:52.661+0000 7f27d9141700 1 -- 172.21.15.98:0/338248627 wait complete. 2024-04-25T16:22:53.240 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mon service... 2024-04-25T16:22:53.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:53 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:53.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:53 smithi098 ceph-mon[29162]: Added host smithi098 2024-04-25T16:22:53.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:53 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:22:55.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T16:22:55.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.751+0000 7f30eb624700 1 Processor -- start 2024-04-25T16:22:55.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.751+0000 7f30eb624700 1 -- start start 2024-04-25T16:22:55.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.752+0000 7f30eb624700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e410c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:55.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.752+0000 7f30eb624700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30e410cb40 con 0x7f30e407d580 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.752+0000 7f30e93c0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e410c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.752+0000 7f30e93c0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e410c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51722/0 (socket says 172.21.15.98:51722) 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.752+0000 7f30e93c0700 1 -- 172.21.15.98:0/2429087892 learned_addr learned my addr 172.21.15.98:0/2429087892 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.752+0000 7f30e93c0700 1 -- 172.21.15.98:0/2429087892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e410ccc0 con 0x7f30e407d580 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.752+0000 7f30e93c0700 1 --2- 172.21.15.98:0/2429087892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e410c530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30e00098a0 tx=0x7f30e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d8d0912368d1c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.752+0000 7f30dbfff700 1 -- 172.21.15.98:0/2429087892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30e0004160 con 0x7f30e407d580 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30dbfff700 1 -- 172.21.15.98:0/2429087892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f30e0004300 con 0x7f30e407d580 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30dbfff700 1 -- 172.21.15.98:0/2429087892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30e00119e0 con 0x7f30e407d580 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30eb624700 1 -- 172.21.15.98:0/2429087892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 msgr2=0x7f30e410c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30eb624700 1 --2- 172.21.15.98:0/2429087892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e410c530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f30e00098a0 tx=0x7f30e0009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30eb624700 1 -- 172.21.15.98:0/2429087892 shutdown_connections 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30eb624700 1 --2- 172.21.15.98:0/2429087892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e410c530 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:55.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30eb624700 1 -- 172.21.15.98:0/2429087892 >> 172.21.15.98:0/2429087892 conn(0x7f30e4103fe0 msgr2=0x7f30e4106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30eb624700 1 -- 172.21.15.98:0/2429087892 shutdown_connections 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.753+0000 7f30eb624700 1 -- 172.21.15.98:0/2429087892 wait complete. 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.754+0000 7f30eb624700 1 Processor -- start 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.754+0000 7f30eb624700 1 -- start start 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.754+0000 7f30eb624700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e4109ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.754+0000 7f30eb624700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30e410dc00 con 0x7f30e407d580 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.754+0000 7f30e93c0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e4109ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.754+0000 7f30e93c0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e4109ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51736/0 (socket says 172.21.15.98:51736) 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.754+0000 7f30e93c0700 1 -- 172.21.15.98:0/985573776 learned_addr learned my addr 172.21.15.98:0/985573776 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.755+0000 7f30e93c0700 1 -- 172.21.15.98:0/985573776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e410a1e0 con 0x7f30e407d580 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.755+0000 7f30e93c0700 1 --2- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e4109ca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30e00119b0 tx=0x7f30e0011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.755+0000 7f30da7fc700 1 -- 172.21.15.98:0/985573776 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30e0003af0 con 0x7f30e407d580 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.755+0000 7f30da7fc700 1 -- 172.21.15.98:0/985573776 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f30e001a8a0 con 0x7f30e407d580 2024-04-25T16:22:55.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.755+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e4108430 con 0x7f30e407d580 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.755+0000 7f30da7fc700 1 -- 172.21.15.98:0/985573776 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30e0024830 con 0x7f30e407d580 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.755+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e4108910 con 0x7f30e407d580 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.756+0000 7f30da7fc700 1 -- 172.21.15.98:0/985573776 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f30e0022020 con 0x7f30e407d580 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.756+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e4064cf0 con 0x7f30e407d580 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.756+0000 7f30da7fc700 1 --2- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f30d003d630 0x7f30d003fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.756+0000 7f30da7fc700 1 -- 172.21.15.98:0/985573776 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f30e00528c0 con 0x7f30e407d580 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.756+0000 7f30e8bbf700 1 --2- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f30d003d630 0x7f30d003fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.757+0000 7f30e8bbf700 1 --2- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f30d003d630 0x7f30d003fb40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f30d4006fd0 tx=0x7f30d4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.762+0000 7f30da7fc700 1 -- 172.21.15.98:0/985573776 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30e000e260 con 0x7f30e407d580 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.983+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f30e41090c0 con 0x7f30d003d630 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.991+0000 7f30da7fc700 1 -- 172.21.15.98:0/985573776 <== mgr.14118 v2:172.21.15.98:6800/217035597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f30e41090c0 con 0x7f30d003d630 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f30d003d630 msgr2=0x7f30d003fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:55.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 --2- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f30d003d630 0x7f30d003fb40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f30d4006fd0 tx=0x7f30d4006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 msgr2=0x7f30e4109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 --2- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e4109ca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f30e00119b0 tx=0x7f30e0011a90 comp rx=0 tx=0).stop 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 shutdown_connections 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 --2- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f30d003d630 0x7f30d003fb40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 --2- 172.21.15.98:0/985573776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f30e407d580 0x7f30e4109ca0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 >> 172.21.15.98:0/985573776 conn(0x7f30e4103fe0 msgr2=0x7f30e4105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 shutdown_connections 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:53.993+0000 7f30eb624700 1 -- 172.21.15.98:0/985573776 wait complete. 2024-04-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi098.stdout:Deploying unmanaged mgr service... 2024-04-25T16:22:55.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:54 smithi098 ceph-mon[29162]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:55.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:54 smithi098 ceph-mon[29162]: Saving service mon spec with placement count:5 2024-04-25T16:22:55.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:54 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:56.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T16:22:56.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.546+0000 7f756eaca700 1 Processor -- start 2024-04-25T16:22:56.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.546+0000 7f756eaca700 1 -- start start 2024-04-25T16:22:56.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f756eaca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f756810c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:56.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f756eaca700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f756810cb40 con 0x7f756807d580 2024-04-25T16:22:56.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f756c866700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f756810c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:56.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f756c866700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f756810c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40108/0 (socket says 172.21.15.98:40108) 2024-04-25T16:22:56.600 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f756c866700 1 -- 172.21.15.98:0/2985765078 learned_addr learned my addr 172.21.15.98:0/2985765078 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f756c866700 1 -- 172.21.15.98:0/2985765078 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756810ccc0 con 0x7f756807d580 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f756c866700 1 --2- 172.21.15.98:0/2985765078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f756810c530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f755c0098a0 tx=0x7f755c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2287d90c7f56c9d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f75677fe700 1 -- 172.21.15.98:0/2985765078 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f755c004160 con 0x7f756807d580 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.547+0000 7f75677fe700 1 -- 172.21.15.98:0/2985765078 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f755c004300 con 0x7f756807d580 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.548+0000 7f75677fe700 1 -- 172.21.15.98:0/2985765078 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f755c0119e0 con 0x7f756807d580 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.548+0000 7f756eaca700 1 -- 172.21.15.98:0/2985765078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 msgr2=0x7f756810c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.548+0000 7f756eaca700 1 --2- 172.21.15.98:0/2985765078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f756810c530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f755c0098a0 tx=0x7f755c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.548+0000 7f756eaca700 1 -- 172.21.15.98:0/2985765078 shutdown_connections 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.548+0000 7f756eaca700 1 --2- 172.21.15.98:0/2985765078 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f756810c530 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.548+0000 7f756eaca700 1 -- 172.21.15.98:0/2985765078 >> 172.21.15.98:0/2985765078 conn(0x7f7568103fe0 msgr2=0x7f7568106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.548+0000 7f756eaca700 1 -- 172.21.15.98:0/2985765078 shutdown_connections 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.548+0000 7f756eaca700 1 -- 172.21.15.98:0/2985765078 wait complete. 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.549+0000 7f756eaca700 1 Processor -- start 2024-04-25T16:22:56.601 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.549+0000 7f756eaca700 1 -- start start 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.549+0000 7f756eaca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f7568109ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.549+0000 7f756eaca700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f756810dc00 con 0x7f756807d580 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.549+0000 7f756c866700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f7568109ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.549+0000 7f756c866700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f7568109ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40118/0 (socket says 172.21.15.98:40118) 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.549+0000 7f756c866700 1 -- 172.21.15.98:0/2587321153 learned_addr learned my addr 172.21.15.98:0/2587321153 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.550+0000 7f756c866700 1 -- 172.21.15.98:0/2587321153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756810a1e0 con 0x7f756807d580 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.550+0000 7f756c866700 1 --2- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f7568109ca0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f755c0119b0 tx=0x7f755c011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.550+0000 7f7565ffb700 1 -- 172.21.15.98:0/2587321153 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f755c003af0 con 0x7f756807d580 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.550+0000 7f7565ffb700 1 -- 172.21.15.98:0/2587321153 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1032+0+0 (secure 0 0 0) 0x7f755c01a8a0 con 0x7f756807d580 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.550+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7568108430 con 0x7f756807d580 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.550+0000 7f7565ffb700 1 -- 172.21.15.98:0/2587321153 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f755c024830 con 0x7f756807d580 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.550+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7568108910 con 0x7f756807d580 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.551+0000 7f7565ffb700 1 -- 172.21.15.98:0/2587321153 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f755c022020 con 0x7f756807d580 2024-04-25T16:22:56.602 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.551+0000 7f7565ffb700 1 --2- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f755803d680 0x7f755803fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.551+0000 7f7565ffb700 1 -- 172.21.15.98:0/2587321153 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f755c053530 con 0x7f756807d580 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.551+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7568064cf0 con 0x7f756807d580 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.551+0000 7f7567fff700 1 --2- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f755803d680 0x7f755803fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.552+0000 7f7567fff700 1 --2- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f755803d680 0x7f755803fb90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7554006fd0 tx=0x7f7554006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.557+0000 7f7565ffb700 1 -- 172.21.15.98:0/2587321153 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f755c0094f0 con 0x7f756807d580 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.778+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f75681090d0 con 0x7f755803d680 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.783+0000 7f7565ffb700 1 -- 172.21.15.98:0/2587321153 <== mgr.14118 v2:172.21.15.98:6800/217035597 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f75681090d0 con 0x7f755803d680 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f755803d680 msgr2=0x7f755803fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 --2- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f755803d680 0x7f755803fb90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7554006fd0 tx=0x7f7554006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 msgr2=0x7f7568109ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 --2- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f7568109ca0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f755c0119b0 tx=0x7f755c011a90 comp rx=0 tx=0).stop 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 shutdown_connections 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 --2- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f755803d680 0x7f755803fb90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 --2- 172.21.15.98:0/2587321153 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f756807d580 0x7f7568109ca0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:56.603 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 >> 172.21.15.98:0/2587321153 conn(0x7f7568103fe0 msgr2=0x7f7568105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:56.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 shutdown_connections 2024-04-25T16:22:56.604 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:55.785+0000 7f756eaca700 1 -- 172.21.15.98:0/2587321153 wait complete. 2024-04-25T16:22:57.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:56 smithi098 ceph-mon[29162]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:22:57.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:56 smithi098 ceph-mon[29162]: Saving service mgr spec with placement count:2 2024-04-25T16:22:57.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:56 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:57.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:56 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:57.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:56 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:22:58.208 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.033+0000 7fe6b774f700 1 Processor -- start 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.033+0000 7fe6b774f700 1 -- start start 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.033+0000 7fe6b774f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b010e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.033+0000 7fe6b774f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b010ed90 con 0x7fe6b007ccb0 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.033+0000 7fe6b54eb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b010e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.033+0000 7fe6b54eb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b010e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40128/0 (socket says 172.21.15.98:40128) 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.033+0000 7fe6b54eb700 1 -- 172.21.15.98:0/3914380817 learned_addr learned my addr 172.21.15.98:0/3914380817 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.034+0000 7fe6b54eb700 1 -- 172.21.15.98:0/3914380817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b010ef10 con 0x7fe6b007ccb0 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.034+0000 7fe6b54eb700 1 --2- 172.21.15.98:0/3914380817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b010e850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0098a0 tx=0x7fe6ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1015421d06fa4588 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.034+0000 7fe6a7fff700 1 -- 172.21.15.98:0/3914380817 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6ac004160 con 0x7fe6b007ccb0 2024-04-25T16:22:58.209 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.034+0000 7fe6a7fff700 1 -- 172.21.15.98:0/3914380817 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6ac004300 con 0x7fe6b007ccb0 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.034+0000 7fe6a7fff700 1 -- 172.21.15.98:0/3914380817 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6ac011960 con 0x7fe6b007ccb0 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.035+0000 7fe6b774f700 1 -- 172.21.15.98:0/3914380817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 msgr2=0x7fe6b010e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.035+0000 7fe6b774f700 1 --2- 172.21.15.98:0/3914380817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b010e850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac0098a0 tx=0x7fe6ac009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.035+0000 7fe6b774f700 1 -- 172.21.15.98:0/3914380817 shutdown_connections 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.035+0000 7fe6b774f700 1 --2- 172.21.15.98:0/3914380817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b010e850 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.035+0000 7fe6b774f700 1 -- 172.21.15.98:0/3914380817 >> 172.21.15.98:0/3914380817 conn(0x7fe6b0103ee0 msgr2=0x7fe6b0106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.035+0000 7fe6b774f700 1 -- 172.21.15.98:0/3914380817 shutdown_connections 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.035+0000 7fe6b774f700 1 -- 172.21.15.98:0/3914380817 wait complete. 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b774f700 1 Processor -- start 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b774f700 1 -- start start 2024-04-25T16:22:58.210 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b774f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b774f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b010fe00 con 0x7fe6b007ccb0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b54eb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b54eb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b019fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40140/0 (socket says 172.21.15.98:40140) 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b54eb700 1 -- 172.21.15.98:0/191689902 learned_addr learned my addr 172.21.15.98:0/191689902 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b54eb700 1 -- 172.21.15.98:0/191689902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b01a0250 con 0x7fe6b007ccb0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.036+0000 7fe6b54eb700 1 --2- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b019fd10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac003e50 tx=0x7fe6ac011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.037+0000 7fe6a67fc700 1 -- 172.21.15.98:0/191689902 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6ac003a60 con 0x7fe6b007ccb0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.037+0000 7fe6a67fc700 1 -- 172.21.15.98:0/191689902 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe6ac01b8d0 con 0x7fe6b007ccb0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.037+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b01a04e0 con 0x7fe6b007ccb0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.037+0000 7fe6a67fc700 1 -- 172.21.15.98:0/191689902 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6ac0237d0 con 0x7fe6b007ccb0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.037+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b01a30e0 con 0x7fe6b007ccb0 2024-04-25T16:22:58.211 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.038+0000 7fe6a67fc700 1 -- 172.21.15.98:0/191689902 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fe6ac02c3f0 con 0x7fe6b007ccb0 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.038+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b0064cf0 con 0x7fe6b007ccb0 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.038+0000 7fe6a67fc700 1 --2- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe69c03d5e0 0x7fe69c03faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.038+0000 7fe6a67fc700 1 -- 172.21.15.98:0/191689902 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe6ac052e80 con 0x7fe6b007ccb0 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.038+0000 7fe6b4cea700 1 --2- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe69c03d5e0 0x7fe69c03faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.038+0000 7fe6b4cea700 1 --2- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe69c03d5e0 0x7fe69c03faf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0006fd0 tx=0x7fe6a0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.043+0000 7fe6a67fc700 1 -- 172.21.15.98:0/191689902 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6ac01f030 con 0x7fe6b007ccb0 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.271+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fe6b01a36e0 con 0x7fe6b007ccb0 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.275+0000 7fe6a67fc700 1 -- 172.21.15.98:0/191689902 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fe6ac01bd80 con 0x7fe6b007ccb0 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe69c03d5e0 msgr2=0x7fe69c03faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 --2- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe69c03d5e0 0x7fe69c03faf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0006fd0 tx=0x7fe6a0006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 msgr2=0x7fe6b019fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:58.212 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 --2- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b019fd10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe6ac003e50 tx=0x7fe6ac011a30 comp rx=0 tx=0).stop 2024-04-25T16:22:58.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 shutdown_connections 2024-04-25T16:22:58.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 --2- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe69c03d5e0 0x7fe69c03faf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:58.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 --2- 172.21.15.98:0/191689902 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe6b007ccb0 0x7fe6b019fd10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:58.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 >> 172.21.15.98:0/191689902 conn(0x7fe6b0103ee0 msgr2=0x7fe6b0105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:58.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 shutdown_connections 2024-04-25T16:22:58.213 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:57.278+0000 7fe6b774f700 1 -- 172.21.15.98:0/191689902 wait complete. 2024-04-25T16:22:58.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:58 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/191689902' entity='client.admin' 2024-04-25T16:22:59.576 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.614+0000 7f25b8f6c700 1 Processor -- start 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.614+0000 7f25b8f6c700 1 -- start start 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.614+0000 7f25b8f6c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b410c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.614+0000 7f25b8f6c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410cb40 con 0x7f25b407d540 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.614+0000 7f25b259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b410c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.614+0000 7f25b259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b410c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40154/0 (socket says 172.21.15.98:40154) 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.614+0000 7f25b259c700 1 -- 172.21.15.98:0/4108704979 learned_addr learned my addr 172.21.15.98:0/4108704979 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.615+0000 7f25b259c700 1 -- 172.21.15.98:0/4108704979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b410ccc0 con 0x7f25b407d540 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.615+0000 7f25b259c700 1 --2- 172.21.15.98:0/4108704979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b410c570 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f25a40098a0 tx=0x7f25a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f3dff30618ccce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.615+0000 7f25b159a700 1 -- 172.21.15.98:0/4108704979 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25a4004160 con 0x7f25b407d540 2024-04-25T16:22:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.615+0000 7f25b159a700 1 -- 172.21.15.98:0/4108704979 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25a4004300 con 0x7f25b407d540 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.615+0000 7f25b159a700 1 -- 172.21.15.98:0/4108704979 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25a4011960 con 0x7f25b407d540 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.616+0000 7f25b8f6c700 1 -- 172.21.15.98:0/4108704979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 msgr2=0x7f25b410c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.616+0000 7f25b8f6c700 1 --2- 172.21.15.98:0/4108704979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b410c570 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f25a40098a0 tx=0x7f25a4009bb0 comp rx=0 tx=0).stop 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.616+0000 7f25b8f6c700 1 -- 172.21.15.98:0/4108704979 shutdown_connections 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.616+0000 7f25b8f6c700 1 --2- 172.21.15.98:0/4108704979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b410c570 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.616+0000 7f25b8f6c700 1 -- 172.21.15.98:0/4108704979 >> 172.21.15.98:0/4108704979 conn(0x7f25b4103fe0 msgr2=0x7f25b4106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.616+0000 7f25b8f6c700 1 -- 172.21.15.98:0/4108704979 shutdown_connections 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.616+0000 7f25b8f6c700 1 -- 172.21.15.98:0/4108704979 wait complete. 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.617+0000 7f25b8f6c700 1 Processor -- start 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.617+0000 7f25b8f6c700 1 -- start start 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.617+0000 7f25b8f6c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:59.578 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.617+0000 7f25b8f6c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410dd20 con 0x7f25b407d540 2024-04-25T16:22:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.617+0000 7f25b259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.617+0000 7f25b259c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b419db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40168/0 (socket says 172.21.15.98:40168) 2024-04-25T16:22:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.617+0000 7f25b259c700 1 -- 172.21.15.98:0/1873434245 learned_addr learned my addr 172.21.15.98:0/1873434245 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:22:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.617+0000 7f25b259c700 1 -- 172.21.15.98:0/1873434245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b419e070 con 0x7f25b407d540 2024-04-25T16:22:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.618+0000 7f25b259c700 1 --2- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b419db30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f25a4003e50 tx=0x7f25a4011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.618+0000 7f25a37fe700 1 -- 172.21.15.98:0/1873434245 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25a4003a60 con 0x7f25b407d540 2024-04-25T16:22:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.618+0000 7f25a37fe700 1 -- 172.21.15.98:0/1873434245 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f25a401b8d0 con 0x7f25b407d540 2024-04-25T16:22:59.579 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.618+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b419e300 con 0x7f25b407d540 2024-04-25T16:22:59.580 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.618+0000 7f25a37fe700 1 -- 172.21.15.98:0/1873434245 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25a40237d0 con 0x7f25b407d540 2024-04-25T16:22:59.580 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.618+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b41a0f10 con 0x7f25b407d540 2024-04-25T16:22:59.580 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.619+0000 7f25a37fe700 1 -- 172.21.15.98:0/1873434245 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f25a402c3f0 con 0x7f25b407d540 2024-04-25T16:22:59.580 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.619+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b4064cf0 con 0x7f25b407d540 2024-04-25T16:22:59.580 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.619+0000 7f25a37fe700 1 --2- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f259c03d5e0 0x7f259c03faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:22:59.580 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.619+0000 7f25a37fe700 1 -- 172.21.15.98:0/1873434245 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f25a4052e60 con 0x7f25b407d540 2024-04-25T16:22:59.580 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.619+0000 7f25b1d9b700 1 --2- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f259c03d5e0 0x7f259c03faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:22:59.580 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.620+0000 7f25b1d9b700 1 --2- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f259c03d5e0 0x7f259c03faf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f25a8006fd0 tx=0x7f25a8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.625+0000 7f25a37fe700 1 -- 172.21.15.98:0/1873434245 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25a401f030 con 0x7f25b407d540 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.823+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f25b41a10e0 con 0x7f25b407d540 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.825+0000 7f25a37fe700 1 -- 172.21.15.98:0/1873434245 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f25a401bd80 con 0x7f25b407d540 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f259c03d5e0 msgr2=0x7f259c03faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 --2- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f259c03d5e0 0x7f259c03faf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f25a8006fd0 tx=0x7f25a8006e40 comp rx=0 tx=0).stop 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 msgr2=0x7f25b419db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 --2- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b419db30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f25a4003e50 tx=0x7f25a4011a30 comp rx=0 tx=0).stop 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 shutdown_connections 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 --2- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f259c03d5e0 0x7f259c03faf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 --2- 172.21.15.98:0/1873434245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b407d540 0x7f25b419db30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 >> 172.21.15.98:0/1873434245 conn(0x7f25b4103fe0 msgr2=0x7f25b4105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 shutdown_connections 2024-04-25T16:22:59.581 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:22:58.828+0000 7f25b8f6c700 1 -- 172.21.15.98:0/1873434245 wait complete. 2024-04-25T16:22:59.582 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling the dashboard module... 2024-04-25T16:23:00.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:59 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1873434245' entity='client.admin' 2024-04-25T16:23:00.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:22:59 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:23:01.150 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:00 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3143554640' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T16:23:01.150 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:00 smithi098 ceph-mon[29162]: from='mgr.14118 172.21.15.98:0/172079185' entity='mgr.x' 2024-04-25T16:23:01.151 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:00 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setuser ceph since I am not root 2024-04-25T16:23:01.151 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:00 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: ignoring --setgroup ceph since I am not root 2024-04-25T16:23:01.151 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:01.045+0000 7fb074ab1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:23:01.479 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:01.244+0000 7fb074ab1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:23:01.479 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:01.351+0000 7fb074ab1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:23:01.754 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:01.668+0000 7fb074ab1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:23:01.965 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.001+0000 7f9691455700 1 Processor -- start 2024-04-25T16:23:01.970 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.001+0000 7f9691455700 1 -- start start 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.002+0000 7f9691455700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c10c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.002+0000 7f9691455700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f968c10ca20 con 0x7f968c07ccb0 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.002+0000 7f968affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c10c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.002+0000 7f968affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c10c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40176/0 (socket says 172.21.15.98:40176) 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.002+0000 7f968affd700 1 -- 172.21.15.98:0/2380399748 learned_addr learned my addr 172.21.15.98:0/2380399748 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.002+0000 7f968affd700 1 -- 172.21.15.98:0/2380399748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c10cba0 con 0x7f968c07ccb0 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.002+0000 7f968affd700 1 --2- 172.21.15.98:0/2380399748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c10c450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96800098a0 tx=0x7f9680009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e373c033fad3e67a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.002+0000 7f9689ffb700 1 -- 172.21.15.98:0/2380399748 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9680004160 con 0x7f968c07ccb0 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.003+0000 7f9689ffb700 1 -- 172.21.15.98:0/2380399748 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9680004300 con 0x7f968c07ccb0 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.003+0000 7f9689ffb700 1 -- 172.21.15.98:0/2380399748 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9680011960 con 0x7f968c07ccb0 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.004+0000 7f9691455700 1 -- 172.21.15.98:0/2380399748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 msgr2=0x7f968c10c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:01.971 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.004+0000 7f9691455700 1 --2- 172.21.15.98:0/2380399748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c10c450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f96800098a0 tx=0x7f9680009bb0 comp rx=0 tx=0).stop 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.004+0000 7f9691455700 1 -- 172.21.15.98:0/2380399748 shutdown_connections 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.004+0000 7f9691455700 1 --2- 172.21.15.98:0/2380399748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c10c450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.004+0000 7f9691455700 1 -- 172.21.15.98:0/2380399748 >> 172.21.15.98:0/2380399748 conn(0x7f968c103ee0 msgr2=0x7f968c106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.004+0000 7f9691455700 1 -- 172.21.15.98:0/2380399748 shutdown_connections 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.004+0000 7f9691455700 1 -- 172.21.15.98:0/2380399748 wait complete. 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.005+0000 7f9691455700 1 Processor -- start 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.005+0000 7f9691455700 1 -- start start 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.005+0000 7f9691455700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c19db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.005+0000 7f9691455700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f968c10dc00 con 0x7f968c07ccb0 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.005+0000 7f968affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c19db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.005+0000 7f968affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c19db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40186/0 (socket says 172.21.15.98:40186) 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.005+0000 7f968affd700 1 -- 172.21.15.98:0/3143554640 learned_addr learned my addr 172.21.15.98:0/3143554640 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.006+0000 7f968affd700 1 -- 172.21.15.98:0/3143554640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c19e070 con 0x7f968c07ccb0 2024-04-25T16:23:01.972 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.006+0000 7f968affd700 1 --2- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c19db30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9680011670 tx=0x7f9680011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.006+0000 7f9673fff700 1 -- 172.21.15.98:0/3143554640 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9680003a60 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.006+0000 7f9673fff700 1 -- 172.21.15.98:0/3143554640 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f968001b8d0 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.006+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968c19e300 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.006+0000 7f9673fff700 1 -- 172.21.15.98:0/3143554640 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9680023790 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.006+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f968c1a0f10 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.007+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f968c064cf0 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.007+0000 7f9673fff700 1 -- 172.21.15.98:0/3143554640 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f9680023a40 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.007+0000 7f9673fff700 1 --2- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f967403d680 0x7f967403fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.007+0000 7f9673fff700 1 -- 172.21.15.98:0/3143554640 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9680052d40 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.007+0000 7f968a7fc700 1 --2- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f967403d680 0x7f967403fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.008+0000 7f968a7fc700 1 --2- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f967403d680 0x7f967403fb90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f967c006fd0 tx=0x7f967c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.013+0000 7f9673fff700 1 -- 172.21.15.98:0/3143554640 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f968001f030 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.281+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f968c1a1510 con 0x7f968c07ccb0 2024-04-25T16:23:01.973 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.829+0000 7f9673fff700 1 -- 172.21.15.98:0/3143554640 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f9680019e20 con 0x7f968c07ccb0 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.831+0000 7f9673fff700 1 -- 172.21.15.98:0/3143554640 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7f968002c3f0 con 0x7f968c07ccb0 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.831+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f967403d680 msgr2=0x7f967403fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 --2- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f967403d680 0x7f967403fb90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f967c006fd0 tx=0x7f967c006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 msgr2=0x7f968c19db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 --2- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c19db30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9680011670 tx=0x7f9680011a30 comp rx=0 tx=0).stop 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 shutdown_connections 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 --2- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7f967403d680 0x7f967403fb90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 --2- 172.21.15.98:0/3143554640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f968c07ccb0 0x7f968c19db30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 >> 172.21.15.98:0/3143554640 conn(0x7f968c103ee0 msgr2=0x7f968c1057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 shutdown_connections 2024-04-25T16:23:01.974 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:00.832+0000 7f9691455700 1 -- 172.21.15.98:0/3143554640 wait complete. 2024-04-25T16:23:02.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:01 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3143554640' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T16:23:02.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:01 smithi098 ceph-mon[29162]: mgrmap e9: x(active, since 19s) 2024-04-25T16:23:02.091 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:01.754+0000 7fb074ab1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:23:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:03 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2260862681' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T16:23:03.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:02 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:02.966+0000 7fb074ab1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:23:03.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:03.147+0000 7fb074ab1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:23:03.622 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:03.324+0000 7fb074ab1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:23:03.622 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:03.620+0000 7fb074ab1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:23:03.647 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:23:03.647 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T16:23:03.647 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T16:23:03.647 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T16:23:03.647 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.745+0000 7fa7bc82b700 1 Processor -- start 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.745+0000 7fa7bc82b700 1 -- start start 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.746+0000 7fa7bc82b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.746+0000 7fa7bc82b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b810b850 con 0x7fa7b810ae70 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.746+0000 7fa7b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.746+0000 7fa7b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40214/0 (socket says 172.21.15.98:40214) 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.746+0000 7fa7b659c700 1 -- 172.21.15.98:0/4107843451 learned_addr learned my addr 172.21.15.98:0/4107843451 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.746+0000 7fa7b659c700 1 -- 172.21.15.98:0/4107843451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b810b9d0 con 0x7fa7b810ae70 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.746+0000 7fa7b659c700 1 --2- 172.21.15.98:0/4107843451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810b280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80098a0 tx=0x7fa7a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02d3427eec50200 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.746+0000 7fa7b559a700 1 -- 172.21.15.98:0/4107843451 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8004160 con 0x7fa7b810ae70 2024-04-25T16:23:03.648 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.747+0000 7fa7b559a700 1 -- 172.21.15.98:0/4107843451 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7a8004300 con 0x7fa7b810ae70 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.747+0000 7fa7b559a700 1 -- 172.21.15.98:0/4107843451 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8011960 con 0x7fa7b810ae70 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.747+0000 7fa7bc82b700 1 -- 172.21.15.98:0/4107843451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 msgr2=0x7fa7b810b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.747+0000 7fa7bc82b700 1 --2- 172.21.15.98:0/4107843451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810b280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80098a0 tx=0x7fa7a8009bb0 comp rx=0 tx=0).stop 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.748+0000 7fa7bc82b700 1 -- 172.21.15.98:0/4107843451 shutdown_connections 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.748+0000 7fa7bc82b700 1 --2- 172.21.15.98:0/4107843451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810b280 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.748+0000 7fa7bc82b700 1 -- 172.21.15.98:0/4107843451 >> 172.21.15.98:0/4107843451 conn(0x7fa7b81035f0 msgr2=0x7fa7b8105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.748+0000 7fa7bc82b700 1 -- 172.21.15.98:0/4107843451 shutdown_connections 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.748+0000 7fa7bc82b700 1 -- 172.21.15.98:0/4107843451 wait complete. 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.748+0000 7fa7bc82b700 1 Processor -- start 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.748+0000 7fa7bc82b700 1 -- start start 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.749+0000 7fa7bc82b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.749+0000 7fa7bc82b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b8108700 con 0x7fa7b810ae70 2024-04-25T16:23:03.649 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.749+0000 7fa7b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.749+0000 7fa7b659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40222/0 (socket says 172.21.15.98:40222) 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.749+0000 7fa7b659c700 1 -- 172.21.15.98:0/2260862681 learned_addr learned my addr 172.21.15.98:0/2260862681 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.749+0000 7fa7b659c700 1 -- 172.21.15.98:0/2260862681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b810a8a0 con 0x7fa7b810ae70 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.749+0000 7fa7b659c700 1 --2- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810a360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8000c00 tx=0x7fa7a8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.749+0000 7fa7a77fe700 1 -- 172.21.15.98:0/2260862681 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8003b20 con 0x7fa7b810ae70 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.750+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b8108af0 con 0x7fa7b810ae70 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.750+0000 7fa7a77fe700 1 -- 172.21.15.98:0/2260862681 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7a801b8d0 con 0x7fa7b810ae70 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.750+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b8108fd0 con 0x7fa7b810ae70 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.750+0000 7fa7a77fe700 1 -- 172.21.15.98:0/2260862681 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a8023820 con 0x7fa7b810ae70 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.751+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa798005360 con 0x7fa7b810ae70 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.751+0000 7fa7a77fe700 1 -- 172.21.15.98:0/2260862681 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7fa7a802c3f0 con 0x7fa7b810ae70 2024-04-25T16:23:03.650 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.751+0000 7fa7a77fe700 1 --2- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fa7a003d6e0 0x7fa7a003fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.751+0000 7fa7a77fe700 1 -- 172.21.15.98:0/2260862681 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa7a8052bc0 con 0x7fa7b810ae70 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.751+0000 7fa7b5d9b700 1 -- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fa7a003d6e0 msgr2=0x7fa7a003fbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/217035597 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.751+0000 7fa7b5d9b700 1 --2- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fa7a003d6e0 0x7fa7a003fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.756+0000 7fa7a77fe700 1 -- 172.21.15.98:0/2260862681 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa7a801f030 con 0x7fa7b810ae70 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.952+0000 7fa7b5d9b700 1 -- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fa7a003d6e0 msgr2=0x7fa7a003fbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/217035597 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:02.952+0000 7fa7b5d9b700 1 --2- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fa7a003d6e0 0x7fa7a003fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.016+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa798005e60 con 0x7fa7b810ae70 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.017+0000 7fa7a77fe700 1 -- 172.21.15.98:0/2260862681 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fa7a80172b0 con 0x7fa7b810ae70 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fa7a003d6e0 msgr2=0x7fa7a003fbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 --2- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fa7a003d6e0 0x7fa7a003fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 msgr2=0x7fa7b810a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 --2- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810a360 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8000c00 tx=0x7fa7a8011a30 comp rx=0 tx=0).stop 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 shutdown_connections 2024-04-25T16:23:03.651 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 --2- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fa7a003d6e0 0x7fa7a003fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:03.652 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 --2- 172.21.15.98:0/2260862681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa7b810ae70 0x7fa7b810a360 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:03.652 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 >> 172.21.15.98:0/2260862681 conn(0x7fa7b81035f0 msgr2=0x7fa7b8103e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:03.652 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 shutdown_connections 2024-04-25T16:23:03.652 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:03.019+0000 7fa7bc82b700 1 -- 172.21.15.98:0/2260862681 wait complete. 2024-04-25T16:23:03.652 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for the mgr to restart... 2024-04-25T16:23:03.652 INFO:teuthology.orchestra.run.smithi098.stdout:Waiting for mgr epoch 9... 2024-04-25T16:23:03.892 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:03.723+0000 7fb074ab1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:23:04.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:03.933+0000 7fb074ab1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:23:04.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:04.137+0000 7fb074ab1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:23:05.004 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:04.739+0000 7fb074ab1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:23:05.477 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:05 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:05.171+0000 7fb074ab1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:23:05.477 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:05 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:05.269+0000 7fb074ab1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:23:05.477 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:05 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:05.476+0000 7fb074ab1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:23:06.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:06.000+0000 7fb074ab1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:23:06.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:06.096+0000 7fb074ab1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:23:06.699 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:06.345+0000 7fb074ab1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:23:06.699 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:06.481+0000 7fb074ab1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:23:06.699 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:06.579+0000 7fb074ab1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:23:06.699 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:06.698+0000 7fb074ab1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:23:07.254 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:06.998+0000 7fb074ab1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:23:07.885 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:07.604+0000 7fb074ab1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:23:07.885 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:23:07.827+0000 7fb074ab1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:23:08.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: Active manager daemon x restarted 2024-04-25T16:23:08.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: Activating manager daemon x 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: mgrmap e10: x(active, starting, since 0.0069401s) 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: Manager daemon x is now available 2024-04-25T16:23:08.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:09.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T16:23:09.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T16:23:09.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:08 smithi098 ceph-mon[29162]: mgrmap e11: x(active, since 1.00664s) 2024-04-25T16:23:09.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout { 2024-04-25T16:23:09.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T16:23:09.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T16:23:09.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout } 2024-04-25T16:23:09.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.125+0000 7fe120218700 1 Processor -- start 2024-04-25T16:23:09.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.125+0000 7fe120218700 1 -- start start 2024-04-25T16:23:09.616 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.125+0000 7fe120218700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe118108bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.125+0000 7fe120218700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181091e0 con 0x7fe1181087e0 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.125+0000 7fe11dfb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe118108bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.125+0000 7fe11dfb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe118108bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40238/0 (socket says 172.21.15.98:40238) 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.125+0000 7fe11dfb4700 1 -- 172.21.15.98:0/459684530 learned_addr learned my addr 172.21.15.98:0/459684530 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.126+0000 7fe11dfb4700 1 -- 172.21.15.98:0/459684530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe118109910 con 0x7fe1181087e0 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.126+0000 7fe11dfb4700 1 --2- 172.21.15.98:0/459684530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe118108bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe1140098a0 tx=0x7fe114009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d63e68d05db06a05 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.126+0000 7fe11cfb2700 1 -- 172.21.15.98:0/459684530 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe114004160 con 0x7fe1181087e0 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.126+0000 7fe11cfb2700 1 -- 172.21.15.98:0/459684530 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe114004300 con 0x7fe1181087e0 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.126+0000 7fe11cfb2700 1 -- 172.21.15.98:0/459684530 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe114011960 con 0x7fe1181087e0 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.127+0000 7fe120218700 1 -- 172.21.15.98:0/459684530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 msgr2=0x7fe118108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.127+0000 7fe120218700 1 --2- 172.21.15.98:0/459684530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe118108bf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe1140098a0 tx=0x7fe114009bb0 comp rx=0 tx=0).stop 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.127+0000 7fe120218700 1 -- 172.21.15.98:0/459684530 shutdown_connections 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.127+0000 7fe120218700 1 --2- 172.21.15.98:0/459684530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe118108bf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:09.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.127+0000 7fe120218700 1 -- 172.21.15.98:0/459684530 >> 172.21.15.98:0/459684530 conn(0x7fe118103ee0 msgr2=0x7fe118106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.127+0000 7fe120218700 1 -- 172.21.15.98:0/459684530 shutdown_connections 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.127+0000 7fe120218700 1 -- 172.21.15.98:0/459684530 wait complete. 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.128+0000 7fe120218700 1 Processor -- start 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.128+0000 7fe120218700 1 -- start start 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.128+0000 7fe120218700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe11819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.128+0000 7fe120218700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe11810ccc0 con 0x7fe1181087e0 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.128+0000 7fe11dfb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe11819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.128+0000 7fe11dfb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe11819ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40254/0 (socket says 172.21.15.98:40254) 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.128+0000 7fe11dfb4700 1 -- 172.21.15.98:0/1857767671 learned_addr learned my addr 172.21.15.98:0/1857767671 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.129+0000 7fe11dfb4700 1 -- 172.21.15.98:0/1857767671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11819cfc0 con 0x7fe1181087e0 2024-04-25T16:23:09.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.129+0000 7fe11dfb4700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe11819ca80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe114006af0 tx=0x7fe114011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.129+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe114003af0 con 0x7fe1181087e0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.129+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe11401b8d0 con 0x7fe1181087e0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.129+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11819d250 con 0x7fe1181087e0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.129+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe114023820 con 0x7fe1181087e0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.129+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11819fe60 con 0x7fe1181087e0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.130+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7fe11402c3f0 con 0x7fe1181087e0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.130+0000 7fe10effd700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 0x7fe10403fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.130+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 --> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe104040300 con 0x7fe10403d6e0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.130+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe114052d40 con 0x7fe1181087e0 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.130+0000 7fe11d7b3700 1 -- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 msgr2=0x7fe10403fbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/217035597 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.130+0000 7fe11d7b3700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 0x7fe10403fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.331+0000 7fe11d7b3700 1 -- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 msgr2=0x7fe10403fbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/217035597 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.331+0000 7fe11d7b3700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 0x7fe10403fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:23:09.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.732+0000 7fe11d7b3700 1 -- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 msgr2=0x7fe10403fbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/217035597 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:04.732+0000 7fe11d7b3700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 0x7fe10403fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:05.533+0000 7fe11d7b3700 1 -- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 msgr2=0x7fe10403fbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/217035597 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:05.533+0000 7fe11d7b3700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 0x7fe10403fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:07.134+0000 7fe11d7b3700 1 -- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 msgr2=0x7fe10403fbf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.98:6800/217035597 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:07.134+0000 7fe11d7b3700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 0x7fe10403fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:07.834+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mgrmap(e 10) ==== 49296+0+0 (secure 0 0 0) 0x7fe114018350 con 0x7fe1181087e0 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:07.835+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 msgr2=0x7fe10403fbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:07.835+0000 7fe10effd700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 0x7fe10403fbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.833+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7fe114054e40 con 0x7fe1181087e0 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.833+0000 7fe10effd700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe1140528d0 0x7fe104042fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.833+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe104040300 con 0x7fe1140528d0 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.833+0000 7fe11d7b3700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe1140528d0 0x7fe104042fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.834+0000 7fe11d7b3700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe1140528d0 0x7fe104042fe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe108003a10 tx=0x7fe108009270 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:09.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.835+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fe104040300 con 0x7fe1140528d0 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe118064cf0 con 0x7fe1140528d0 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe10effd700 1 -- 172.21.15.98:0/1857767671 <== mgr.14150 v2:172.21.15.98:6800/4283487583 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fe118064cf0 con 0x7fe1140528d0 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe1140528d0 msgr2=0x7fe104042fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe1140528d0 0x7fe104042fe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe108003a10 tx=0x7fe108009270 comp rx=0 tx=0).stop 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 msgr2=0x7fe11819ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe11819ca80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe114006af0 tx=0x7fe114011a30 comp rx=0 tx=0).stop 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 shutdown_connections 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe1140528d0 0x7fe104042fe0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe1181087e0 0x7fe11819ca80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 --2- 172.21.15.98:0/1857767671 >> [v2:172.21.15.98:6800/217035597,v1:172.21.15.98:6801/217035597] conn(0x7fe10403d6e0 0x7fe10403fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 >> 172.21.15.98:0/1857767671 conn(0x7fe118103ee0 msgr2=0x7fe118105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 shutdown_connections 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:08.843+0000 7fe120218700 1 -- 172.21.15.98:0/1857767671 wait complete. 2024-04-25T16:23:09.621 INFO:teuthology.orchestra.run.smithi098.stdout:mgr epoch 9 is available 2024-04-25T16:23:09.622 INFO:teuthology.orchestra.run.smithi098.stdout:Generating a dashboard self-signed certificate... 2024-04-25T16:23:10.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:09 smithi098 ceph-mon[29162]: [25/Apr/2024:16:23:08] ENGINE Bus STARTING 2024-04-25T16:23:10.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:09 smithi098 ceph-mon[29162]: [25/Apr/2024:16:23:08] ENGINE Serving on https://172.21.15.98:7150 2024-04-25T16:23:10.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:09 smithi098 ceph-mon[29162]: [25/Apr/2024:16:23:08] ENGINE Serving on http://172.21.15.98:8765 2024-04-25T16:23:10.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:09 smithi098 ceph-mon[29162]: [25/Apr/2024:16:23:08] ENGINE Bus STARTED 2024-04-25T16:23:10.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:09 smithi098 ceph-mon[29162]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T16:23:10.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:09 smithi098 ceph-mon[29162]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T16:23:10.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:11.106 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.060+0000 7fa69d0d3700 1 Processor -- start 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.060+0000 7fa69d0d3700 1 -- start start 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.060+0000 7fa69d0d3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69810c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.060+0000 7fa69d0d3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa69810ca20 con 0x7fa69807ccd0 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.061+0000 7fa696d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69810c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.061+0000 7fa696d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69810c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51496/0 (socket says 172.21.15.98:51496) 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.061+0000 7fa696d9d700 1 -- 172.21.15.98:0/421180958 learned_addr learned my addr 172.21.15.98:0/421180958 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.061+0000 7fa696d9d700 1 -- 172.21.15.98:0/421180958 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69810cba0 con 0x7fa69807ccd0 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.061+0000 7fa696d9d700 1 --2- 172.21.15.98:0/421180958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69810c430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa6880098a0 tx=0x7fa688009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6148f0ae7276643 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.061+0000 7fa695d9b700 1 -- 172.21.15.98:0/421180958 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa688004160 con 0x7fa69807ccd0 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.061+0000 7fa695d9b700 1 -- 172.21.15.98:0/421180958 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa688004300 con 0x7fa69807ccd0 2024-04-25T16:23:11.107 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.061+0000 7fa695d9b700 1 -- 172.21.15.98:0/421180958 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa688011960 con 0x7fa69807ccd0 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.062+0000 7fa69d0d3700 1 -- 172.21.15.98:0/421180958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 msgr2=0x7fa69810c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.062+0000 7fa69d0d3700 1 --2- 172.21.15.98:0/421180958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69810c430 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa6880098a0 tx=0x7fa688009bb0 comp rx=0 tx=0).stop 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.062+0000 7fa69d0d3700 1 -- 172.21.15.98:0/421180958 shutdown_connections 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.062+0000 7fa69d0d3700 1 --2- 172.21.15.98:0/421180958 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69810c430 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.062+0000 7fa69d0d3700 1 -- 172.21.15.98:0/421180958 >> 172.21.15.98:0/421180958 conn(0x7fa698103ee0 msgr2=0x7fa698106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.062+0000 7fa69d0d3700 1 -- 172.21.15.98:0/421180958 shutdown_connections 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.062+0000 7fa69d0d3700 1 -- 172.21.15.98:0/421180958 wait complete. 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.063+0000 7fa69d0d3700 1 Processor -- start 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.063+0000 7fa69d0d3700 1 -- start start 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.063+0000 7fa69d0d3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.063+0000 7fa69d0d3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa69810dc00 con 0x7fa69807ccd0 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.063+0000 7fa696d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:11.108 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.063+0000 7fa696d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69819db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51510/0 (socket says 172.21.15.98:51510) 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.063+0000 7fa696d9d700 1 -- 172.21.15.98:0/3541487378 learned_addr learned my addr 172.21.15.98:0/3541487378 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.064+0000 7fa696d9d700 1 -- 172.21.15.98:0/3541487378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69819e050 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.064+0000 7fa696d9d700 1 --2- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69819db10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa688011670 tx=0x7fa688011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.064+0000 7fa67ffff700 1 -- 172.21.15.98:0/3541487378 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa688003a60 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.064+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa69819e2e0 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.064+0000 7fa67ffff700 1 -- 172.21.15.98:0/3541487378 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa68801b8d0 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.064+0000 7fa67ffff700 1 -- 172.21.15.98:0/3541487378 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa688023790 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.064+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6981a0ef0 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.065+0000 7fa67ffff700 1 -- 172.21.15.98:0/3541487378 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7fa688023a40 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.065+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa698064cf0 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.065+0000 7fa67ffff700 1 --2- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa68003d5d0 0x7fa68003fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.066+0000 7fa67ffff700 1 -- 172.21.15.98:0/3541487378 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa68801bda0 con 0x7fa69807ccd0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.066+0000 7fa69659c700 1 --2- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa68003d5d0 0x7fa68003fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:11.109 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.066+0000 7fa69659c700 1 --2- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa68003d5d0 0x7fa68003fae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa68c006fd0 tx=0x7fa68c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.071+0000 7fa67ffff700 1 -- 172.21.15.98:0/3541487378 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa68801f030 con 0x7fa69807ccd0 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.284+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fa698105ec0 con 0x7fa68003d5d0 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.352+0000 7fa67ffff700 1 -- 172.21.15.98:0/3541487378 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fa698105ec0 con 0x7fa68003d5d0 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.354+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa68003d5d0 msgr2=0x7fa68003fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.354+0000 7fa69d0d3700 1 --2- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa68003d5d0 0x7fa68003fae0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa68c006fd0 tx=0x7fa68c006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.354+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 msgr2=0x7fa69819db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.354+0000 7fa69d0d3700 1 --2- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69819db10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa688011670 tx=0x7fa688011a30 comp rx=0 tx=0).stop 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.354+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 shutdown_connections 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.354+0000 7fa69d0d3700 1 --2- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa68003d5d0 0x7fa68003fae0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.354+0000 7fa69d0d3700 1 --2- 172.21.15.98:0/3541487378 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa69807ccd0 0x7fa69819db10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.354+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 >> 172.21.15.98:0/3541487378 conn(0x7fa698103ee0 msgr2=0x7fa6981057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.355+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 shutdown_connections 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:10.355+0000 7fa69d0d3700 1 -- 172.21.15.98:0/3541487378 wait complete. 2024-04-25T16:23:11.110 INFO:teuthology.orchestra.run.smithi098.stdout:Creating initial admin user... 2024-04-25T16:23:11.710 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:11 smithi098 ceph-mon[29162]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:11.710 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:11.710 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:11.710 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:11 smithi098 ceph-mon[29162]: mgrmap e12: x(active, since 2s) 2024-04-25T16:23:11.710 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:12.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:12 smithi098 ceph-mon[29162]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:12.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:12 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:13.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$yEFRKu6KzIgpaAi1HVo0HuDv7puUTks48zFb.w2vD7Lw.6E9lxRE6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714062192, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T16:23:13.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.558+0000 7f1680f09700 1 Processor -- start 2024-04-25T16:23:13.187 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.558+0000 7f1680f09700 1 -- start start 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.559+0000 7f1680f09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c10c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.559+0000 7f1680f09700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f167c10ca20 con 0x7f167c07ccd0 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.559+0000 7f167a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c10c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.559+0000 7f167a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c10c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51524/0 (socket says 172.21.15.98:51524) 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.559+0000 7f167a59c700 1 -- 172.21.15.98:0/1192646914 learned_addr learned my addr 172.21.15.98:0/1192646914 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.559+0000 7f167a59c700 1 -- 172.21.15.98:0/1192646914 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c10cba0 con 0x7f167c07ccd0 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.559+0000 7f167a59c700 1 --2- 172.21.15.98:0/1192646914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c10c430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f166c0098a0 tx=0x7f166c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5a981a3808a8a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.560+0000 7f167959a700 1 -- 172.21.15.98:0/1192646914 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c004160 con 0x7f167c07ccd0 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.560+0000 7f167959a700 1 -- 172.21.15.98:0/1192646914 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f166c004300 con 0x7f167c07ccd0 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.560+0000 7f167959a700 1 -- 172.21.15.98:0/1192646914 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c011960 con 0x7f167c07ccd0 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.561+0000 7f1680f09700 1 -- 172.21.15.98:0/1192646914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 msgr2=0x7f167c10c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.561+0000 7f1680f09700 1 --2- 172.21.15.98:0/1192646914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c10c430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f166c0098a0 tx=0x7f166c009bb0 comp rx=0 tx=0).stop 2024-04-25T16:23:13.188 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.561+0000 7f1680f09700 1 -- 172.21.15.98:0/1192646914 shutdown_connections 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.561+0000 7f1680f09700 1 --2- 172.21.15.98:0/1192646914 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c10c430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.561+0000 7f1680f09700 1 -- 172.21.15.98:0/1192646914 >> 172.21.15.98:0/1192646914 conn(0x7f167c103ee0 msgr2=0x7f167c106330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.561+0000 7f1680f09700 1 -- 172.21.15.98:0/1192646914 shutdown_connections 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.561+0000 7f1680f09700 1 -- 172.21.15.98:0/1192646914 wait complete. 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.562+0000 7f1680f09700 1 Processor -- start 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.562+0000 7f1680f09700 1 -- start start 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f1680f09700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c19db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f1680f09700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f167c10dc00 con 0x7f167c07ccd0 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f167a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c19db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f167a59c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c19db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51536/0 (socket says 172.21.15.98:51536) 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f167a59c700 1 -- 172.21.15.98:0/2689200567 learned_addr learned my addr 172.21.15.98:0/2689200567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f167a59c700 1 -- 172.21.15.98:0/2689200567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c19e070 con 0x7f167c07ccd0 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f167a59c700 1 --2- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c19db30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f166c011670 tx=0x7f166c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:13.189 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f166b7fe700 1 -- 172.21.15.98:0/2689200567 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c003a60 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f166b7fe700 1 -- 172.21.15.98:0/2689200567 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f166c01b8d0 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.563+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c19e300 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.564+0000 7f166b7fe700 1 -- 172.21.15.98:0/2689200567 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f166c023750 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.564+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c1a0f10 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.565+0000 7f166b7fe700 1 -- 172.21.15.98:0/2689200567 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f166c023a40 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.565+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167c064cf0 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.565+0000 7f166b7fe700 1 --2- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f166403d690 0x7f166403fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.565+0000 7f166b7fe700 1 -- 172.21.15.98:0/2689200567 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f166c053d60 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.565+0000 7f1679d9b700 1 --2- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f166403d690 0x7f166403fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.566+0000 7f1679d9b700 1 --2- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f166403d690 0x7f166403fba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1670006fd0 tx=0x7f1670006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.575+0000 7f166b7fe700 1 -- 172.21.15.98:0/2689200567 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f166c01f030 con 0x7f167c07ccd0 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:11.799+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f167c105f10 con 0x7f166403d690 2024-04-25T16:23:13.190 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.023+0000 7f166b7fe700 1 -- 172.21.15.98:0/2689200567 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f167c105f10 con 0x7f166403d690 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.025+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f166403d690 msgr2=0x7f166403fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.025+0000 7f1680f09700 1 --2- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f166403d690 0x7f166403fba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1670006fd0 tx=0x7f1670006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.025+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 msgr2=0x7f167c19db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.026+0000 7f1680f09700 1 --2- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c19db30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f166c011670 tx=0x7f166c011a30 comp rx=0 tx=0).stop 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.026+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 shutdown_connections 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.026+0000 7f1680f09700 1 --2- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f166403d690 0x7f166403fba0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.026+0000 7f1680f09700 1 --2- 172.21.15.98:0/2689200567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f167c07ccd0 0x7f167c19db30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.026+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 >> 172.21.15.98:0/2689200567 conn(0x7f167c103ee0 msgr2=0x7f167c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.026+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 shutdown_connections 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:12.026+0000 7f1680f09700 1 -- 172.21.15.98:0/2689200567 wait complete. 2024-04-25T16:23:13.191 INFO:teuthology.orchestra.run.smithi098.stdout:Fetching dashboard port number... 2024-04-25T16:23:14.119 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:13 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1964978433' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T16:23:14.119 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:13 smithi098 ceph-mon[29162]: mgrmap e13: x(active, since 6s) 2024-04-25T16:23:14.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T16:23:14.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.633+0000 7f600ccde700 1 Processor -- start 2024-04-25T16:23:14.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.633+0000 7f600ccde700 1 -- start start 2024-04-25T16:23:14.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.633+0000 7f600ccde700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600807daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:14.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.633+0000 7f600ccde700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f600807e030 con 0x7f600807f620 2024-04-25T16:23:14.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.633+0000 7f600659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600807daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:14.377 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.633+0000 7f600659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600807daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51550/0 (socket says 172.21.15.98:51550) 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.633+0000 7f600659c700 1 -- 172.21.15.98:0/989235717 learned_addr learned my addr 172.21.15.98:0/989235717 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.634+0000 7f600659c700 1 -- 172.21.15.98:0/989235717 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600807e1b0 con 0x7f600807f620 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.634+0000 7f600659c700 1 --2- 172.21.15.98:0/989235717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600807daf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80098a0 tx=0x7f5ff8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f731a2d85ab1abc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.634+0000 7f600559a700 1 -- 172.21.15.98:0/989235717 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ff8004160 con 0x7f600807f620 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.634+0000 7f600559a700 1 -- 172.21.15.98:0/989235717 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ff8004300 con 0x7f600807f620 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.634+0000 7f600559a700 1 -- 172.21.15.98:0/989235717 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ff8011960 con 0x7f600807f620 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.635+0000 7f600ccde700 1 -- 172.21.15.98:0/989235717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 msgr2=0x7f600807daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.635+0000 7f600ccde700 1 --2- 172.21.15.98:0/989235717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600807daf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5ff80098a0 tx=0x7f5ff8009bb0 comp rx=0 tx=0).stop 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.635+0000 7f600ccde700 1 -- 172.21.15.98:0/989235717 shutdown_connections 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.635+0000 7f600ccde700 1 --2- 172.21.15.98:0/989235717 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600807daf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.635+0000 7f600ccde700 1 -- 172.21.15.98:0/989235717 >> 172.21.15.98:0/989235717 conn(0x7f60081050e0 msgr2=0x7f6008107530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.635+0000 7f600ccde700 1 -- 172.21.15.98:0/989235717 shutdown_connections 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.635+0000 7f600ccde700 1 -- 172.21.15.98:0/989235717 wait complete. 2024-04-25T16:23:14.378 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600ccde700 1 Processor -- start 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600ccde700 1 -- start start 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600ccde700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600819fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600ccde700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f600810ff60 con 0x7f600807f620 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600819fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600659c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600819fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51566/0 (socket says 172.21.15.98:51566) 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600659c700 1 -- 172.21.15.98:0/1964978433 learned_addr learned my addr 172.21.15.98:0/1964978433 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600659c700 1 -- 172.21.15.98:0/1964978433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60081a0290 con 0x7f600807f620 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f600659c700 1 --2- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600819fd50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8011670 tx=0x7f5ff8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.636+0000 7f5ff77fe700 1 -- 172.21.15.98:0/1964978433 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ff8003a60 con 0x7f600807f620 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.637+0000 7f5ff77fe700 1 -- 172.21.15.98:0/1964978433 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5ff801b8d0 con 0x7f600807f620 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.637+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60081a0520 con 0x7f600807f620 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.637+0000 7f5ff77fe700 1 -- 172.21.15.98:0/1964978433 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5ff8023790 con 0x7f600807f620 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.637+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60081a3120 con 0x7f600807f620 2024-04-25T16:23:14.379 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.637+0000 7f5ff77fe700 1 -- 172.21.15.98:0/1964978433 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f5ff8023a40 con 0x7f600807f620 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.637+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6008064cf0 con 0x7f600807f620 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.638+0000 7f5ff77fe700 1 --2- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff003d6e0 0x7f5ff003fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.638+0000 7f5ff77fe700 1 -- 172.21.15.98:0/1964978433 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5ff8052e40 con 0x7f600807f620 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.638+0000 7f6005d9b700 1 --2- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff003d6e0 0x7f5ff003fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.638+0000 7f6005d9b700 1 --2- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff003d6e0 0x7f5ff003fbf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc006fd0 tx=0x7f5ffc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.643+0000 7f5ff77fe700 1 -- 172.21.15.98:0/1964978433 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5ff801f030 con 0x7f600807f620 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.840+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f60081a3410 con 0x7f600807f620 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.841+0000 7f5ff77fe700 1 -- 172.21.15.98:0/1964978433 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f5ff8019e20 con 0x7f600807f620 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff003d6e0 msgr2=0x7f5ff003fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 --2- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff003d6e0 0x7f5ff003fbf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc006fd0 tx=0x7f5ffc006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 msgr2=0x7f600819fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 --2- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600819fd50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8011670 tx=0x7f5ff8011a30 comp rx=0 tx=0).stop 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 shutdown_connections 2024-04-25T16:23:14.380 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 --2- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff003d6e0 0x7f5ff003fbf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 --2- 172.21.15.98:0/1964978433 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f600807f620 0x7f600819fd50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 >> 172.21.15.98:0/1964978433 conn(0x7f60081050e0 msgr2=0x7f6008106a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 shutdown_connections 2024-04-25T16:23:14.381 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:13.843+0000 7f600ccde700 1 -- 172.21.15.98:0/1964978433 wait complete. 2024-04-25T16:23:14.383 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T16:23:14.383 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout disabled 2024-04-25T16:23:14.390 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T16:23:14.390 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stdout inactive 2024-04-25T16:23:14.390 INFO:teuthology.orchestra.run.smithi098.stdout:firewalld.service is not enabled 2024-04-25T16:23:14.390 INFO:teuthology.orchestra.run.smithi098.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T16:23:14.391 INFO:teuthology.orchestra.run.smithi098.stdout:Ceph Dashboard is now available at: 2024-04-25T16:23:14.391 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: URL: https://smithi098.front.sepia.ceph.com:8443/ 2024-04-25T16:23:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: User: admin 2024-04-25T16:23:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: Password: 2crwm06zvt 2024-04-25T16:23:14.392 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:14.392 INFO:teuthology.orchestra.run.smithi098.stdout:Saving cluster configuration to /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config directory 2024-04-25T16:23:14.392 INFO:teuthology.orchestra.run.smithi098.stdout:Enabling autotune for osd_memory_target 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.833+0000 7f49b36f4700 1 Processor -- start 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.833+0000 7f49b36f4700 1 -- start start 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49b36f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac10c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49b36f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac10cb30 con 0x7f49ac07d700 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49b1490700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac10c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49b1490700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac10c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51570/0 (socket says 172.21.15.98:51570) 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49b1490700 1 -- 172.21.15.98:0/4036194484 learned_addr learned my addr 172.21.15.98:0/4036194484 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49b1490700 1 -- 172.21.15.98:0/4036194484 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac10ccb0 con 0x7f49ac07d700 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49b1490700 1 --2- 172.21.15.98:0/4036194484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac10c540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f49a80098a0 tx=0x7f49a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408911004b5332cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49a3fff700 1 -- 172.21.15.98:0/4036194484 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a8004160 con 0x7f49ac07d700 2024-04-25T16:23:15.617 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49a3fff700 1 -- 172.21.15.98:0/4036194484 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f49a8004300 con 0x7f49ac07d700 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.834+0000 7f49a3fff700 1 -- 172.21.15.98:0/4036194484 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a8011960 con 0x7f49ac07d700 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.835+0000 7f49b36f4700 1 -- 172.21.15.98:0/4036194484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 msgr2=0x7f49ac10c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.835+0000 7f49b36f4700 1 --2- 172.21.15.98:0/4036194484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac10c540 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f49a80098a0 tx=0x7f49a8009bb0 comp rx=0 tx=0).stop 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.835+0000 7f49b36f4700 1 -- 172.21.15.98:0/4036194484 shutdown_connections 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.835+0000 7f49b36f4700 1 --2- 172.21.15.98:0/4036194484 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac10c540 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.835+0000 7f49b36f4700 1 -- 172.21.15.98:0/4036194484 >> 172.21.15.98:0/4036194484 conn(0x7f49ac103f50 msgr2=0x7f49ac1063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.835+0000 7f49b36f4700 1 -- 172.21.15.98:0/4036194484 shutdown_connections 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.835+0000 7f49b36f4700 1 -- 172.21.15.98:0/4036194484 wait complete. 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.836+0000 7f49b36f4700 1 Processor -- start 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.836+0000 7f49b36f4700 1 -- start start 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.836+0000 7f49b36f4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac19dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.836+0000 7f49b36f4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac10dd10 con 0x7f49ac07d700 2024-04-25T16:23:15.618 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.836+0000 7f49b1490700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac19dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.836+0000 7f49b1490700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac19dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:51574/0 (socket says 172.21.15.98:51574) 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.836+0000 7f49b1490700 1 -- 172.21.15.98:0/934049588 learned_addr learned my addr 172.21.15.98:0/934049588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.836+0000 7f49b1490700 1 -- 172.21.15.98:0/934049588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac19e1a0 con 0x7f49ac07d700 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.837+0000 7f49b1490700 1 --2- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac19dc60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f49a8011670 tx=0x7f49a8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.837+0000 7f49a27fc700 1 -- 172.21.15.98:0/934049588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a8003a60 con 0x7f49ac07d700 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.837+0000 7f49a27fc700 1 -- 172.21.15.98:0/934049588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f49a801b8d0 con 0x7f49ac07d700 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.837+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ac19e430 con 0x7f49ac07d700 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.837+0000 7f49a27fc700 1 -- 172.21.15.98:0/934049588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49a8023750 con 0x7f49ac07d700 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.837+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ac1a1040 con 0x7f49ac07d700 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.838+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ac064cf0 con 0x7f49ac07d700 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.838+0000 7f49a27fc700 1 -- 172.21.15.98:0/934049588 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f49a8023a50 con 0x7f49ac07d700 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.838+0000 7f49a27fc700 1 --2- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f499803d6d0 0x7f499803fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:15.619 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.838+0000 7f49a27fc700 1 -- 172.21.15.98:0/934049588 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f49a8052f50 con 0x7f49ac07d700 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.838+0000 7f49b0c8f700 1 --2- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f499803d6d0 0x7f499803fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.839+0000 7f49b0c8f700 1 --2- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f499803d6d0 0x7f499803fbe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f499c006fd0 tx=0x7f499c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:14.843+0000 7f49a27fc700 1 -- 172.21.15.98:0/934049588 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49a801f030 con 0x7f49ac07d700 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.043+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f49ac1a1210 con 0x7f49ac07d700 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.043+0000 7f49a27fc700 1 -- 172.21.15.98:0/934049588 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f49a800b3e0 con 0x7f49ac07d700 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f499803d6d0 msgr2=0x7f499803fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 --2- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f499803d6d0 0x7f499803fbe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f499c006fd0 tx=0x7f499c006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 msgr2=0x7f49ac19dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 --2- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac19dc60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f49a8011670 tx=0x7f49a8011a30 comp rx=0 tx=0).stop 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 shutdown_connections 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 --2- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f499803d6d0 0x7f499803fbe0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 --2- 172.21.15.98:0/934049588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49ac07d700 0x7f49ac19dc60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 >> 172.21.15.98:0/934049588 conn(0x7f49ac103f50 msgr2=0x7f49ac104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:15.620 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 shutdown_connections 2024-04-25T16:23:15.621 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:15.045+0000 7f49b36f4700 1 -- 172.21.15.98:0/934049588 wait complete. 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.047+0000 7f833dfe4700 1 Processor -- start 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.047+0000 7f833dfe4700 1 -- start start 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.047+0000 7f833dfe4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f8338109d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.047+0000 7f833dfe4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f833810a370 con 0x7f8338109970 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.048+0000 7f83377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f8338109d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.048+0000 7f83377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f8338109d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47722/0 (socket says 172.21.15.98:47722) 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.048+0000 7f83377fe700 1 -- 172.21.15.98:0/2642947917 learned_addr learned my addr 172.21.15.98:0/2642947917 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.048+0000 7f83377fe700 1 -- 172.21.15.98:0/2642947917 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833810aaa0 con 0x7f8338109970 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.048+0000 7f83377fe700 1 --2- 172.21.15.98:0/2642947917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f8338109d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f83280098a0 tx=0x7f8328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45100e52adf9a8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:16.927 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.048+0000 7f8336ffd700 1 -- 172.21.15.98:0/2642947917 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8328004160 con 0x7f8338109970 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.048+0000 7f8336ffd700 1 -- 172.21.15.98:0/2642947917 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8328004300 con 0x7f8338109970 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.048+0000 7f8336ffd700 1 -- 172.21.15.98:0/2642947917 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8328011960 con 0x7f8338109970 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.049+0000 7f833dfe4700 1 -- 172.21.15.98:0/2642947917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 msgr2=0x7f8338109d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.049+0000 7f833dfe4700 1 --2- 172.21.15.98:0/2642947917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f8338109d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f83280098a0 tx=0x7f8328009bb0 comp rx=0 tx=0).stop 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.049+0000 7f833dfe4700 1 -- 172.21.15.98:0/2642947917 shutdown_connections 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.049+0000 7f833dfe4700 1 --2- 172.21.15.98:0/2642947917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f8338109d80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.049+0000 7f833dfe4700 1 -- 172.21.15.98:0/2642947917 >> 172.21.15.98:0/2642947917 conn(0x7f8338105010 msgr2=0x7f8338107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.049+0000 7f833dfe4700 1 -- 172.21.15.98:0/2642947917 shutdown_connections 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.049+0000 7f833dfe4700 1 -- 172.21.15.98:0/2642947917 wait complete. 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f833dfe4700 1 Processor -- start 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f833dfe4700 1 -- start start 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f833dfe4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f833819dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f833dfe4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f833810de50 con 0x7f8338109970 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f83377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f833819dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:16.928 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f83377fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f833819dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47726/0 (socket says 172.21.15.98:47726) 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f83377fe700 1 -- 172.21.15.98:0/3846148430 learned_addr learned my addr 172.21.15.98:0/3846148430 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f83377fe700 1 -- 172.21.15.98:0/3846148430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833819e150 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.050+0000 7f83377fe700 1 --2- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f833819dc10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f83280119b0 tx=0x7f8328011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.051+0000 7f83357fa700 1 -- 172.21.15.98:0/3846148430 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8328003aa0 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.051+0000 7f83357fa700 1 -- 172.21.15.98:0/3846148430 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f832801b8d0 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.051+0000 7f83357fa700 1 -- 172.21.15.98:0/3846148430 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8328023860 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.051+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f833819e3e0 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.051+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83381a0ff0 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.052+0000 7f83357fa700 1 -- 172.21.15.98:0/3846148430 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f832802c3f0 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.052+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8338064cf0 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.052+0000 7f83357fa700 1 --2- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f832403d730 0x7f832403fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.052+0000 7f83357fa700 1 -- 172.21.15.98:0/3846148430 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8328052f50 con 0x7f8338109970 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.052+0000 7f832effd700 1 --2- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f832403d730 0x7f832403fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:16.929 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.052+0000 7f832effd700 1 --2- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f832403d730 0x7f832403fc40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8320006fd0 tx=0x7f8320006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.057+0000 7f83357fa700 1 -- 172.21.15.98:0/3846148430 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f832801f030 con 0x7f8338109970 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.363+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f83381a11c0 con 0x7f8338109970 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.366+0000 7f83357fa700 1 -- 172.21.15.98:0/3846148430 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f832801baf0 con 0x7f8338109970 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f832403d730 msgr2=0x7f832403fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 --2- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f832403d730 0x7f832403fc40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8320006fd0 tx=0x7f8320006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 msgr2=0x7f833819dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 --2- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f833819dc10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f83280119b0 tx=0x7f8328011a90 comp rx=0 tx=0).stop 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 shutdown_connections 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 --2- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f832403d730 0x7f832403fc40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 --2- 172.21.15.98:0/3846148430 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8338109970 0x7f833819dc10 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 >> 172.21.15.98:0/3846148430 conn(0x7f8338105010 msgr2=0x7f8338105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 shutdown_connections 2024-04-25T16:23:16.930 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr 2024-04-25T16:23:16.368+0000 7f833dfe4700 1 -- 172.21.15.98:0/3846148430 wait complete. 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: ceph telemetry on 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout:For more information see: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:16.931 INFO:teuthology.orchestra.run.smithi098.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T16:23:16.932 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:16.932 INFO:teuthology.orchestra.run.smithi098.stdout:Bootstrap complete. 2024-04-25T16:23:16.962 INFO:tasks.cephadm:Fetching config... 2024-04-25T16:23:16.962 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:23:16.962 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T16:23:16.977 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T16:23:16.977 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:23:16.977 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T16:23:17.032 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T16:23:17.032 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:23:17.032 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T16:23:17.096 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T16:23:17.096 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:23:17.096 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T16:23:17.151 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T16:23:17.151 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAC/RVSqSaEOy9NgdSFWdOb8xovoBYy0xK5zmBzzMo2wHbDZnXXYyFOv66PhC7Mj3Jqr/Jczme8AA28cQtkAzxxAhH2ftobP0n/Ru5LSBFIlTrKMHUmxFwcCj3pvCCLJwnRINJqksu3AMteGcg2lVT2f7VHeIEfDECWCv3WiwSJVcleYK4dybRhiol2bZ/k9fPnnjxUQ/U/CpiUYnLBOsI8lvMGKefsOEb+K8U8InURtCuQzrJwS0LOU/V92w2EU9usXcwy88EL/ZG+9R59iDB/s+eQLkp9zlCb2rF54aNrnAcllWwX8B9h7p2IndASJ8W5U7eMd5qZkBCbs49Bl33ENnFLIHdYH0FWMegdAic3moGXVecCoCrecgDp7Q8aK3KJ/QxHfajATDfX65UC+hdnkyQnIZ5/RnFFcJ7NhmOPD0DtNd+kkOts/LO+zxZX1vGYxUo7MPVcHbcypjuo2n7140Vp8YTBv+pC4j0xkeAm3YezlmUNPzB7Gc5nNJDN8= ceph-c520eaa8-031f-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T16:23:17.224 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAC/RVSqSaEOy9NgdSFWdOb8xovoBYy0xK5zmBzzMo2wHbDZnXXYyFOv66PhC7Mj3Jqr/Jczme8AA28cQtkAzxxAhH2ftobP0n/Ru5LSBFIlTrKMHUmxFwcCj3pvCCLJwnRINJqksu3AMteGcg2lVT2f7VHeIEfDECWCv3WiwSJVcleYK4dybRhiol2bZ/k9fPnnjxUQ/U/CpiUYnLBOsI8lvMGKefsOEb+K8U8InURtCuQzrJwS0LOU/V92w2EU9usXcwy88EL/ZG+9R59iDB/s+eQLkp9zlCb2rF54aNrnAcllWwX8B9h7p2IndASJ8W5U7eMd5qZkBCbs49Bl33ENnFLIHdYH0FWMegdAic3moGXVecCoCrecgDp7Q8aK3KJ/QxHfajATDfX65UC+hdnkyQnIZ5/RnFFcJ7NhmOPD0DtNd+kkOts/LO+zxZX1vGYxUo7MPVcHbcypjuo2n7140Vp8YTBv+pC4j0xkeAm3YezlmUNPzB7Gc5nNJDN8= ceph-c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:23:17.234 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAC/RVSqSaEOy9NgdSFWdOb8xovoBYy0xK5zmBzzMo2wHbDZnXXYyFOv66PhC7Mj3Jqr/Jczme8AA28cQtkAzxxAhH2ftobP0n/Ru5LSBFIlTrKMHUmxFwcCj3pvCCLJwnRINJqksu3AMteGcg2lVT2f7VHeIEfDECWCv3WiwSJVcleYK4dybRhiol2bZ/k9fPnnjxUQ/U/CpiUYnLBOsI8lvMGKefsOEb+K8U8InURtCuQzrJwS0LOU/V92w2EU9usXcwy88EL/ZG+9R59iDB/s+eQLkp9zlCb2rF54aNrnAcllWwX8B9h7p2IndASJ8W5U7eMd5qZkBCbs49Bl33ENnFLIHdYH0FWMegdAic3moGXVecCoCrecgDp7Q8aK3KJ/QxHfajATDfX65UC+hdnkyQnIZ5/RnFFcJ7NhmOPD0DtNd+kkOts/LO+zxZX1vGYxUo7MPVcHbcypjuo2n7140Vp8YTBv+pC4j0xkeAm3YezlmUNPzB7Gc5nNJDN8= ceph-c520eaa8-031f-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T16:23:17.273 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAC/RVSqSaEOy9NgdSFWdOb8xovoBYy0xK5zmBzzMo2wHbDZnXXYyFOv66PhC7Mj3Jqr/Jczme8AA28cQtkAzxxAhH2ftobP0n/Ru5LSBFIlTrKMHUmxFwcCj3pvCCLJwnRINJqksu3AMteGcg2lVT2f7VHeIEfDECWCv3WiwSJVcleYK4dybRhiol2bZ/k9fPnnjxUQ/U/CpiUYnLBOsI8lvMGKefsOEb+K8U8InURtCuQzrJwS0LOU/V92w2EU9usXcwy88EL/ZG+9R59iDB/s+eQLkp9zlCb2rF54aNrnAcllWwX8B9h7p2IndASJ8W5U7eMd5qZkBCbs49Bl33ENnFLIHdYH0FWMegdAic3moGXVecCoCrecgDp7Q8aK3KJ/QxHfajATDfX65UC+hdnkyQnIZ5/RnFFcJ7NhmOPD0DtNd+kkOts/LO+zxZX1vGYxUo7MPVcHbcypjuo2n7140Vp8YTBv+pC4j0xkeAm3YezlmUNPzB7Gc5nNJDN8= ceph-c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:23:17.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAC/RVSqSaEOy9NgdSFWdOb8xovoBYy0xK5zmBzzMo2wHbDZnXXYyFOv66PhC7Mj3Jqr/Jczme8AA28cQtkAzxxAhH2ftobP0n/Ru5LSBFIlTrKMHUmxFwcCj3pvCCLJwnRINJqksu3AMteGcg2lVT2f7VHeIEfDECWCv3WiwSJVcleYK4dybRhiol2bZ/k9fPnnjxUQ/U/CpiUYnLBOsI8lvMGKefsOEb+K8U8InURtCuQzrJwS0LOU/V92w2EU9usXcwy88EL/ZG+9R59iDB/s+eQLkp9zlCb2rF54aNrnAcllWwX8B9h7p2IndASJ8W5U7eMd5qZkBCbs49Bl33ENnFLIHdYH0FWMegdAic3moGXVecCoCrecgDp7Q8aK3KJ/QxHfajATDfX65UC+hdnkyQnIZ5/RnFFcJ7NhmOPD0DtNd+kkOts/LO+zxZX1vGYxUo7MPVcHbcypjuo2n7140Vp8YTBv+pC4j0xkeAm3YezlmUNPzB7Gc5nNJDN8= ceph-c520eaa8-031f-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T16:23:17.327 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJAC/RVSqSaEOy9NgdSFWdOb8xovoBYy0xK5zmBzzMo2wHbDZnXXYyFOv66PhC7Mj3Jqr/Jczme8AA28cQtkAzxxAhH2ftobP0n/Ru5LSBFIlTrKMHUmxFwcCj3pvCCLJwnRINJqksu3AMteGcg2lVT2f7VHeIEfDECWCv3WiwSJVcleYK4dybRhiol2bZ/k9fPnnjxUQ/U/CpiUYnLBOsI8lvMGKefsOEb+K8U8InURtCuQzrJwS0LOU/V92w2EU9usXcwy88EL/ZG+9R59iDB/s+eQLkp9zlCb2rF54aNrnAcllWwX8B9h7p2IndASJ8W5U7eMd5qZkBCbs49Bl33ENnFLIHdYH0FWMegdAic3moGXVecCoCrecgDp7Q8aK3KJ/QxHfajATDfX65UC+hdnkyQnIZ5/RnFFcJ7NhmOPD0DtNd+kkOts/LO+zxZX1vGYxUo7MPVcHbcypjuo2n7140Vp8YTBv+pC4j0xkeAm3YezlmUNPzB7Gc5nNJDN8= ceph-c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:23:17.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T16:23:17.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:17 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3846148430' entity='client.admin' 2024-04-25T16:23:17.576 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:20.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:20.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:20.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:20.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:20.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:20.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:20.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.203+0000 7fe318486700 1 -- 172.21.15.98:0/1490468100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 msgr2=0x7fe310104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.203+0000 7fe318486700 1 --2- 172.21.15.98:0/1490468100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 0x7fe310104360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009910 tx=0x7fe30c009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.203+0000 7fe318486700 1 -- 172.21.15.98:0/1490468100 shutdown_connections 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.203+0000 7fe318486700 1 --2- 172.21.15.98:0/1490468100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 0x7fe310104360 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.203+0000 7fe318486700 1 -- 172.21.15.98:0/1490468100 >> 172.21.15.98:0/1490468100 conn(0x7fe3100fd730 msgr2=0x7fe3100ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.203+0000 7fe318486700 1 -- 172.21.15.98:0/1490468100 shutdown_connections 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.203+0000 7fe318486700 1 -- 172.21.15.98:0/1490468100 wait complete. 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.204+0000 7fe318486700 1 Processor -- start 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.204+0000 7fe318486700 1 -- start start 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.204+0000 7fe318486700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 0x7fe3101024e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.204+0000 7fe318486700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101087e0 con 0x7fe310103f50 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.204+0000 7fe316222700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 0x7fe3101024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.204+0000 7fe316222700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 0x7fe3101024e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47742/0 (socket says 172.21.15.98:47742) 2024-04-25T16:23:20.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.204+0000 7fe316222700 1 -- 172.21.15.98:0/881989499 learned_addr learned my addr 172.21.15.98:0/881989499 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:20.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.205+0000 7fe316222700 1 -- 172.21.15.98:0/881989499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe310102a20 con 0x7fe310103f50 2024-04-25T16:23:20.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.205+0000 7fe316222700 1 --2- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 0x7fe3101024e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe30c005760 tx=0x7fe30c004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:20.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.205+0000 7fe3077fe700 1 -- 172.21.15.98:0/881989499 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe30c027070 con 0x7fe310103f50 2024-04-25T16:23:20.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.205+0000 7fe3077fe700 1 -- 172.21.15.98:0/881989499 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe30c00bdf0 con 0x7fe310103f50 2024-04-25T16:23:20.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.205+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31019c2a0 con 0x7fe310103f50 2024-04-25T16:23:20.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.205+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31019c720 con 0x7fe310103f50 2024-04-25T16:23:20.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.205+0000 7fe3077fe700 1 -- 172.21.15.98:0/881989499 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe30c00fd70 con 0x7fe310103f50 2024-04-25T16:23:20.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.206+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe310191630 con 0x7fe310103f50 2024-04-25T16:23:20.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.206+0000 7fe3077fe700 1 -- 172.21.15.98:0/881989499 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fe30c02a070 con 0x7fe310103f50 2024-04-25T16:23:20.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.206+0000 7fe3077fe700 1 --2- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe2fc03d670 0x7fe2fc03fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:20.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.206+0000 7fe3077fe700 1 -- 172.21.15.98:0/881989499 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe30c052750 con 0x7fe310103f50 2024-04-25T16:23:20.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.207+0000 7fe315a21700 1 --2- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe2fc03d670 0x7fe2fc03fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:20.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.207+0000 7fe315a21700 1 --2- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe2fc03d670 0x7fe2fc03fb80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe300006fd0 tx=0x7fe300006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:20.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.212+0000 7fe3077fe700 1 -- 172.21.15.98:0/881989499 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe30c013070 con 0x7fe310103f50 2024-04-25T16:23:20.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.422+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fe31002e700 con 0x7fe310103f50 2024-04-25T16:23:20.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.425+0000 7fe3077fe700 1 -- 172.21.15.98:0/881989499 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fe30c017db0 con 0x7fe310103f50 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe2fc03d670 msgr2=0x7fe2fc03fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 --2- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe2fc03d670 0x7fe2fc03fb80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe300006fd0 tx=0x7fe300006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 msgr2=0x7fe3101024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 --2- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 0x7fe3101024e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe30c005760 tx=0x7fe30c004f60 comp rx=0 tx=0).stop 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 shutdown_connections 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 --2- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe2fc03d670 0x7fe2fc03fb80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 --2- 172.21.15.98:0/881989499 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe310103f50 0x7fe3101024e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 >> 172.21.15.98:0/881989499 conn(0x7fe3100fd730 msgr2=0x7fe3100fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:20.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 shutdown_connections 2024-04-25T16:23:20.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:20.427+0000 7fe318486700 1 -- 172.21.15.98:0/881989499 wait complete. 2024-04-25T16:23:21.079 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T16:23:21.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T16:23:21.497 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:21.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:21 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/881989499' entity='client.admin' 2024-04-25T16:23:21.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:21.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.936+0000 7ff542a20700 1 -- 172.21.15.98:0/1935351203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 msgr2=0x7ff53c102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.936+0000 7ff542a20700 1 --2- 172.21.15.98:0/1935351203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 0x7ff53c102170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff52c009910 tx=0x7ff52c009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.937+0000 7ff542a20700 1 -- 172.21.15.98:0/1935351203 shutdown_connections 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.937+0000 7ff542a20700 1 --2- 172.21.15.98:0/1935351203 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 0x7ff53c102170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.937+0000 7ff542a20700 1 -- 172.21.15.98:0/1935351203 >> 172.21.15.98:0/1935351203 conn(0x7ff53c0fd760 msgr2=0x7ff53c0ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.937+0000 7ff542a20700 1 -- 172.21.15.98:0/1935351203 shutdown_connections 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.937+0000 7ff542a20700 1 -- 172.21.15.98:0/1935351203 wait complete. 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.937+0000 7ff542a20700 1 Processor -- start 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.937+0000 7ff542a20700 1 -- start start 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff542a20700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 0x7ff53c199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:22.940 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff542a20700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c109910 con 0x7ff53c101d60 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff53bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 0x7ff53c199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff53bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 0x7ff53c199520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47756/0 (socket says 172.21.15.98:47756) 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff53bfff700 1 -- 172.21.15.98:0/2976169650 learned_addr learned my addr 172.21.15.98:0/2976169650 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff53bfff700 1 -- 172.21.15.98:0/2976169650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c199a60 con 0x7ff53c101d60 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff53bfff700 1 --2- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 0x7ff53c199520 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff52c005760 tx=0x7ff52c004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff5397fa700 1 -- 172.21.15.98:0/2976169650 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff52c027070 con 0x7ff53c101d60 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff5397fa700 1 -- 172.21.15.98:0/2976169650 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff52c00bdf0 con 0x7ff53c101d60 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53c199cf0 con 0x7ff53c101d60 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff5397fa700 1 -- 172.21.15.98:0/2976169650 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff52c00fd70 con 0x7ff53c101d60 2024-04-25T16:23:22.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.938+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff53c19c900 con 0x7ff53c101d60 2024-04-25T16:23:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.939+0000 7ff5397fa700 1 -- 172.21.15.98:0/2976169650 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7ff52c02a070 con 0x7ff53c101d60 2024-04-25T16:23:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.939+0000 7ff5397fa700 1 --2- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff52403d610 0x7ff52403fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.940+0000 7ff5397fa700 1 -- 172.21.15.98:0/2976169650 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff52c0535a0 con 0x7ff53c101d60 2024-04-25T16:23:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.940+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53c192760 con 0x7ff53c101d60 2024-04-25T16:23:22.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.940+0000 7ff53b7fe700 1 --2- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff52403d610 0x7ff52403fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:22.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.940+0000 7ff53b7fe700 1 --2- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff52403d610 0x7ff52403fb20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff530006fd0 tx=0x7ff530006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:22.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:22.945+0000 7ff5397fa700 1 -- 172.21.15.98:0/2976169650 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff52c031020 con 0x7ff53c101d60 2024-04-25T16:23:23.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.149+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7ff53c063ad0 con 0x7ff52403d610 2024-04-25T16:23:23.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.153+0000 7ff5397fa700 1 -- 172.21.15.98:0/2976169650 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff53c063ad0 con 0x7ff52403d610 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff52403d610 msgr2=0x7ff52403fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 --2- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff52403d610 0x7ff52403fb20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff530006fd0 tx=0x7ff530006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 msgr2=0x7ff53c199520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 --2- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 0x7ff53c199520 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff52c005760 tx=0x7ff52c004f60 comp rx=0 tx=0).stop 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 shutdown_connections 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 --2- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff52403d610 0x7ff52403fb20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 --2- 172.21.15.98:0/2976169650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff53c101d60 0x7ff53c199520 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 >> 172.21.15.98:0/2976169650 conn(0x7ff53c0fd760 msgr2=0x7ff53c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 shutdown_connections 2024-04-25T16:23:23.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:23.155+0000 7ff542a20700 1 -- 172.21.15.98:0/2976169650 wait complete. 2024-04-25T16:23:23.883 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-04-25T16:23:23.883 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:23:23.883 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-04-25T16:23:23.902 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:23:23.902 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:23:23.960 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-04-25T16:23:23.960 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch host add smithi173 2024-04-25T16:23:24.204 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:24.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:24.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:24.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:24.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:24.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:24.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T16:23:24.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: Updating smithi098:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:23:24.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:23:24.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: Updating smithi098:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:23:24.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:24.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:24.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:25.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.730+0000 7f8a518fd700 1 -- 172.21.15.98:0/1029123369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 msgr2=0x7f8a4c108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.730+0000 7f8a518fd700 1 --2- 172.21.15.98:0/1029123369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 0x7f8a4c108d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009910 tx=0x7f8a34009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.730+0000 7f8a518fd700 1 -- 172.21.15.98:0/1029123369 shutdown_connections 2024-04-25T16:23:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.730+0000 7f8a518fd700 1 --2- 172.21.15.98:0/1029123369 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 0x7f8a4c108d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.730+0000 7f8a518fd700 1 -- 172.21.15.98:0/1029123369 >> 172.21.15.98:0/1029123369 conn(0x7f8a4c078460 msgr2=0x7f8a4c0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.730+0000 7f8a518fd700 1 -- 172.21.15.98:0/1029123369 shutdown_connections 2024-04-25T16:23:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.730+0000 7f8a518fd700 1 -- 172.21.15.98:0/1029123369 wait complete. 2024-04-25T16:23:25.732 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.730+0000 7f8a518fd700 1 Processor -- start 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.731+0000 7f8a518fd700 1 -- start start 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.731+0000 7f8a518fd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 0x7f8a4c0ff870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.731+0000 7f8a518fd700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4c10cd40 con 0x7f8a4c108940 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.731+0000 7f8a4affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 0x7f8a4c0ff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.731+0000 7f8a4affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 0x7f8a4c0ff870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48362/0 (socket says 172.21.15.98:48362) 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.731+0000 7f8a4affd700 1 -- 172.21.15.98:0/2720702600 learned_addr learned my addr 172.21.15.98:0/2720702600 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.731+0000 7f8a4affd700 1 -- 172.21.15.98:0/2720702600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a4c1016c0 con 0x7f8a4c108940 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.731+0000 7f8a4affd700 1 --2- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 0x7f8a4c0ff870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8a34005660 tx=0x7f8a34005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.732+0000 7f8a508fb700 1 -- 172.21.15.98:0/2720702600 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a34026070 con 0x7f8a4c108940 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.732+0000 7f8a508fb700 1 -- 172.21.15.98:0/2720702600 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a3400b6f0 con 0x7f8a4c108940 2024-04-25T16:23:25.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.732+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a4c0ffdb0 con 0x7f8a4c108940 2024-04-25T16:23:25.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.732+0000 7f8a508fb700 1 -- 172.21.15.98:0/2720702600 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a340037d0 con 0x7f8a4c108940 2024-04-25T16:23:25.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.732+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4c100270 con 0x7f8a4c108940 2024-04-25T16:23:25.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.733+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4c104050 con 0x7f8a4c108940 2024-04-25T16:23:25.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.733+0000 7f8a508fb700 1 -- 172.21.15.98:0/2720702600 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f8a34029070 con 0x7f8a4c108940 2024-04-25T16:23:25.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.733+0000 7f8a508fb700 1 --2- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a3803d610 0x7f8a3803fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:25.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.733+0000 7f8a508fb700 1 -- 172.21.15.98:0/2720702600 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8a3401e070 con 0x7f8a4c108940 2024-04-25T16:23:25.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.733+0000 7f8a4a7fc700 1 --2- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a3803d610 0x7f8a3803fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:25.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.733+0000 7f8a4a7fc700 1 --2- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a3803d610 0x7f8a3803fb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c006fd0 tx=0x7f8a3c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:25.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.738+0000 7f8a508fb700 1 -- 172.21.15.98:0/2720702600 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a34013070 con 0x7f8a4c108940 2024-04-25T16:23:25.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:25.940+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) -- 0x7f8a4c1005b0 con 0x7f8a3803d610 2024-04-25T16:23:27.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:26 smithi098 ceph-mon[29162]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:27.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:26 smithi098 ceph-mon[29162]: Deploying cephadm binary to smithi173 2024-04-25T16:23:29.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:28 smithi098 ceph-mon[29162]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:30.082 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-04-25T16:23:30.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.080+0000 7f8a508fb700 1 -- 172.21.15.98:0/2720702600 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f8a4c1005b0 con 0x7f8a3803d610 2024-04-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a3803d610 msgr2=0x7f8a3803fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 --2- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a3803d610 0x7f8a3803fb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c006fd0 tx=0x7f8a3c006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 msgr2=0x7f8a4c0ff870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:30.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 --2- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 0x7f8a4c0ff870 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8a34005660 tx=0x7f8a34005740 comp rx=0 tx=0).stop 2024-04-25T16:23:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 shutdown_connections 2024-04-25T16:23:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 --2- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a3803d610 0x7f8a3803fb20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 --2- 172.21.15.98:0/2720702600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8a4c108940 0x7f8a4c0ff870 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 >> 172.21.15.98:0/2720702600 conn(0x7f8a4c078460 msgr2=0x7f8a4c106420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 shutdown_connections 2024-04-25T16:23:30.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:30.083+0000 7f8a518fd700 1 -- 172.21.15.98:0/2720702600 wait complete. 2024-04-25T16:23:30.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T16:23:30.944 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:31.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:30 smithi098 ceph-mon[29162]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:31.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:30 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:31.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:30 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:31.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:30 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:31.982 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:31 smithi098 ceph-mon[29162]: Added host smithi173 2024-04-25T16:23:31.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.478+0000 7fb2afcae700 1 -- 172.21.15.98:0/10677291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 msgr2=0x7fb2a8107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.478+0000 7fb2afcae700 1 --2- 172.21.15.98:0/10677291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 0x7fb2a8107d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009910 tx=0x7fb2a4009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.478+0000 7fb2afcae700 1 -- 172.21.15.98:0/10677291 shutdown_connections 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.478+0000 7fb2afcae700 1 --2- 172.21.15.98:0/10677291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 0x7fb2a8107d60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.478+0000 7fb2afcae700 1 -- 172.21.15.98:0/10677291 >> 172.21.15.98:0/10677291 conn(0x7fb2a8078c30 msgr2=0x7fb2a8079070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.479+0000 7fb2afcae700 1 -- 172.21.15.98:0/10677291 shutdown_connections 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.479+0000 7fb2afcae700 1 -- 172.21.15.98:0/10677291 wait complete. 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.479+0000 7fb2afcae700 1 Processor -- start 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.479+0000 7fb2afcae700 1 -- start start 2024-04-25T16:23:32.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.479+0000 7fb2afcae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 0x7fb2a819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.479+0000 7fb2afcae700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a810cea0 con 0x7fb2a8107950 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb2ada4a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 0x7fb2a819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb2ada4a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 0x7fb2a819a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48382/0 (socket says 172.21.15.98:48382) 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb2ada4a700 1 -- 172.21.15.98:0/2422787881 learned_addr learned my addr 172.21.15.98:0/2422787881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb2ada4a700 1 -- 172.21.15.98:0/2422787881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a8102ed0 con 0x7fb2a8107950 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb2ada4a700 1 --2- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 0x7fb2a819a610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4005660 tx=0x7fb2a4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb29affd700 1 -- 172.21.15.98:0/2422787881 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2a4026070 con 0x7fb2a8107950 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb29affd700 1 -- 172.21.15.98:0/2422787881 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb2a400b750 con 0x7fb2a8107950 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a819abb0 con 0x7fb2a8107950 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb29affd700 1 -- 172.21.15.98:0/2422787881 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2a40037b0 con 0x7fb2a8107950 2024-04-25T16:23:32.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.480+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a819b070 con 0x7fb2a8107950 2024-04-25T16:23:32.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.481+0000 7fb29affd700 1 -- 172.21.15.98:0/2422787881 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fb2a4029070 con 0x7fb2a8107950 2024-04-25T16:23:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.481+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a81040e0 con 0x7fb2a8107950 2024-04-25T16:23:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.481+0000 7fb29affd700 1 --2- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb29403d670 0x7fb29403fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.481+0000 7fb29affd700 1 -- 172.21.15.98:0/2422787881 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb2a401e070 con 0x7fb2a8107950 2024-04-25T16:23:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.481+0000 7fb2ad249700 1 --2- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb29403d670 0x7fb29403fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:32.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.482+0000 7fb2ad249700 1 --2- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb29403d670 0x7fb29403fb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb29c006fd0 tx=0x7fb29c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:32.488 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.487+0000 7fb29affd700 1 -- 172.21.15.98:0/2422787881 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2a4013070 con 0x7fb2a8107950 2024-04-25T16:23:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.689+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb2a8102620 con 0x7fb29403d670 2024-04-25T16:23:32.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.690+0000 7fb29affd700 1 -- 172.21.15.98:0/2422787881 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7fb2a8102620 con 0x7fb29403d670 2024-04-25T16:23:32.692 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:32.692 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-04-25T16:23:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb29403d670 msgr2=0x7fb29403fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 --2- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb29403d670 0x7fb29403fb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb29c006fd0 tx=0x7fb29c006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 msgr2=0x7fb2a819a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 --2- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 0x7fb2a819a610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4005660 tx=0x7fb2a4005740 comp rx=0 tx=0).stop 2024-04-25T16:23:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 shutdown_connections 2024-04-25T16:23:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 --2- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb29403d670 0x7fb29403fb80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 --2- 172.21.15.98:0/2422787881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb2a8107950 0x7fb2a819a610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 >> 172.21.15.98:0/2422787881 conn(0x7fb2a8078c30 msgr2=0x7fb2a81099c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 shutdown_connections 2024-04-25T16:23:32.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:32.693+0000 7fb2afcae700 1 -- 172.21.15.98:0/2422787881 wait complete. 2024-04-25T16:23:33.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:32 smithi098 ceph-mon[29162]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:33.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:33.308 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-04-25T16:23:33.308 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:23:33.308 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-04-25T16:23:33.327 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:23:33.327 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:23:33.384 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-04-25T16:23:33.384 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch host add smithi203 2024-04-25T16:23:33.615 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:34.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:33 smithi098 ceph-mon[29162]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:23:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.042+0000 7f18ae2da700 1 -- 172.21.15.98:0/73241278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 msgr2=0x7f18a8102600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.042+0000 7f18ae2da700 1 --2- 172.21.15.98:0/73241278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 0x7f18a8102600 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1898009910 tx=0x7f1898009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.042+0000 7f18ae2da700 1 -- 172.21.15.98:0/73241278 shutdown_connections 2024-04-25T16:23:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.042+0000 7f18ae2da700 1 --2- 172.21.15.98:0/73241278 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 0x7f18a8102600 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.042+0000 7f18ae2da700 1 -- 172.21.15.98:0/73241278 >> 172.21.15.98:0/73241278 conn(0x7f18a80fdd00 msgr2=0x7f18a8100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:35.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.042+0000 7f18ae2da700 1 -- 172.21.15.98:0/73241278 shutdown_connections 2024-04-25T16:23:35.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.042+0000 7f18ae2da700 1 -- 172.21.15.98:0/73241278 wait complete. 2024-04-25T16:23:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.043+0000 7f18ae2da700 1 Processor -- start 2024-04-25T16:23:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.043+0000 7f18ae2da700 1 -- start start 2024-04-25T16:23:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18ae2da700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 0x7f18a81a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18ae2da700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f18a810de50 con 0x7f18a81021f0 2024-04-25T16:23:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18a7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 0x7f18a81a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18a7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 0x7f18a81a1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48392/0 (socket says 172.21.15.98:48392) 2024-04-25T16:23:35.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18a7fff700 1 -- 172.21.15.98:0/3327937264 learned_addr learned my addr 172.21.15.98:0/3327937264 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:35.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18a7fff700 1 -- 172.21.15.98:0/3327937264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a81a1ac0 con 0x7f18a81021f0 2024-04-25T16:23:35.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18a7fff700 1 --2- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 0x7f18a81a1580 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18980092e0 tx=0x7f1898005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:35.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18a57fa700 1 -- 172.21.15.98:0/3327937264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1898026070 con 0x7f18a81021f0 2024-04-25T16:23:35.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a81a1db0 con 0x7f18a81021f0 2024-04-25T16:23:35.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18a57fa700 1 -- 172.21.15.98:0/3327937264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f189800b6f0 con 0x7f18a81021f0 2024-04-25T16:23:35.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18a57fa700 1 -- 172.21.15.98:0/3327937264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18980036c0 con 0x7f18a81021f0 2024-04-25T16:23:35.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.044+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a819b7c0 con 0x7f18a81021f0 2024-04-25T16:23:35.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.045+0000 7f18a57fa700 1 -- 172.21.15.98:0/3327937264 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f1898029070 con 0x7f18a81021f0 2024-04-25T16:23:35.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.045+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18a8105280 con 0x7f18a81021f0 2024-04-25T16:23:35.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.046+0000 7f18a57fa700 1 --2- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f189003d610 0x7f189003fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:35.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.046+0000 7f18a57fa700 1 -- 172.21.15.98:0/3327937264 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f189801e070 con 0x7f18a81021f0 2024-04-25T16:23:35.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.046+0000 7f18a77fe700 1 --2- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f189003d610 0x7f189003fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:35.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.046+0000 7f18a77fe700 1 --2- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f189003d610 0x7f189003fb20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f189c006fd0 tx=0x7f189c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:35.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.051+0000 7f18a57fa700 1 -- 172.21.15.98:0/3327937264 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1898013070 con 0x7f18a81021f0 2024-04-25T16:23:35.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:35.250+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) -- 0x7f18a819ccd0 con 0x7f189003d610 2024-04-25T16:23:35.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:34 smithi098 ceph-mon[29162]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T16:23:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: Deploying cephadm binary to smithi203 2024-04-25T16:23:37.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:36 smithi098 ceph-mon[29162]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:38.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:37 smithi098 ceph-mon[29162]: Updating smithi173:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:23:38.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:37 smithi098 ceph-mon[29162]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:23:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:38 smithi098 ceph-mon[29162]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:39.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.325+0000 7f18a57fa700 1 -- 172.21.15.98:0/3327937264 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f18a819ccd0 con 0x7f189003d610 2024-04-25T16:23:39.327 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi203' with addr '172.21.15.203' 2024-04-25T16:23:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f189003d610 msgr2=0x7f189003fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 --2- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f189003d610 0x7f189003fb20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f189c006fd0 tx=0x7f189c006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 msgr2=0x7f18a81a1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 --2- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 0x7f18a81a1580 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f18980092e0 tx=0x7f1898005710 comp rx=0 tx=0).stop 2024-04-25T16:23:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 shutdown_connections 2024-04-25T16:23:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 --2- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f189003d610 0x7f189003fb20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 --2- 172.21.15.98:0/3327937264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f18a81021f0 0x7f18a81a1580 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 >> 172.21.15.98:0/3327937264 conn(0x7f18a80fdd00 msgr2=0x7f18a8105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 shutdown_connections 2024-04-25T16:23:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:39.327+0000 7f18ae2da700 1 -- 172.21.15.98:0/3327937264 wait complete. 2024-04-25T16:23:39.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:39 smithi098 ceph-mon[29162]: Updating smithi173:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:23:39.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:39.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:39.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:39.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:39.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:39.962 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T16:23:40.191 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:41.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:40 smithi098 ceph-mon[29162]: Added host smithi203 2024-04-25T16:23:41.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:40 smithi098 ceph-mon[29162]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:41.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:41.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:41.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.779+0000 7f1a275b8700 1 -- 172.21.15.98:0/3018925309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 msgr2=0x7f1a20108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:41.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.779+0000 7f1a275b8700 1 --2- 172.21.15.98:0/3018925309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 0x7f1a20108d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009910 tx=0x7f1a10009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:41.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.779+0000 7f1a275b8700 1 -- 172.21.15.98:0/3018925309 shutdown_connections 2024-04-25T16:23:41.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.779+0000 7f1a275b8700 1 --2- 172.21.15.98:0/3018925309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 0x7f1a20108d10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:41.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.780+0000 7f1a275b8700 1 -- 172.21.15.98:0/3018925309 >> 172.21.15.98:0/3018925309 conn(0x7f1a200fe410 msgr2=0x7f1a20100860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:41.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.780+0000 7f1a275b8700 1 -- 172.21.15.98:0/3018925309 shutdown_connections 2024-04-25T16:23:41.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.780+0000 7f1a275b8700 1 -- 172.21.15.98:0/3018925309 wait complete. 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.780+0000 7f1a275b8700 1 Processor -- start 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.780+0000 7f1a275b8700 1 -- start start 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a275b8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 0x7f1a20198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a275b8700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1a2010cd00 con 0x7f1a20108900 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a25354700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 0x7f1a20198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a25354700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 0x7f1a20198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38010/0 (socket says 172.21.15.98:38010) 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a25354700 1 -- 172.21.15.98:0/3270336872 learned_addr learned my addr 172.21.15.98:0/3270336872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a25354700 1 -- 172.21.15.98:0/3270336872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a20198b40 con 0x7f1a20108900 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a25354700 1 --2- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 0x7f1a20198600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a10005660 tx=0x7f1a10005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a167fc700 1 -- 172.21.15.98:0/3270336872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a10026070 con 0x7f1a20108900 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a167fc700 1 -- 172.21.15.98:0/3270336872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1a1000b6f0 con 0x7f1a20108900 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a20198e30 con 0x7f1a20108900 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a167fc700 1 -- 172.21.15.98:0/3270336872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a100037d0 con 0x7f1a20108900 2024-04-25T16:23:41.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.781+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a2019ce30 con 0x7f1a20108900 2024-04-25T16:23:41.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.782+0000 7f1a167fc700 1 -- 172.21.15.98:0/3270336872 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f1a10029070 con 0x7f1a20108900 2024-04-25T16:23:41.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.782+0000 7f1a167fc700 1 --2- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1a0c03d610 0x7f1a0c03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:41.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.782+0000 7f1a167fc700 1 -- 172.21.15.98:0/3270336872 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1a1001e070 con 0x7f1a20108900 2024-04-25T16:23:41.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.782+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a20104010 con 0x7f1a20108900 2024-04-25T16:23:41.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.783+0000 7f1a24b53700 1 --2- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1a0c03d610 0x7f1a0c03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:41.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.783+0000 7f1a24b53700 1 --2- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1a0c03d610 0x7f1a0c03fb20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c006fd0 tx=0x7f1a1c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:41.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.788+0000 7f1a167fc700 1 -- 172.21.15.98:0/3270336872 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a10030080 con 0x7f1a20108900 2024-04-25T16:23:41.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.991+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a20199960 con 0x7f1a0c03d610 2024-04-25T16:23:41.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.992+0000 7f1a167fc700 1 -- 172.21.15.98:0/3270336872 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7f1a20199960 con 0x7f1a0c03d610 2024-04-25T16:23:41.993 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:23:41.993 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}, {"addr": "172.21.15.203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1a0c03d610 msgr2=0x7f1a0c03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 --2- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1a0c03d610 0x7f1a0c03fb20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a1c006fd0 tx=0x7f1a1c006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 msgr2=0x7f1a20198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 --2- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 0x7f1a20198600 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a10005660 tx=0x7f1a10005740 comp rx=0 tx=0).stop 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 shutdown_connections 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 --2- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1a0c03d610 0x7f1a0c03fb20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 --2- 172.21.15.98:0/3270336872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1a20108900 0x7f1a20198600 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 >> 172.21.15.98:0/3270336872 conn(0x7f1a200fe410 msgr2=0x7f1a200ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 shutdown_connections 2024-04-25T16:23:41.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:41.994+0000 7f1a275b8700 1 -- 172.21.15.98:0/3270336872 wait complete. 2024-04-25T16:23:42.669 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T16:23:42.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T16:23:42.905 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:23:43.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:42 smithi098 ceph-mon[29162]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:43.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:44.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:43 smithi098 ceph-mon[29162]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:23:44.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.453+0000 7f5d258e0700 1 -- 172.21.15.98:0/1007277709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 msgr2=0x7f5d20104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:44.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.453+0000 7f5d258e0700 1 --2- 172.21.15.98:0/1007277709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 0x7f5d20104fb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5d10009910 tx=0x7f5d10009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:44.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.453+0000 7f5d258e0700 1 -- 172.21.15.98:0/1007277709 shutdown_connections 2024-04-25T16:23:44.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.453+0000 7f5d258e0700 1 --2- 172.21.15.98:0/1007277709 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 0x7f5d20104fb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:44.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.453+0000 7f5d258e0700 1 -- 172.21.15.98:0/1007277709 >> 172.21.15.98:0/1007277709 conn(0x7f5d20078460 msgr2=0x7f5d200788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:44.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.453+0000 7f5d258e0700 1 -- 172.21.15.98:0/1007277709 shutdown_connections 2024-04-25T16:23:44.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.453+0000 7f5d258e0700 1 -- 172.21.15.98:0/1007277709 wait complete. 2024-04-25T16:23:44.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.454+0000 7f5d258e0700 1 Processor -- start 2024-04-25T16:23:44.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.454+0000 7f5d258e0700 1 -- start start 2024-04-25T16:23:44.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d258e0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 0x7f5d20199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:44.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d258e0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2010deb0 con 0x7f5d20104ba0 2024-04-25T16:23:44.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d1effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 0x7f5d20199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:44.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d1effd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 0x7f5d20199750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38024/0 (socket says 172.21.15.98:38024) 2024-04-25T16:23:44.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d1effd700 1 -- 172.21.15.98:0/2717113200 learned_addr learned my addr 172.21.15.98:0/2717113200 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:23:44.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d1effd700 1 -- 172.21.15.98:0/2717113200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d20199c90 con 0x7f5d20104ba0 2024-04-25T16:23:44.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d1effd700 1 --2- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 0x7f5d20199750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5d10006010 tx=0x7f5d10005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:44.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d248de700 1 -- 172.21.15.98:0/2717113200 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d10026070 con 0x7f5d20104ba0 2024-04-25T16:23:44.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d248de700 1 -- 172.21.15.98:0/2717113200 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5d1000b750 con 0x7f5d20104ba0 2024-04-25T16:23:44.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d20199f20 con 0x7f5d20104ba0 2024-04-25T16:23:44.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d248de700 1 -- 172.21.15.98:0/2717113200 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d100038a0 con 0x7f5d20104ba0 2024-04-25T16:23:44.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.455+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2019a380 con 0x7f5d20104ba0 2024-04-25T16:23:44.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.456+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2006b540 con 0x7f5d20104ba0 2024-04-25T16:23:44.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.456+0000 7f5d248de700 1 -- 172.21.15.98:0/2717113200 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f5d10029070 con 0x7f5d20104ba0 2024-04-25T16:23:44.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.457+0000 7f5d248de700 1 --2- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5d0803d670 0x7f5d0803fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:44.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.457+0000 7f5d248de700 1 -- 172.21.15.98:0/2717113200 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5d1001e070 con 0x7f5d20104ba0 2024-04-25T16:23:44.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.457+0000 7f5d1e7fc700 1 --2- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5d0803d670 0x7f5d0803fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:44.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.457+0000 7f5d1e7fc700 1 --2- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5d0803d670 0x7f5d0803fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d14006fd0 tx=0x7f5d14006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:44.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.462+0000 7f5d248de700 1 -- 172.21.15.98:0/2717113200 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d10013070 con 0x7f5d20104ba0 2024-04-25T16:23:44.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.663+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f5d2019aab0 con 0x7f5d20104ba0 2024-04-25T16:23:44.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.898+0000 7f5d248de700 1 -- 172.21.15.98:0/2717113200 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f5d1000e3e0 con 0x7f5d20104ba0 2024-04-25T16:23:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5d0803d670 msgr2=0x7f5d0803fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 --2- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5d0803d670 0x7f5d0803fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5d14006fd0 tx=0x7f5d14006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 msgr2=0x7f5d20199750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 --2- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 0x7f5d20199750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5d10006010 tx=0x7f5d10005710 comp rx=0 tx=0).stop 2024-04-25T16:23:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 shutdown_connections 2024-04-25T16:23:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 --2- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5d0803d670 0x7f5d0803fb80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:44.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 --2- 172.21.15.98:0/2717113200 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5d20104ba0 0x7f5d20199750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:44.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 >> 172.21.15.98:0/2717113200 conn(0x7f5d20078460 msgr2=0x7f5d201059e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:44.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.900+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 shutdown_connections 2024-04-25T16:23:44.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:23:44.901+0000 7f5d258e0700 1 -- 172.21.15.98:0/2717113200 wait complete. 2024-04-25T16:23:44.904 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-04-25T16:23:45.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:44 smithi098 ceph-mon[29162]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:45.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:44 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2717113200' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T16:23:45.606 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-04-25T16:23:45.607 INFO:tasks.cephadm:Adding mon.b on smithi173 2024-04-25T16:23:45.607 INFO:tasks.cephadm:Adding mon.c on smithi203 2024-04-25T16:23:45.607 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c' 2024-04-25T16:23:45.841 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:23:46.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2717113200' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T16:23:46.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:23:46.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:46.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:47.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:46 smithi098 ceph-mon[29162]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T16:23:47.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:46 smithi098 ceph-mon[29162]: Updating smithi203:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:23:47.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:46 smithi098 ceph-mon[29162]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:23:47.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:46 smithi098 ceph-mon[29162]: Updating smithi203:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:23:47.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:46 smithi098 ceph-mon[29162]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.293+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1100749804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 msgr2=0x7fb8d0108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.293+0000 7fb8d86f2700 1 --2- 172.21.15.203:0/1100749804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 0x7fb8d0108d20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009910 tx=0x7fb8c0009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.293+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1100749804 shutdown_connections 2024-04-25T16:23:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.293+0000 7fb8d86f2700 1 --2- 172.21.15.203:0/1100749804 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 0x7fb8d0108d20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.293+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1100749804 >> 172.21.15.203:0/1100749804 conn(0x7fb8d00fe440 msgr2=0x7fb8d0100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.293+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1100749804 shutdown_connections 2024-04-25T16:23:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.293+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1100749804 wait complete. 2024-04-25T16:23:47.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.294+0000 7fb8d86f2700 1 Processor -- start 2024-04-25T16:23:47.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.294+0000 7fb8d86f2700 1 -- start start 2024-04-25T16:23:47.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.294+0000 7fb8d86f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 0x7fb8d019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:47.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.294+0000 7fb8d86f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d010cd10 con 0x7fb8d0108910 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.294+0000 7fb8d648e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 0x7fb8d019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.294+0000 7fb8d648e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 0x7fb8d019a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.203:39102/0 (socket says 172.21.15.203:39102) 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.294+0000 7fb8d648e700 1 -- 172.21.15.203:0/1710103575 learned_addr learned my addr 172.21.15.203:0/1710103575 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.295+0000 7fb8d648e700 1 -- 172.21.15.203:0/1710103575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d006c940 con 0x7fb8d0108910 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.295+0000 7fb8d648e700 1 --2- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 0x7fb8d019a580 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0005660 tx=0x7fb8c0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.295+0000 7fb8c77fe700 1 -- 172.21.15.203:0/1710103575 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8c0026070 con 0x7fb8d0108910 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.295+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8d019ab80 con 0x7fb8d0108910 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.295+0000 7fb8c77fe700 1 -- 172.21.15.203:0/1710103575 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8c000b750 con 0x7fb8d0108910 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.295+0000 7fb8c77fe700 1 -- 172.21.15.203:0/1710103575 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb8c0003810 con 0x7fb8d0108910 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.295+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8d019b040 con 0x7fb8d0108910 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.296+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8d0104020 con 0x7fb8d0108910 2024-04-25T16:23:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.296+0000 7fb8c77fe700 1 -- 172.21.15.203:0/1710103575 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fb8c0029070 con 0x7fb8d0108910 2024-04-25T16:23:47.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.296+0000 7fb8c77fe700 1 --2- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc03d610 0x7fb8bc03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:47.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.297+0000 7fb8c77fe700 1 -- 172.21.15.203:0/1710103575 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb8c001e070 con 0x7fb8d0108910 2024-04-25T16:23:47.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.297+0000 7fb8d5c8d700 1 --2- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc03d610 0x7fb8bc03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:47.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.297+0000 7fb8d5c8d700 1 --2- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc03d610 0x7fb8bc03fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc006fd0 tx=0x7fb8cc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:47.303 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.302+0000 7fb8c77fe700 1 -- 172.21.15.203:0/1710103575 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8d006c940 con 0x7fb8d0108910 2024-04-25T16:23:47.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.512+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}) -- 0x7fb8d019b320 con 0x7fb8bc03d610 2024-04-25T16:23:47.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.519+0000 7fb8c77fe700 1 -- 172.21.15.203:0/1710103575 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb8d019b320 con 0x7fb8bc03d610 2024-04-25T16:23:47.520 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-04-25T16:23:47.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc03d610 msgr2=0x7fb8bc03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:47.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 --2- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc03d610 0x7fb8bc03fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc006fd0 tx=0x7fb8cc006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:47.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 msgr2=0x7fb8d019a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:47.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 --2- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 0x7fb8d019a580 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0005660 tx=0x7fb8c0005740 comp rx=0 tx=0).stop 2024-04-25T16:23:47.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 shutdown_connections 2024-04-25T16:23:47.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 --2- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc03d610 0x7fb8bc03fb20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:47.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 --2- 172.21.15.203:0/1710103575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0108910 0x7fb8d019a580 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:47.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 >> 172.21.15.203:0/1710103575 conn(0x7fb8d00fe440 msgr2=0x7fb8d00ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:47.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 shutdown_connections 2024-04-25T16:23:47.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:47.521+0000 7fb8d86f2700 1 -- 172.21.15.203:0/1710103575 wait complete. 2024-04-25T16:23:48.111 DEBUG:teuthology.orchestra.run.smithi173:mon.b> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T16:23:48.114 DEBUG:teuthology.orchestra.run.smithi203:mon.c> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T16:23:48.116 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T16:23:48.116 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T16:23:49.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:23:49.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-04-25T16:23:49.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:49.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:23:49.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:49.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:23:49.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:49.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:23:49.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:49.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: Deploying daemon mon.c on smithi203 2024-04-25T16:23:49.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:48 smithi098 ceph-mon[29162]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:49.804 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:23:51.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:50 smithi098 ceph-mon[29162]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:51.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.853+0000 7f770647c700 1 -- 172.21.15.203:0/2280776282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 msgr2=0x7f7700104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:51.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.853+0000 7f770647c700 1 --2- 172.21.15.203:0/2280776282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 0x7f7700104f80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f76f4009910 tx=0x7f76f4009c20 comp rx=0 tx=0).stop 2024-04-25T16:23:51.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.853+0000 7f770647c700 1 -- 172.21.15.203:0/2280776282 shutdown_connections 2024-04-25T16:23:51.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.853+0000 7f770647c700 1 --2- 172.21.15.203:0/2280776282 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 0x7f7700104f80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:51.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.853+0000 7f770647c700 1 -- 172.21.15.203:0/2280776282 >> 172.21.15.203:0/2280776282 conn(0x7f77000fe420 msgr2=0x7f7700100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:51.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.853+0000 7f770647c700 1 -- 172.21.15.203:0/2280776282 shutdown_connections 2024-04-25T16:23:51.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.853+0000 7f770647c700 1 -- 172.21.15.203:0/2280776282 wait complete. 2024-04-25T16:23:51.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.854+0000 7f770647c700 1 Processor -- start 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.854+0000 7f770647c700 1 -- start start 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.854+0000 7f770647c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 0x7f7700199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.854+0000 7f770647c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f770010de80 con 0x7f7700104b70 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f76fffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 0x7f7700199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f76fffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 0x7f7700199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.203:51624/0 (socket says 172.21.15.203:51624) 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f76fffff700 1 -- 172.21.15.203:0/270932364 learned_addr learned my addr 172.21.15.203:0/270932364 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f76fffff700 1 -- 172.21.15.203:0/270932364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7700199ca0 con 0x7f7700104b70 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f76fffff700 1 --2- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 0x7f7700199760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f76f4005060 tx=0x7f76f4005710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f76fd7fa700 1 -- 172.21.15.203:0/270932364 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76f4026070 con 0x7f7700104b70 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7700199f30 con 0x7f7700104b70 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f76fd7fa700 1 -- 172.21.15.203:0/270932364 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f76f400b750 con 0x7f7700104b70 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.855+0000 7f76fd7fa700 1 -- 172.21.15.203:0/270932364 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f76f4003850 con 0x7f7700104b70 2024-04-25T16:23:51.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.856+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770019a390 con 0x7f7700104b70 2024-04-25T16:23:51.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.857+0000 7f76fd7fa700 1 -- 172.21.15.203:0/270932364 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f76f4029070 con 0x7f7700104b70 2024-04-25T16:23:51.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.857+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f770006c290 con 0x7f7700104b70 2024-04-25T16:23:51.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.857+0000 7f76fd7fa700 1 --2- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f76e803d670 0x7f76e803fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:51.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.857+0000 7f76fd7fa700 1 -- 172.21.15.203:0/270932364 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f76f401e070 con 0x7f7700104b70 2024-04-25T16:23:51.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.857+0000 7f76ff7fe700 1 --2- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f76e803d670 0x7f76e803fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:51.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.858+0000 7f76ff7fe700 1 --2- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f76e803d670 0x7f76e803fb80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f76f0006fd0 tx=0x7f76f0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:51.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:51.863+0000 7f76fd7fa700 1 -- 172.21.15.203:0/270932364 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f76f4013070 con 0x7f7700104b70 2024-04-25T16:23:52.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.151+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f770019aac0 con 0x7f7700104b70 2024-04-25T16:23:52.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.152+0000 7f76fd7fa700 1 -- 172.21.15.203:0/270932364 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f76f4030300 con 0x7f7700104b70 2024-04-25T16:23:52.153 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:23:52.154 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","modified":"2024-04-25T16:22:06.623046Z","created":"2024-04-25T16:22:06.623046Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.154+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f76e803d670 msgr2=0x7f76e803fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.154+0000 7f770647c700 1 --2- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f76e803d670 0x7f76e803fb80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f76f0006fd0 tx=0x7f76f0006e40 comp rx=0 tx=0).stop 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.154+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 msgr2=0x7f7700199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.154+0000 7f770647c700 1 --2- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 0x7f7700199760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f76f4005060 tx=0x7f76f4005710 comp rx=0 tx=0).stop 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.154+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 shutdown_connections 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.154+0000 7f770647c700 1 --2- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f76e803d670 0x7f76e803fb80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.154+0000 7f770647c700 1 --2- 172.21.15.203:0/270932364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7700104b70 0x7f7700199760 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.154+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 >> 172.21.15.203:0/270932364 conn(0x7f77000fe420 msgr2=0x7f7700109d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.155+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 shutdown_connections 2024-04-25T16:23:52.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:52.155+0000 7f770647c700 1 -- 172.21.15.203:0/270932364 wait complete. 2024-04-25T16:23:52.157 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-04-25T16:23:53.938 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T16:23:53.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T16:23:54.203 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:23:57.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:23:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:57.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:23:57.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: mon.a calling monitor election 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: mon.c calling monitor election 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: monmap epoch 2 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: last_changed 2024-04-25T16:23:52.247764+0000 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: created 2024-04-25T16:22:06.623046+0000 2024-04-25T16:23:57.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: min_mon_release 19 (squid) 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: election_strategy: 1 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: fsmap 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: mgrmap e13: x(active, since 49s) 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: overall HEALTH_OK 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:23:57.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:23:58.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.673+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1853504164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 msgr2=0x7f6e9c005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:58.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.673+0000 7f6eb2ec3700 1 --2- 172.21.15.203:0/1853504164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 0x7f6e9c005aa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6eac102920 tx=0x7f6ea000feb0 comp rx=0 tx=0).stop 2024-04-25T16:23:58.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.673+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1853504164 shutdown_connections 2024-04-25T16:23:58.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.673+0000 7f6eb2ec3700 1 --2- 172.21.15.203:0/1853504164 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 0x7f6e9c005aa0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:58.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.673+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1853504164 >> 172.21.15.203:0/1853504164 conn(0x7f6eac0fe530 msgr2=0x7f6eac100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:58.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.673+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1853504164 shutdown_connections 2024-04-25T16:23:58.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.673+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1853504164 wait complete. 2024-04-25T16:23:58.677 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:23:58 smithi203 ceph-mon[33115]: Deploying daemon mon.b on smithi173 2024-04-25T16:23:58.677 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:23:58 smithi203 ceph-mon[33115]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:58.677 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:23:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.674+0000 7f6eb2ec3700 1 Processor -- start 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.674+0000 7f6eb2ec3700 1 -- start start 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.674+0000 7f6eb2ec3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 0x7f6eac19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.674+0000 7f6eb2ec3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 0x7f6eac19b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.674+0000 7f6eb2ec3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac10a880 con 0x7f6e9c0035c0 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.674+0000 7f6eb2ec3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6eac10a700 con 0x7f6eac19acb0 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eabfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 0x7f6eac19b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eabfff700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 0x7f6eac19b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:50820/0 (socket says 172.21.15.203:50820) 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eabfff700 1 -- 172.21.15.203:0/1636897391 learned_addr learned my addr 172.21.15.203:0/1636897391 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:23:58.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eb0c5f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 0x7f6eac19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eabfff700 1 -- 172.21.15.203:0/1636897391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 msgr2=0x7f6eac19b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eabfff700 1 -- 172.21.15.203:0/1636897391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 msgr2=0x7f6eac19b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eabfff700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 0x7f6eac19b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eabfff700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 0x7f6eac19b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eb0c5f700 1 -- 172.21.15.203:0/1636897391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 msgr2=0x7f6eac19b0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eb0c5f700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 0x7f6eac19b0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eb0c5f700 1 -- 172.21.15.203:0/1636897391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eac1a37e0 con 0x7f6e9c0035c0 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.675+0000 7f6eb0c5f700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 0x7f6eac19a770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0000c00 tx=0x7f6ea0013c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.676+0000 7f6ea9ffb700 1 -- 172.21.15.203:0/1636897391 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f6ea00053b0 con 0x7f6e9c0035c0 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.676+0000 7f6ea9ffb700 1 -- 172.21.15.203:0/1636897391 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6ea00044c0 con 0x7f6e9c0035c0 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.676+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eac1a39b0 con 0x7f6e9c0035c0 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.676+0000 7f6ea9ffb700 1 -- 172.21.15.203:0/1636897391 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f6ea0003c60 con 0x7f6e9c0035c0 2024-04-25T16:23:58.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.676+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eac1a3eb0 con 0x7f6e9c0035c0 2024-04-25T16:23:58.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.677+0000 7f6ea9ffb700 1 -- 172.21.15.203:0/1636897391 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f6ea0023070 con 0x7f6e9c0035c0 2024-04-25T16:23:58.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.677+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eac104130 con 0x7f6e9c0035c0 2024-04-25T16:23:58.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.677+0000 7f6ea9ffb700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e9003d400 0x7f6e9003f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:23:58.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.677+0000 7f6ea9ffb700 1 -- 172.21.15.203:0/1636897391 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6ea0020050 con 0x7f6e9c0035c0 2024-04-25T16:23:58.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.678+0000 7f6eabfff700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e9003d400 0x7f6e9003f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:23:58.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.678+0000 7f6eabfff700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e9003d400 0x7f6e9003f910 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6eac102060 tx=0x7f6e98008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:23:58.685 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.683+0000 7f6ea9ffb700 1 -- 172.21.15.203:0/1636897391 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6ea0028020 con 0x7f6e9c0035c0 2024-04-25T16:23:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:58 smithi098 ceph-mon[29162]: Deploying daemon mon.b on smithi173 2024-04-25T16:23:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:58 smithi098 ceph-mon[29162]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:23:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:23:58.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.989+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6eac02e510 con 0x7f6e9c0035c0 2024-04-25T16:23:58.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.990+0000 7f6ea9ffb700 1 -- 172.21.15.203:0/1636897391 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f6ea0024ce0 con 0x7f6e9c0035c0 2024-04-25T16:23:58.992 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:23:58.992 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":2,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","modified":"2024-04-25T16:23:52.247764Z","created":"2024-04-25T16:22:06.623046Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T16:23:58.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e9003d400 msgr2=0x7f6e9003f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:58.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e9003d400 0x7f6e9003f910 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6eac102060 tx=0x7f6e98008040 comp rx=0 tx=0).stop 2024-04-25T16:23:58.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 msgr2=0x7f6eac19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:23:58.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 0x7f6eac19a770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0000c00 tx=0x7f6ea0013c40 comp rx=0 tx=0).stop 2024-04-25T16:23:58.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 shutdown_connections 2024-04-25T16:23:58.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e9003d400 0x7f6e9003f910 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:58.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e9c0035c0 0x7f6eac19a770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:58.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 --2- 172.21.15.203:0/1636897391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6eac19acb0 0x7f6eac19b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:23:58.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 >> 172.21.15.203:0/1636897391 conn(0x7f6eac0fe530 msgr2=0x7f6eac1060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:23:58.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 shutdown_connections 2024-04-25T16:23:58.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:23:58.992+0000 7f6eb2ec3700 1 -- 172.21.15.203:0/1636897391 wait complete. 2024-04-25T16:23:58.995 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 2 2024-04-25T16:23:59.569 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:23:59 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/1636897391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:23:59.655 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:23:59 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/1636897391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:24:00.571 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T16:24:00.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T16:24:00.599 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:00 smithi203 ceph-mon[33115]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:00.671 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:00 smithi098 ceph-mon[29162]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:00.829 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:24:00.910 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:00 smithi173 podman[32535]: 2024-04-25 16:24:00.712206104 +0000 UTC m=+0.018482937 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:24:01.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:00 smithi173 podman[32535]: 2024-04-25 16:24:00.997554945 +0000 UTC m=+0.303831787 container init 8af9d65e7b935c6eb496e1d3e29c15322b86868dd1ea6e54ec8f8a23e95d6997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 podman[32535]: 2024-04-25 16:24:01.000889046 +0000 UTC m=+0.307165873 container start 8af9d65e7b935c6eb496e1d3e29c15322b86868dd1ea6e54ec8f8a23e95d6997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pidfile_write: ignore empty --pid-file 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: load: jerasure load: lrc 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: RocksDB version: 7.9.2 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Git sha 0 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: DB SUMMARY 2024-04-25T16:24:01.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: DB Session ID: BA3PJCS6CMPJCKIT4DSK 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: CURRENT file: CURRENT 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.error_if_exists: 0 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.create_if_missing: 0 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.paranoid_checks: 1 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T16:24:01.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.env: 0x55e95486e500 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.info_log: 0x55e955a5ffc0 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.statistics: (nil) 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.use_fsync: 0 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_log_file_size: 0 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.allow_fallocate: 1 2024-04-25T16:24:01.269 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.use_direct_reads: 0 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.db_log_dir: 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.wal_dir: 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T16:24:01.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.write_buffer_manager: 0x55e955a308c0 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.unordered_write: 0 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T16:24:01.271 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.row_cache: None 2024-04-25T16:24:01.272 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.wal_filter: None 2024-04-25T16:24:01.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T16:24:01.273 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.two_write_queues: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.wal_compression: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.atomic_flush: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.log_readahead_size: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T16:24:01.274 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_background_jobs: 2 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_background_compactions: -1 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_subcompactions: 1 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T16:24:01.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_open_files: -1 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_background_flushes: -1 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Compression algorithms supported: 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: kZSTD supported: 0 2024-04-25T16:24:01.276 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: kXpressCompression supported: 0 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: kZlibCompression supported: 1 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: kSnappyCompression supported: 1 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: kLZ4Compression supported: 1 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: kBZip2Compression supported: 0 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T16:24:01.277 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.merge_operator: 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_filter: None 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e955b661a0) 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_type: 0 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_index_type: 0 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_shortening: 1 2024-04-25T16:24:01.278 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: checksum: 4 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: no_block_cache: 0 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache: 0x55e955a631f0 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_options: 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: capacity : 536870912 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_shard_bits : 4 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: strict_capacity_limit : 0 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_compressed: (nil) 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: persistent_cache: (nil) 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size: 4096 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size_deviation: 10 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_restart_interval: 16 2024-04-25T16:24:01.279 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_block_restart_interval: 1 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: metadata_block_size: 4096 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: partition_filters: 0 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: use_delta_encoding: 1 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: filter_policy: bloomfilter 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: whole_key_filtering: 1 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: verify_compression: 0 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: format_version: 5 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_index_compression: 1 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_align: 0 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: max_auto_readahead_size: 262144 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: prepopulate_block_cache: 0 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: initial_auto_readahead_size: 8192 2024-04-25T16:24:01.280 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression: NoCompression 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.num_levels: 7 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T16:24:01.281 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T16:24:01.282 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T16:24:01.283 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T16:24:01.284 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.inplace_update_support: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.bloom_locality: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.max_successive_merges: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.ttl: 2592000 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T16:24:01.285 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.enable_blob_files: false 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.min_blob_size: 0 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T16:24:01.286 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c73ff377-7d28-4be5-884c-b0f32b451761 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062241041723, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062241042376, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714062241, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c73ff377-7d28-4be5-884c-b0f32b451761", "db_session_id": "BA3PJCS6CMPJCKIT4DSK", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714062241042476, "job": 1, "event": "recovery_finished"} 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e955a47c00 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: DB pointer 0x55e95a83a000 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** DB Stats ** 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T16:24:01.287 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-04-25T16:24:01.288 INFO:journalctl@ceph.mon.b.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval compaction: 0.00 GB write, 0.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T16:24:01.289 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache BinnedLRUCache@0x55e955a631f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: starting mon.b rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(???) e0 preinit fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout: e1 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:24:01.290 INFO:journalctl@ceph.mon.b.smithi173.stdout: legacy client fscid: -1 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout: No filesystems configured 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T16:24:01.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Deploying cephadm binary to smithi203 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Updating smithi173:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Updating smithi173:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:24:01.292 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Added host smithi203 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.293 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2717113200' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2717113200' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi203", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Updating smithi203:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Updating smithi203:/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Updating smithi203:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:01.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Deploying daemon mon.c on smithi203 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.a calling monitor election 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.c calling monitor election 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:01.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: monmap epoch 2 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: last_changed 2024-04-25T16:23:52.247764+0000 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: created 2024-04-25T16:22:06.623046+0000 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: min_mon_release 19 (squid) 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: election_strategy: 1 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: fsmap 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mgrmap e13: x(active, since 49s) 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: overall HEALTH_OK 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.297 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: Deploying daemon mon.b on smithi173 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/1636897391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 ceph-mon[32556]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 bash[32535]: 8af9d65e7b935c6eb496e1d3e29c15322b86868dd1ea6e54ec8f8a23e95d6997 2024-04-25T16:24:01.298 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:01 smithi173 systemd[1]: Started Ceph mon.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:24:06.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:24:06.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: mon.a calling monitor election 2024-04-25T16:24:06.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: mon.c calling monitor election 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: mon.b calling monitor election 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: monmap epoch 3 2024-04-25T16:24:06.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: last_changed 2024-04-25T16:24:01.074691+0000 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: created 2024-04-25T16:22:06.623046+0000 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: min_mon_release 19 (squid) 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: election_strategy: 1 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: fsmap 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: mgrmap e13: x(active, since 58s) 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: overall HEALTH_OK 2024-04-25T16:24:06.442 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.443 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.443 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:24:06.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: mon.a calling monitor election 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: mon.c calling monitor election 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: mon.b calling monitor election 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T16:24:06.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: monmap epoch 3 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: last_changed 2024-04-25T16:24:01.074691+0000 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: created 2024-04-25T16:22:06.623046+0000 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: min_mon_release 19 (squid) 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: election_strategy: 1 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: fsmap 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: mgrmap e13: x(active, since 58s) 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: overall HEALTH_OK 2024-04-25T16:24:06.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:24:06.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: mon.a calling monitor election 2024-04-25T16:24:06.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:24:06.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: mon.c calling monitor election 2024-04-25T16:24:06.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: mon.b calling monitor election 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: monmap epoch 3 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: last_changed 2024-04-25T16:24:01.074691+0000 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: created 2024-04-25T16:22:06.623046+0000 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: min_mon_release 19 (squid) 2024-04-25T16:24:06.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: election_strategy: 1 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: 0: [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon.a 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: 1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.c 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: 2: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: fsmap 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: mgrmap e13: x(active, since 58s) 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: overall HEALTH_OK 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:06.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.642+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3938223626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 msgr2=0x7f6e48005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:06.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.642+0000 7f6e63e3b700 1 --2- 172.21.15.203:0/3938223626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 0x7f6e48005aa0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c103a10 tx=0x7f6e5800fe60 comp rx=0 tx=0).stop 2024-04-25T16:24:06.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.642+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3938223626 shutdown_connections 2024-04-25T16:24:06.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.642+0000 7f6e63e3b700 1 --2- 172.21.15.203:0/3938223626 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 0x7f6e48005aa0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:06.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.642+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3938223626 >> 172.21.15.203:0/3938223626 conn(0x7f6e5c0ff620 msgr2=0x7f6e5c101a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:06.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.643+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3938223626 shutdown_connections 2024-04-25T16:24:06.645 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.643+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3938223626 wait complete. 2024-04-25T16:24:06.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e63e3b700 1 Processor -- start 2024-04-25T16:24:06.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e63e3b700 1 -- start start 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e63e3b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 0x7f6e5c1a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e63e3b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 0x7f6e5c19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e63e3b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 0x7f6e5c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e63e3b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c075fa0 con 0x7f6e480035c0 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e63e3b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c075e20 con 0x7f6e5c1a1de0 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e63e3b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5c076120 con 0x7f6e5c19c000 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e623d8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 0x7f6e5c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e623d8700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 0x7f6e5c19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:50830/0 (socket says 172.21.15.203:50830) 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e623d8700 1 -- 172.21.15.203:0/3768789698 learned_addr learned my addr 172.21.15.203:0/3768789698 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e613d6700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 0x7f6e5c19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e623d8700 1 -- 172.21.15.203:0/3768789698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 msgr2=0x7f6e5c19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-25T16:24:06.648 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e623d8700 1 -- 172.21.15.203:0/3768789698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 msgr2=0x7f6e5c19c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T16:24:06.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e623d8700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 0x7f6e5c19c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:24:06.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e613d6700 1 -- 172.21.15.203:0/3768789698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 msgr2=0x7f6e5c19bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-25T16:24:06.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e613d6700 1 -- 172.21.15.203:0/3768789698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 msgr2=0x7f6e5c19bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T16:24:06.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e613d6700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 0x7f6e5c19bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:24:06.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e623d8700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 0x7f6e5c19c410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:24:06.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.644+0000 7f6e613d6700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 0x7f6e5c19bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:24:06.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.646+0000 7f6e61bd7700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 0x7f6e5c1a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:06.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.647+0000 7f6e61bd7700 1 -- 172.21.15.203:0/3768789698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 msgr2=0x7f6e5c19c410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:24:06.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.647+0000 7f6e61bd7700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 0x7f6e5c19c410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:06.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.647+0000 7f6e61bd7700 1 -- 172.21.15.203:0/3768789698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 msgr2=0x7f6e5c19bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:24:06.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.647+0000 7f6e61bd7700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 0x7f6e5c19bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:06.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.647+0000 7f6e61bd7700 1 -- 172.21.15.203:0/3768789698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5c1a8730 con 0x7f6e480035c0 2024-04-25T16:24:06.651 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.648+0000 7f6e61bd7700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 0x7f6e5c1a18a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6e58020040 tx=0x7f6e58005ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.649+0000 7f6e52ffd700 1 -- 172.21.15.203:0/3768789698 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e58022040 con 0x7f6e480035c0 2024-04-25T16:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.649+0000 7f6e52ffd700 1 -- 172.21.15.203:0/3768789698 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e5802c070 con 0x7f6e480035c0 2024-04-25T16:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.649+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e5c1a8960 con 0x7f6e480035c0 2024-04-25T16:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.649+0000 7f6e52ffd700 1 -- 172.21.15.203:0/3768789698 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e58014330 con 0x7f6e480035c0 2024-04-25T16:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.649+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e5c1a8ec0 con 0x7f6e480035c0 2024-04-25T16:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.650+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5c064cf0 con 0x7f6e480035c0 2024-04-25T16:24:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.652+0000 7f6e52ffd700 1 -- 172.21.15.203:0/3768789698 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f6e580144d0 con 0x7f6e480035c0 2024-04-25T16:24:06.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.652+0000 7f6e52ffd700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e4003daa0 0x7f6e4003ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:06.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.652+0000 7f6e52ffd700 1 -- 172.21.15.203:0/3768789698 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6e58053fd0 con 0x7f6e480035c0 2024-04-25T16:24:06.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.652+0000 7f6e613d6700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e4003daa0 0x7f6e4003ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:06.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.653+0000 7f6e613d6700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e4003daa0 0x7f6e4003ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c107e50 tx=0x7f6e4c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:06.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.658+0000 7f6e52ffd700 1 -- 172.21.15.203:0/3768789698 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e58003c80 con 0x7f6e480035c0 2024-04-25T16:24:06.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.961+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f6e5c02e510 con 0x7f6e480035c0 2024-04-25T16:24:06.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.961+0000 7f6e52ffd700 1 -- 172.21.15.203:0/3768789698 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f6e5800de30 con 0x7f6e480035c0 2024-04-25T16:24:06.964 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-25T16:24:06.964 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":3,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","modified":"2024-04-25T16:24:01.074691Z","created":"2024-04-25T16:22:06.623046Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:3300","nonce":0},{"type":"v1","addr":"172.21.15.203:6789","nonce":0}]},"addr":"172.21.15.203:6789/0","public_addr":"172.21.15.203:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e4003daa0 msgr2=0x7f6e4003ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e4003daa0 0x7f6e4003ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c107e50 tx=0x7f6e4c008040 comp rx=0 tx=0).stop 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 msgr2=0x7f6e5c1a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 0x7f6e5c1a18a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6e58020040 tx=0x7f6e58005ab0 comp rx=0 tx=0).stop 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 shutdown_connections 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e4003daa0 0x7f6e4003ffb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e5c19c000 0x7f6e5c19c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e480035c0 0x7f6e5c1a18a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:06.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 --2- 172.21.15.203:0/3768789698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e5c1a1de0 0x7f6e5c19bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:06.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 >> 172.21.15.203:0/3768789698 conn(0x7f6e5c0ff620 msgr2=0x7f6e5c1065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:06.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 shutdown_connections 2024-04-25T16:24:06.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:06.964+0000 7f6e63e3b700 1 -- 172.21.15.203:0/3768789698 wait complete. 2024-04-25T16:24:06.967 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 3 2024-04-25T16:24:07.285 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:07.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:07.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:07 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/3768789698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:24:07.286 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:07.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:07.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:07.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:07 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/3768789698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:24:07.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:07.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:07.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:07.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:07 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/3768789698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T16:24:07.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:24:08.284 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T16:24:08.285 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T16:24:08.504 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:24:08.071+0000 7fb04252c700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T16:24:08.526 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:24:08.850 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: Updating smithi098:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: Updating smithi173:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: Updating smithi203:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.851 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:08.852 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:08.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:08.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: Updating smithi098:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: Updating smithi173:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: Updating smithi203:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:08.942 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:09.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: Updating smithi098:/etc/ceph/ceph.conf 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: Updating smithi173:/etc/ceph/ceph.conf 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: Updating smithi203:/etc/ceph/ceph.conf 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: Updating smithi098:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: Updating smithi173:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: Updating smithi203:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:09.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:09.730 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:09 smithi098 ceph-mon[29162]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T16:24:09.730 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:09 smithi098 ceph-mon[29162]: Reconfiguring daemon mon.a on smithi098 2024-04-25T16:24:09.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:09 smithi203 ceph-mon[33115]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T16:24:09.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:09 smithi203 ceph-mon[33115]: Reconfiguring daemon mon.a on smithi098 2024-04-25T16:24:10.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:09 smithi173 ceph-mon[32556]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T16:24:10.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:09 smithi173 ceph-mon[32556]: Reconfiguring daemon mon.a on smithi098 2024-04-25T16:24:10.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.604+0000 7ff6f39f6700 1 -- 172.21.15.98:0/3590982575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 msgr2=0x7ff6ec104fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:10.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.604+0000 7ff6f39f6700 1 --2- 172.21.15.98:0/3590982575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 0x7ff6ec104fd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400b0a0 tx=0x7ff6e400b3b0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.604+0000 7ff6f39f6700 1 -- 172.21.15.98:0/3590982575 shutdown_connections 2024-04-25T16:24:10.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.604+0000 7ff6f39f6700 1 --2- 172.21.15.98:0/3590982575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 0x7ff6ec104fd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.604+0000 7ff6f39f6700 1 -- 172.21.15.98:0/3590982575 >> 172.21.15.98:0/3590982575 conn(0x7ff6ec0fe430 msgr2=0x7ff6ec100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.605+0000 7ff6f39f6700 1 -- 172.21.15.98:0/3590982575 shutdown_connections 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.605+0000 7ff6f39f6700 1 -- 172.21.15.98:0/3590982575 wait complete. 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.605+0000 7ff6f39f6700 1 Processor -- start 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.605+0000 7ff6f39f6700 1 -- start start 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.605+0000 7ff6f39f6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 0x7ff6ec19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.605+0000 7ff6f39f6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6ec19fb40 0x7ff6ec199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.605+0000 7ff6f39f6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6ec199d80 0x7ff6ec19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f39f6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec10de20 con 0x7ff6ec104bc0 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f39f6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec10dca0 con 0x7ff6ec199d80 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f39f6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec10dfa0 con 0x7ff6ec19fb40 2024-04-25T16:24:10.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 0x7ff6ec19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 0x7ff6ec19f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40978/0 (socket says 172.21.15.98:40978) 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 -- 172.21.15.98:0/2819236096 learned_addr learned my addr 172.21.15.98:0/2819236096 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 -- 172.21.15.98:0/2819236096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6ec19fb40 msgr2=0x7ff6ec199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6ec19fb40 0x7ff6ec199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 -- 172.21.15.98:0/2819236096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6ec199d80 msgr2=0x7ff6ec19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1f93700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6ec199d80 0x7ff6ec19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6ec199d80 0x7ff6ec19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 -- 172.21.15.98:0/2819236096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ec1a6460 con 0x7ff6ec104bc0 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f1792700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 0x7ff6ec19f600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400d010 tx=0x7ff6e4007920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6ea7fc700 1 -- 172.21.15.98:0/2819236096 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e401f020 con 0x7ff6ec104bc0 2024-04-25T16:24:10.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6ea7fc700 1 -- 172.21.15.98:0/2819236096 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6e400bdd0 con 0x7ff6ec104bc0 2024-04-25T16:24:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ec1a66f0 con 0x7ff6ec104bc0 2024-04-25T16:24:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6ea7fc700 1 -- 172.21.15.98:0/2819236096 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4017040 con 0x7ff6ec104bc0 2024-04-25T16:24:10.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.606+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ec1a6c00 con 0x7ff6ec104bc0 2024-04-25T16:24:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.607+0000 7ff6ea7fc700 1 -- 172.21.15.98:0/2819236096 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7ff6e4022020 con 0x7ff6ec104bc0 2024-04-25T16:24:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.607+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ec06c270 con 0x7ff6ec104bc0 2024-04-25T16:24:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.607+0000 7ff6ea7fc700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff6d403daa0 0x7ff6d403ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.607+0000 7ff6ea7fc700 1 -- 172.21.15.98:0/2819236096 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff6e400f020 con 0x7ff6ec104bc0 2024-04-25T16:24:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.608+0000 7ff6f0f91700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff6d403daa0 0x7ff6d403ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:10.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.608+0000 7ff6f0f91700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff6d403daa0 0x7ff6d403ffb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec106fe0 tx=0x7ff6e0006dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:10.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.613+0000 7ff6ea7fc700 1 -- 172.21.15.98:0/2819236096 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6e402b980 con 0x7ff6ec104bc0 2024-04-25T16:24:10.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.819+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ff6ec064cf0 con 0x7ff6ec104bc0 2024-04-25T16:24:10.822 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:10 smithi098 ceph-mon[29162]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:10.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.820+0000 7ff6ea7fc700 1 -- 172.21.15.98:0/2819236096 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7ff6e4018250 con 0x7ff6ec104bc0 2024-04-25T16:24:10.823 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:24:10.823 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-04-25T16:24:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:24:10.823 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] 2024-04-25T16:24:10.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff6d403daa0 msgr2=0x7ff6d403ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff6d403daa0 0x7ff6d403ffb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec106fe0 tx=0x7ff6e0006dd0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 msgr2=0x7ff6ec19f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 0x7ff6ec19f600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400d010 tx=0x7ff6e4007920 comp rx=0 tx=0).stop 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 shutdown_connections 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff6d403daa0 0x7ff6d403ffb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6ec199d80 0x7ff6ec19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff6ec104bc0 0x7ff6ec19f600 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 --2- 172.21.15.98:0/2819236096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6ec19fb40 0x7ff6ec199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 >> 172.21.15.98:0/2819236096 conn(0x7ff6ec0fe430 msgr2=0x7ff6ec10a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 shutdown_connections 2024-04-25T16:24:10.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:10.822+0000 7ff6f39f6700 1 -- 172.21.15.98:0/2819236096 wait complete. 2024-04-25T16:24:10.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:10 smithi203 ceph-mon[33115]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:11.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:10 smithi173 ceph-mon[32556]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:11.327 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T16:24:11.327 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:24:11.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T16:24:11.357 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:24:11.358 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:24:11.422 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:24:11.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T16:24:11.454 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:24:11.454 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:24:11.522 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:24:11.522 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T16:24:11.554 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:24:11.555 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T16:24:11.622 INFO:tasks.cephadm:Adding mgr.x on smithi098 2024-04-25T16:24:11.623 INFO:tasks.cephadm:Adding mgr.y on smithi173 2024-04-25T16:24:11.623 INFO:tasks.cephadm:Adding mgr.z on smithi203 2024-04-25T16:24:11.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi098=x;smithi173=y;smithi203=z' 2024-04-25T16:24:11.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:11 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2819236096' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:11.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:11.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:11.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:11.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:11.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:11.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:11 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2819236096' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:11.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:11.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:11.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:11.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:11.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:11.897 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:11 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2819236096' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:11.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:12.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:12 smithi173 ceph-mon[32556]: Reconfiguring mon.b (monmap changed)... 2024-04-25T16:24:12.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:12 smithi173 ceph-mon[32556]: Reconfiguring daemon mon.b on smithi173 2024-04-25T16:24:12.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:12 smithi173 ceph-mon[32556]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:12.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:12 smithi098 ceph-mon[29162]: Reconfiguring mon.b (monmap changed)... 2024-04-25T16:24:12.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:12 smithi098 ceph-mon[29162]: Reconfiguring daemon mon.b on smithi173 2024-04-25T16:24:12.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:12 smithi098 ceph-mon[29162]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:12.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:12 smithi203 ceph-mon[33115]: Reconfiguring mon.b (monmap changed)... 2024-04-25T16:24:12.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:12 smithi203 ceph-mon[33115]: Reconfiguring daemon mon.b on smithi173 2024-04-25T16:24:12.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:12 smithi203 ceph-mon[33115]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:13.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.314+0000 7fc7db24a700 1 -- 172.21.15.203:0/3485400766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d4102210 msgr2=0x7fc7d4102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:13.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.314+0000 7fc7db24a700 1 --2- 172.21.15.203:0/3485400766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d4102210 0x7fc7d4102620 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009910 tx=0x7fc7c8009c20 comp rx=0 tx=0).stop 2024-04-25T16:24:13.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.314+0000 7fc7db24a700 1 -- 172.21.15.203:0/3485400766 shutdown_connections 2024-04-25T16:24:13.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.314+0000 7fc7db24a700 1 --2- 172.21.15.203:0/3485400766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d4102210 0x7fc7d4102620 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:13.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.314+0000 7fc7db24a700 1 -- 172.21.15.203:0/3485400766 >> 172.21.15.203:0/3485400766 conn(0x7fc7d40fd830 msgr2=0x7fc7d40ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:13.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.314+0000 7fc7db24a700 1 -- 172.21.15.203:0/3485400766 shutdown_connections 2024-04-25T16:24:13.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.314+0000 7fc7db24a700 1 -- 172.21.15.203:0/3485400766 wait complete. 2024-04-25T16:24:13.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7db24a700 1 Processor -- start 2024-04-25T16:24:13.317 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7db24a700 1 -- start start 2024-04-25T16:24:13.318 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7db24a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7d4102210 0x7fc7d41070d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7db24a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7d419c510 0x7fc7d419e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7db24a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d419ee70 0x7fc7d41a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7db24a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4108900 con 0x7fc7d419ee70 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7db24a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4108780 con 0x7fc7d419c510 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7db24a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d4108a80 con 0x7fc7d4102210 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7d8fe6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7d4102210 0x7fc7d41070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7d8fe6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7d4102210 0x7fc7d41070d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:40456/0 (socket says 172.21.15.203:40456) 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7d8fe6700 1 -- 172.21.15.203:0/2708051976 learned_addr learned my addr 172.21.15.203:0/2708051976 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7d97e7700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d419ee70 0x7fc7d41a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.315+0000 7fc7d3fff700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7d419c510 0x7fc7d419e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d97e7700 1 -- 172.21.15.203:0/2708051976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7d4102210 msgr2=0x7fc7d41070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:13.319 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d97e7700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7d4102210 0x7fc7d41070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d97e7700 1 -- 172.21.15.203:0/2708051976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7d419c510 msgr2=0x7fc7d419e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d97e7700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7d419c510 0x7fc7d419e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d97e7700 1 -- 172.21.15.203:0/2708051976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d41a1b40 con 0x7fc7d419ee70 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d8fe6700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7d4102210 0x7fc7d41070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d97e7700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d419ee70 0x7fc7d41a1310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00bd50 tx=0x7fc7cc00bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d1ffb700 1 -- 172.21.15.203:0/2708051976 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc019070 con 0x7fc7d419ee70 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d1ffb700 1 -- 172.21.15.203:0/2708051976 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc7cc00cbb0 con 0x7fc7d419ee70 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d41a1e30 con 0x7fc7d419ee70 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7d1ffb700 1 -- 172.21.15.203:0/2708051976 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0125a0 con 0x7fc7d419ee70 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.316+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d41a22a0 con 0x7fc7d419ee70 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.317+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d4064cf0 con 0x7fc7d419ee70 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.317+0000 7fc7d1ffb700 1 -- 172.21.15.203:0/2708051976 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fc7cc01c070 con 0x7fc7d419ee70 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.317+0000 7fc7d1ffb700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc7c403daa0 0x7fc7c403ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:13.320 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.318+0000 7fc7d1ffb700 1 -- 172.21.15.203:0/2708051976 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc7cc050e30 con 0x7fc7d419ee70 2024-04-25T16:24:13.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.318+0000 7fc7d8fe6700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc7c403daa0 0x7fc7c403ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:13.321 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.318+0000 7fc7d8fe6700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc7c403daa0 0x7fc7c403ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80098e0 tx=0x7fc7c801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:13.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.323+0000 7fc7d1ffb700 1 -- 172.21.15.203:0/2708051976 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc7cc00f160 con 0x7fc7d419ee70 2024-04-25T16:24:13.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.541+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}) -- 0x7fc7d40ff6b0 con 0x7fc7c403daa0 2024-04-25T16:24:13.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.547+0000 7fc7d1ffb700 1 -- 172.21.15.203:0/2708051976 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fc7d40ff6b0 con 0x7fc7c403daa0 2024-04-25T16:24:13.548 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mgr update... 2024-04-25T16:24:13.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc7c403daa0 msgr2=0x7fc7c403ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:13.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc7c403daa0 0x7fc7c403ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80098e0 tx=0x7fc7c801a040 comp rx=0 tx=0).stop 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d419ee70 msgr2=0x7fc7d41a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d419ee70 0x7fc7d41a1310 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc00bd50 tx=0x7fc7cc00bd80 comp rx=0 tx=0).stop 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 shutdown_connections 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc7c403daa0 0x7fc7c403ffb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc7d4102210 0x7fc7d41070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc7d419ee70 0x7fc7d41a1310 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 --2- 172.21.15.203:0/2708051976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7d419c510 0x7fc7d419e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.549+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 >> 172.21.15.203:0/2708051976 conn(0x7fc7d40fd830 msgr2=0x7fc7d40fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.550+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 shutdown_connections 2024-04-25T16:24:13.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:24:13.550+0000 7fc7db24a700 1 -- 172.21.15.203:0/2708051976 wait complete. 2024-04-25T16:24:14.081 DEBUG:teuthology.orchestra.run.smithi173:mgr.y> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T16:24:14.085 DEBUG:teuthology.orchestra.run.smithi203:mgr.z> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T16:24:14.087 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T16:24:14.087 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:24:14.087 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T16:24:14.104 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T16:24:14.104 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-04-25T16:24:14.159 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T16:24:14.159 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:14.159 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T16:24:14.159 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:14.159 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:14.159 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 16:23:20.069640831 +0000 2024-04-25T16:24:14.160 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 16:18:40.275406876 +0000 2024-04-25T16:24:14.160 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 16:18:40.275406876 +0000 2024-04-25T16:24:14.160 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-25 16:18:40.275406876 +0000 2024-04-25T16:24:14.160 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T16:24:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-25T16:24:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-25T16:24:14.222 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000198689 s, 2.6 MB/s 2024-04-25T16:24:14.226 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T16:24:14.281 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-04-25T16:24:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T16:24:14.336 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:14.336 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-25T16:24:14.337 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:14.337 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:14.337 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 16:23:20.102640378 +0000 2024-04-25T16:24:14.337 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 16:18:40.616401874 +0000 2024-04-25T16:24:14.337 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 16:18:40.616401874 +0000 2024-04-25T16:24:14.337 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-25 16:18:40.616401874 +0000 2024-04-25T16:24:14.337 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T16:24:14.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: Reconfiguring mon.c (monmap changed)... 2024-04-25T16:24:14.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:14.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:14.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:14.368 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: Reconfiguring daemon mon.c on smithi203 2024-04-25T16:24:14.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:14.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: Saving service mgr spec with placement smithi098=x;smithi173=y;smithi203=z;count:3 2024-04-25T16:24:14.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.369 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:14 smithi203 ceph-mon[33115]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:14.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: Reconfiguring mon.c (monmap changed)... 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: Reconfiguring daemon mon.c on smithi203 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: Saving service mgr spec with placement smithi098=x;smithi173=y;smithi203=z;count:3 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.397 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:14 smithi098 ceph-mon[29162]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:14.398 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-25T16:24:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-25T16:24:14.399 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000185264 s, 2.8 MB/s 2024-04-25T16:24:14.400 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T16:24:14.455 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-04-25T16:24:14.509 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T16:24:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:14.510 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-25T16:24:14.510 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:14.510 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:14.510 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 16:23:20.134639938 +0000 2024-04-25T16:24:14.510 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 16:18:40.949396989 +0000 2024-04-25T16:24:14.510 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 16:18:40.949396989 +0000 2024-04-25T16:24:14.510 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-25 16:18:40.949396989 +0000 2024-04-25T16:24:14.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: Reconfiguring mon.c (monmap changed)... 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: Reconfiguring daemon mon.c on smithi203 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi098=x;smithi173=y;smithi203=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: Saving service mgr spec with placement smithi098=x;smithi173=y;smithi203=z;count:3 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:14.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:14 smithi173 ceph-mon[32556]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-25T16:24:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-25T16:24:14.572 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000225491 s, 2.3 MB/s 2024-04-25T16:24:14.573 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T16:24:14.628 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-04-25T16:24:14.683 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T16:24:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:14.684 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-25T16:24:14.684 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:14.684 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:14.684 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 16:23:20.166639499 +0000 2024-04-25T16:24:14.684 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 16:18:41.300391840 +0000 2024-04-25T16:24:14.684 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 16:18:41.300391840 +0000 2024-04-25T16:24:14.684 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-04-25 16:18:41.300391840 +0000 2024-04-25T16:24:14.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T16:24:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-04-25T16:24:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-04-25T16:24:14.746 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000181206 s, 2.8 MB/s 2024-04-25T16:24:14.747 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T16:24:14.803 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-04-25T16:24:14.803 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T16:24:14.822 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T16:24:14.822 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-04-25T16:24:14.880 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T16:24:14.880 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:14.880 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-25T16:24:14.880 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:14.880 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:14.880 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 16:24:00.487580216 +0000 2024-04-25T16:24:14.881 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 16:18:41.165586599 +0000 2024-04-25T16:24:14.881 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 16:18:41.165586599 +0000 2024-04-25T16:24:14.881 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 16:18:41.165586599 +0000 2024-04-25T16:24:14.881 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T16:24:14.947 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T16:24:14.947 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T16:24:14.947 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000310362 s, 1.6 MB/s 2024-04-25T16:24:14.949 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T16:24:15.010 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 16:24:00.487580216 +0000 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 16:18:41.489579305 +0000 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 16:18:41.489579305 +0000 2024-04-25T16:24:15.068 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 16:18:41.489579305 +0000 2024-04-25T16:24:15.069 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T16:24:15.134 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T16:24:15.134 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T16:24:15.134 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000229508 s, 2.2 MB/s 2024-04-25T16:24:15.135 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T16:24:15.193 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 16:24:00.488580195 +0000 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 16:18:41.830571629 +0000 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 16:18:41.830571629 +0000 2024-04-25T16:24:15.250 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 16:18:41.830571629 +0000 2024-04-25T16:24:15.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T16:24:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T16:24:15.314 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T16:24:15.315 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000249047 s, 2.1 MB/s 2024-04-25T16:24:15.316 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T16:24:15.375 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-04-25 16:24:00.488580195 +0000 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-04-25 16:18:42.171564230 +0000 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-04-25 16:18:42.171564230 +0000 2024-04-25T16:24:15.431 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-04-25 16:18:42.171564230 +0000 2024-04-25T16:24:15.432 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T16:24:15.495 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-04-25T16:24:15.495 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-04-25T16:24:15.495 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000219244 s, 2.3 MB/s 2024-04-25T16:24:15.497 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T16:24:15.554 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-25T16:24:15.554 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T16:24:15.571 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T16:24:15.571 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-04-25T16:24:15.627 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T16:24:15.627 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:15.627 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T16:24:15.627 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:15.627 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:15.627 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 16:23:51.111178383 +0000 2024-04-25T16:24:15.627 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 16:18:41.280879467 +0000 2024-04-25T16:24:15.627 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 16:18:41.280879467 +0000 2024-04-25T16:24:15.628 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-25 16:18:41.280879467 +0000 2024-04-25T16:24:15.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T16:24:15.691 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T16:24:15.691 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T16:24:15.691 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000347962 s, 1.5 MB/s 2024-04-25T16:24:15.692 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T16:24:15.749 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 16:23:51.111178383 +0000 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 16:18:41.624870759 +0000 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 16:18:41.624870759 +0000 2024-04-25T16:24:15.804 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-25 16:18:41.624870759 +0000 2024-04-25T16:24:15.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T16:24:15.867 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T16:24:15.867 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T16:24:15.867 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000231578 s, 2.2 MB/s 2024-04-25T16:24:15.868 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T16:24:15.923 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 16:23:51.112178359 +0000 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 16:18:41.977861823 +0000 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 16:18:41.977861823 +0000 2024-04-25T16:24:15.979 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-25 16:18:41.977861823 +0000 2024-04-25T16:24:15.980 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T16:24:16.041 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T16:24:16.041 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T16:24:16.041 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000201332 s, 2.5 MB/s 2024-04-25T16:24:16.042 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T16:24:16.098 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-04-25T16:24:16.154 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T16:24:16.154 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T16:24:16.154 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-25T16:24:16.154 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:24:16.154 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T16:24:16.154 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-25 16:23:51.112178359 +0000 2024-04-25T16:24:16.154 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-25 16:18:42.318853191 +0000 2024-04-25T16:24:16.154 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-25 16:18:42.318853191 +0000 2024-04-25T16:24:16.155 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-25 16:18:42.318853191 +0000 2024-04-25T16:24:16.155 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T16:24:16.217 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-25T16:24:16.217 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-25T16:24:16.217 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000209544 s, 2.4 MB/s 2024-04-25T16:24:16.218 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T16:24:16.273 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-04-25T16:24:16.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:24:16.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:16.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:16.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: Deploying daemon mgr.z on smithi203 2024-04-25T16:24:16.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:16 smithi203 ceph-mon[33115]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:16.499 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:24:16.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:16.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:16.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:16.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:16.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:24:16.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:16.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:16.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: Deploying daemon mgr.z on smithi203 2024-04-25T16:24:16.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:16 smithi098 ceph-mon[29162]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:16.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: Deploying daemon mgr.z on smithi203 2024-04-25T16:24:16.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:16 smithi173 ceph-mon[32556]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:17.940 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:17 smithi203 systemd[1]: Starting Ceph mgr.z for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:24:18.280 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 podman[34954]: 2024-04-25 16:24:17.923324777 +0000 UTC m=+0.018829057 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:24:18.280 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 podman[34954]: 2024-04-25 16:24:18.060499144 +0000 UTC m=+0.156003466 container create 8c8b25a83d7d5225493c53ec5fbb34359a86c315b3e4bc02ad929d348263de9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T16:24:18.280 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 podman[34954]: 2024-04-25 16:24:18.279573601 +0000 UTC m=+0.375077882 container init 8c8b25a83d7d5225493c53ec5fbb34359a86c315b3e4bc02ad929d348263de9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T16:24:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T16:24:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T16:24:18.294 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-25T16:24:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-25T16:24:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126752 s, 827 MB/s 2024-04-25T16:24:18.294 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 2024-04-25T16:24:18.294 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-25T16:24:18.639 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 podman[34954]: 2024-04-25 16:24:18.282477155 +0000 UTC m=+0.377981442 container start 8c8b25a83d7d5225493c53ec5fbb34359a86c315b3e4bc02ad929d348263de9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T16:24:18.639 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mgr[34973]: -- 172.21.15.203:0/3779377989 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5559a0dce5a0 con 0x5559a5b58900 2024-04-25T16:24:18.639 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 bash[34954]: 8c8b25a83d7d5225493c53ec5fbb34359a86c315b3e4bc02ad929d348263de9c 2024-04-25T16:24:18.639 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 systemd[1]: Started Ceph mgr.z for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:24:18.639 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:18.455+0000 7f4aee7a7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:24:18.639 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:18.637+0000 7f4aee7a7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:24:18.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-04-25T16:24:18.901 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:18.741+0000 7f4aee7a7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:24:19.123 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:24:19.176 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:18 smithi173 ceph-mon[32556]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:19.176 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.176 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.176 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.176 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:19.176 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:24:19.176 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:19.176 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:19.189 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:19.049+0000 7f4aee7a7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:24:19.189 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:19.132+0000 7f4aee7a7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:24:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mon[33115]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:24:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:19.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:18 smithi098 ceph-mon[29162]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:19.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:19.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:19.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T16:24:19.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:19.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:20.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:19 smithi098 ceph-mon[29162]: Deploying daemon mgr.y on smithi173 2024-04-25T16:24:20.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:19 smithi203 ceph-mon[33115]: Deploying daemon mgr.y on smithi173 2024-04-25T16:24:20.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:19 smithi173 ceph-mon[32556]: Deploying daemon mgr.y on smithi173 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 -- 172.21.15.98:0/1569111308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40105ce0 msgr2=0x7fef4010c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 --2- 172.21.15.98:0/1569111308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40105ce0 0x7fef4010c590 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fef3800b0a0 tx=0x7fef3800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 -- 172.21.15.98:0/1569111308 shutdown_connections 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 --2- 172.21.15.98:0/1569111308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef40104210 0x7fef40104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 --2- 172.21.15.98:0/1569111308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40105ce0 0x7fef4010c590 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 --2- 172.21.15.98:0/1569111308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40105340 0x7fef40105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 -- 172.21.15.98:0/1569111308 >> 172.21.15.98:0/1569111308 conn(0x7fef400ff870 msgr2=0x7fef40101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 -- 172.21.15.98:0/1569111308 shutdown_connections 2024-04-25T16:24:20.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.636+0000 7fef48352700 1 -- 172.21.15.98:0/1569111308 wait complete. 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef48352700 1 Processor -- start 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef48352700 1 -- start start 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef48352700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40104210 0x7fef40078440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef48352700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef40105340 0x7fef40076af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef48352700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40105ce0 0x7fef40077030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef48352700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef4010fb20 con 0x7fef40104210 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef48352700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef4010f9a0 con 0x7fef40105ce0 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef48352700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef4010fca0 con 0x7fef40105340 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef460ee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40104210 0x7fef40078440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef460ee700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40104210 0x7fef40078440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50358/0 (socket says 172.21.15.98:50358) 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef460ee700 1 -- 172.21.15.98:0/1900859748 learned_addr learned my addr 172.21.15.98:0/1900859748 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef458ed700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef40105340 0x7fef40076af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.637+0000 7fef468ef700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40105ce0 0x7fef40077030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef460ee700 1 -- 172.21.15.98:0/1900859748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef40105340 msgr2=0x7fef40076af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:20.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef460ee700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef40105340 0x7fef40076af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef460ee700 1 -- 172.21.15.98:0/1900859748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40105ce0 msgr2=0x7fef40077030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef460ee700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40105ce0 0x7fef40077030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef460ee700 1 -- 172.21.15.98:0/1900859748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef40077790 con 0x7fef40104210 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef460ee700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40104210 0x7fef40078440 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef3c00e940 tx=0x7fef3c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c004220 con 0x7fef40104210 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef3c0043c0 con 0x7fef40104210 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef401a4700 con 0x7fef40104210 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c003bf0 con 0x7fef40104210 2024-04-25T16:24:20.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.638+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef401a4b60 con 0x7fef40104210 2024-04-25T16:24:20.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.639+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef40051190 con 0x7fef40104210 2024-04-25T16:24:20.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.639+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fef3c021440 con 0x7fef40104210 2024-04-25T16:24:20.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.639+0000 7fef377fe700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef2c03d850 0x7fef2c03fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:20.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.639+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fef3c0518c0 con 0x7fef40104210 2024-04-25T16:24:20.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.639+0000 7fef458ed700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef2c03d850 0x7fef2c03fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:20.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.640+0000 7fef458ed700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef2c03d850 0x7fef2c03fd60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fef40077df0 tx=0x7fef300073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:20.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.645+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef3c016950 con 0x7fef40104210 2024-04-25T16:24:20.686 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:20.363+0000 7f4aee7a7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:24:20.686 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:20.536+0000 7f4aee7a7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:24:20.712 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:20 smithi173 systemd[1]: Starting Ceph mgr.y for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:24:20.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:20.853+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fef40063ad0 con 0x7fef2c03d850 2024-04-25T16:24:20.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:20 smithi203 ceph-mon[33115]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:20.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:20 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:20.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:20 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:20.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:20 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:20.940 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:20.712+0000 7f4aee7a7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:20 smithi173 ceph-mon[32556]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:20 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:20 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:20 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:20 smithi173 podman[33934]: 2024-04-25 16:24:20.614104471 +0000 UTC m=+0.017927586 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:20 smithi173 podman[33934]: 2024-04-25 16:24:20.714850487 +0000 UTC m=+0.118673584 container create 4390a4506f4a0d308eb3e2cfd2087e42b25924eeb60843a33501e11f3fc378a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:20 smithi173 podman[33934]: 2024-04-25 16:24:20.8920314 +0000 UTC m=+0.295854496 container init 4390a4506f4a0d308eb3e2cfd2087e42b25924eeb60843a33501e11f3fc378a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:20 smithi173 podman[33934]: 2024-04-25 16:24:20.894608614 +0000 UTC m=+0.298431702 container start 4390a4506f4a0d308eb3e2cfd2087e42b25924eeb60843a33501e11f3fc378a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:24:20.994 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:20 smithi173 ceph-mgr[33953]: -- 172.21.15.173:0/1462980907 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fa210525a0 con 0x55fa25ddc000 2024-04-25T16:24:20.995 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:20 smithi173 bash[33934]: 4390a4506f4a0d308eb3e2cfd2087e42b25924eeb60843a33501e11f3fc378a1 2024-04-25T16:24:21.249 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:20 smithi173 systemd[1]: Started Ceph mgr.y for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:24:21.250 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:21.058+0000 7fa06da9f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T16:24:21.250 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:21.248+0000 7fa06da9f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:24:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:20 smithi098 ceph-mon[29162]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:20 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:20 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:20 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:21.309 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:21.004+0000 7f4aee7a7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:24:21.309 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:21.098+0000 7f4aee7a7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:24:21.309 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:21.308+0000 7f4aee7a7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:24:21.516 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:21.356+0000 7fa06da9f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T16:24:21.690 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:21.509+0000 7f4aee7a7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:24:21.989 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:21.680+0000 7fa06da9f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T16:24:21.989 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:21.778+0000 7fa06da9f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T16:24:21.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-mon[32556]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:21.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:21.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:21.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:21.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:21.990 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:21 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:22.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:21 smithi098 ceph-mon[29162]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:22.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:22.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:22.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:22.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:22.048 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:22.149+0000 7f4aee7a7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-mon[33115]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:22.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:21 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:22.881 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:22.580+0000 7f4aee7a7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:24:22.881 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:22.679+0000 7f4aee7a7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:24:22.882 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:22.879+0000 7f4aee7a7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:24:23.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:23.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:23.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:23.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:23.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:23.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:23.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:23.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:23.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:23.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.198 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:23.027+0000 7fa06da9f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T16:24:23.198 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:23.197+0000 7fa06da9f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T16:24:23.198 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:23.198 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.198 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.198 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.198 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.199 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:23.199 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:23.199 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.199 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T16:24:23.199 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T16:24:23.199 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:23.199 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:23.516 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:23.370+0000 7fa06da9f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T16:24:23.689 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:23.400+0000 7f4aee7a7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:24:23.690 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:23.491+0000 7f4aee7a7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:24:23.911 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:23.657+0000 7fa06da9f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T16:24:23.911 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:23.748+0000 7fa06da9f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T16:24:24.078 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:23.733+0000 7f4aee7a7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:24:24.078 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:23.866+0000 7f4aee7a7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:24:24.078 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:23.962+0000 7f4aee7a7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:24:24.078 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:24.076+0000 7f4aee7a7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:24:24.079 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-mon[33115]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T16:24:24.079 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-mon[33115]: Reconfiguring daemon mgr.x on smithi098 2024-04-25T16:24:24.079 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2160598175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738a5e3f-5992-4c5f-befd-a6130241c120"}]: dispatch 2024-04-25T16:24:24.079 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2160598175' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "738a5e3f-5992-4c5f-befd-a6130241c120"}]': finished 2024-04-25T16:24:24.079 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-mon[33115]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T16:24:24.079 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:23 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:24.190 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:23 smithi098 ceph-mon[29162]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T16:24:24.191 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:23 smithi098 ceph-mon[29162]: Reconfiguring daemon mgr.x on smithi098 2024-04-25T16:24:24.191 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:23 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2160598175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738a5e3f-5992-4c5f-befd-a6130241c120"}]: dispatch 2024-04-25T16:24:24.191 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:23 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2160598175' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "738a5e3f-5992-4c5f-befd-a6130241c120"}]': finished 2024-04-25T16:24:24.191 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:23 smithi098 ceph-mon[29162]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T16:24:24.191 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:23 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:24.266 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:23.954+0000 7fa06da9f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T16:24:24.266 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:24.144+0000 7fa06da9f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T16:24:24.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-mon[32556]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T16:24:24.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-mon[32556]: Reconfiguring daemon mgr.x on smithi098 2024-04-25T16:24:24.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2160598175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "738a5e3f-5992-4c5f-befd-a6130241c120"}]: dispatch 2024-04-25T16:24:24.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2160598175' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "738a5e3f-5992-4c5f-befd-a6130241c120"}]': finished 2024-04-25T16:24:24.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-mon[32556]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T16:24:24.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:23 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:24.689 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:24.380+0000 7f4aee7a7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:24:25.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:24 smithi173 ceph-mon[32556]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:25.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:24 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1123919160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:25.017 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:24.743+0000 7fa06da9f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T16:24:25.189 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:24.977+0000 7f4aee7a7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:24:25.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:24 smithi203 ceph-mon[33115]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:25.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:24 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1123919160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:25.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:24 smithi098 ceph-mon[29162]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:25.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:24 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1123919160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:25.464 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:25.164+0000 7fa06da9f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T16:24:25.464 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:25.260+0000 7fa06da9f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T16:24:25.464 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:25.462+0000 7fa06da9f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T16:24:25.690 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:24:25 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z[34969]: 2024-04-25T16:24:25.191+0000 7f4aee7a7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:24:25.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:25.920+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 14) ==== 98518+0+0 (secure 0 0 0) 0x7fef3c007d10 con 0x7fef40104210 2024-04-25T16:24:26.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:25 smithi203 ceph-mon[33115]: Standby manager daemon z started 2024-04-25T16:24:26.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:25 smithi203 ceph-mon[33115]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:24:26.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:25 smithi203 ceph-mon[33115]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:24:26.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:25 smithi203 ceph-mon[33115]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:24:26.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:25 smithi203 ceph-mon[33115]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:24:26.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:25 smithi098 ceph-mon[29162]: Standby manager daemon z started 2024-04-25T16:24:26.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:25 smithi098 ceph-mon[29162]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:24:26.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:25 smithi098 ceph-mon[29162]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:24:26.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:25 smithi098 ceph-mon[29162]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:24:26.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:25 smithi098 ceph-mon[29162]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:24:26.266 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:26.015+0000 7fa06da9f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T16:24:26.266 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:26.106+0000 7fa06da9f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T16:24:26.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:25 smithi173 ceph-mon[32556]: Standby manager daemon z started 2024-04-25T16:24:26.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:25 smithi173 ceph-mon[32556]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T16:24:26.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:25 smithi173 ceph-mon[32556]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:24:26.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:25 smithi173 ceph-mon[32556]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T16:24:26.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:25 smithi173 ceph-mon[32556]: from='mgr.? 172.21.15.203:0/1591461295' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:24:26.690 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:26.347+0000 7fa06da9f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T16:24:26.690 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:26.480+0000 7fa06da9f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T16:24:26.690 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:26.574+0000 7fa06da9f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T16:24:26.690 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:26.689+0000 7fa06da9f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T16:24:27.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-mon[32556]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:27.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-mon[32556]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T16:24:27.017 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:24:27.187 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:26 smithi098 ceph-mon[29162]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:27.187 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:26 smithi098 ceph-mon[29162]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T16:24:27.188 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:24:27.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:26 smithi203 ceph-mon[33115]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:27.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:26 smithi203 ceph-mon[33115]: mgrmap e14: x(active, since 78s), standbys: z 2024-04-25T16:24:27.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T16:24:27.516 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:27.015+0000 7fa06da9f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T16:24:27.928 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:27.614+0000 7fa06da9f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T16:24:27.928 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:24:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y[33949]: 2024-04-25T16:24:27.832+0000 7fa06da9f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T16:24:27.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:27.933+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7fef3c00fce0 con 0x7fef40104210 2024-04-25T16:24:28.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:27 smithi098 ceph-mon[29162]: Standby manager daemon y started 2024-04-25T16:24:28.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:27 smithi098 ceph-mon[29162]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:24:28.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:27 smithi098 ceph-mon[29162]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:24:28.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:27 smithi098 ceph-mon[29162]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:24:28.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:27 smithi098 ceph-mon[29162]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:24:28.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:27 smithi203 ceph-mon[33115]: Standby manager daemon y started 2024-04-25T16:24:28.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:27 smithi203 ceph-mon[33115]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:24:28.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:27 smithi203 ceph-mon[33115]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:24:28.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:27 smithi203 ceph-mon[33115]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:24:28.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:27 smithi203 ceph-mon[33115]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:24:28.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:27 smithi173 ceph-mon[32556]: Standby manager daemon y started 2024-04-25T16:24:28.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:27 smithi173 ceph-mon[32556]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T16:24:28.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:27 smithi173 ceph-mon[32556]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T16:24:28.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:27 smithi173 ceph-mon[32556]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T16:24:28.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:27 smithi173 ceph-mon[32556]: from='mgr.? 172.21.15.173:0/35794529' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T16:24:29.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:29.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T16:24:29.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:24:29.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:29.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:29.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:29.007 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:29.007 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:29.007 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:29.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:29.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-04-25T16:24:29.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T16:24:29.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:29.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:29.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:31.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:30 smithi203 ceph-mon[33115]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:31.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:30 smithi098 ceph-mon[29162]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:31.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:30 smithi173 ceph-mon[32556]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:32.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T16:24:32.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:32.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T16:24:32.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T16:24:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:33.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:32 smithi203 ceph-mon[33115]: Deploying daemon osd.0 on smithi098 2024-04-25T16:24:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:32 smithi203 ceph-mon[33115]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:33.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:32 smithi098 ceph-mon[29162]: Deploying daemon osd.0 on smithi098 2024-04-25T16:24:33.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:32 smithi098 ceph-mon[29162]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:33.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:32 smithi173 ceph-mon[32556]: Deploying daemon osd.0 on smithi098 2024-04-25T16:24:33.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:32 smithi173 ceph-mon[32556]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:34.194 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:33 smithi098 ceph-mon[29162]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:34.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:33 smithi173 ceph-mon[32556]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:34.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:33 smithi203 ceph-mon[33115]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:37.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:36 smithi098 ceph-mon[29162]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:37.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:37.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:37.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:37.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:36 smithi203 ceph-mon[33115]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:37.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:36 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:37.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:36 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:37.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:36 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:36 smithi173 ceph-mon[32556]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:36 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:36 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:36 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:39.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:38 smithi098 ceph-mon[29162]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:38 smithi203 ceph-mon[33115]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:38 smithi173 ceph-mon[32556]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:40.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:40.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:40.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:40 smithi098 ceph-mon[29162]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:40.312 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-04-25T16:24:40.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.311+0000 7fef377fe700 1 -- 172.21.15.98:0/1900859748 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fef40063ad0 con 0x7fef2c03d850 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef2c03d850 msgr2=0x7fef2c03fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef2c03d850 0x7fef2c03fd60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fef40077df0 tx=0x7fef300073d0 comp rx=0 tx=0).stop 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40104210 msgr2=0x7fef40078440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40104210 0x7fef40078440 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fef3c00e940 tx=0x7fef3c00ec50 comp rx=0 tx=0).stop 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 shutdown_connections 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef2c03d850 0x7fef2c03fd60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef40105ce0 0x7fef40077030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef40104210 0x7fef40078440 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:40.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 --2- 172.21.15.98:0/1900859748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef40105340 0x7fef40076af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:40.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 >> 172.21.15.98:0/1900859748 conn(0x7fef400ff870 msgr2=0x7fef40108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:40.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 shutdown_connections 2024-04-25T16:24:40.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:24:40.313+0000 7fef48352700 1 -- 172.21.15.98:0/1900859748 wait complete. 2024-04-25T16:24:40.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:40.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:40.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:40 smithi203 ceph-mon[33115]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:40.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:40.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:40.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:40.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:40 smithi173 ceph-mon[32556]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:40.985 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T16:24:40.988 INFO:tasks.cephadm:Deploying osd.1 on smithi173 with /dev/vg_nvme/lv_4... 2024-04-25T16:24:40.988 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T16:24:41.241 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:24:41.487 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:41 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:41.487 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:41 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:41.487 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:41 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:41.487 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:41 smithi098 ceph-mon[29162]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T16:24:41.592 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:41 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:41.593 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:41 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:41.593 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:41 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:41.593 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:41 smithi173 ceph-mon[32556]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T16:24:41.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:41 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:41.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:41 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:41.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:41 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:24:41.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:41 smithi203 ceph-mon[33115]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T16:24:42.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:42 smithi098 ceph-mon[29162]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T16:24:42.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:42 smithi098 ceph-mon[29162]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T16:24:42.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:42 smithi098 ceph-mon[29162]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:24:42.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:42.401 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:42 smithi098 ceph-mon[29162]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:42 smithi173 ceph-mon[32556]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T16:24:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:42 smithi173 ceph-mon[32556]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T16:24:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:42 smithi173 ceph-mon[32556]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:24:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:42.618 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:42 smithi173 ceph-mon[32556]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:42.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:42 smithi203 ceph-mon[33115]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T16:24:42.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:42 smithi203 ceph-mon[33115]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T16:24:42.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:42 smithi203 ceph-mon[33115]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:24:42.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:42.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:42 smithi203 ceph-mon[33115]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:43.235 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T16:24:43.235 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T16:24:43.235 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-25T16:24:43.235 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-25T16:24:43.235 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111922 s, 937 MB/s 2024-04-25T16:24:43.235 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-25T16:24:43.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:43 smithi098 ceph-mon[29162]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:24:43.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:43 smithi098 ceph-mon[29162]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T16:24:43.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:43 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:43 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:43 smithi098 ceph-mon[29162]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' 2024-04-25T16:24:43.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:43 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.334 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:24:43 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0[43971]: 2024-04-25T16:24:43.076+0000 7f7e21fa2700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T16:24:43.334 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:24:43 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0[43971]: 2024-04-25T16:24:43.081+0000 7f7e19d4f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:24:43.554 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:43 smithi173 ceph-mon[32556]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:24:43.554 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:43 smithi173 ceph-mon[32556]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T16:24:43.554 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:43 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:43 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:43 smithi173 ceph-mon[32556]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' 2024-04-25T16:24:43.555 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:43 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:43 smithi203 ceph-mon[33115]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:24:43.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:43 smithi203 ceph-mon[33115]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T16:24:43.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:43 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:43 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:43 smithi203 ceph-mon[33115]: from='osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581]' entity='osd.0' 2024-04-25T16:24:43.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:43 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:43.849 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-04-25T16:24:44.111 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] boot 2024-04-25T16:24:44.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T16:24:44.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:44.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:24:44.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:24:44.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:44.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:44.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:44.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:44.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] boot 2024-04-25T16:24:44.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T16:24:44.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:44.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:24:44.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: osd.0 [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] boot 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T16:24:44.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T16:24:45.443 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:45 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi098 2024-04-25T16:24:45.443 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:45 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi098 to 17088M 2024-04-25T16:24:45.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 -- 172.21.15.173:0/805135633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104250 msgr2=0x7eff64104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:45.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 --2- 172.21.15.173:0/805135633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104250 0x7eff64104660 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7eff58009990 tx=0x7eff58009ca0 comp rx=0 tx=0).stop 2024-04-25T16:24:45.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 -- 172.21.15.173:0/805135633 shutdown_connections 2024-04-25T16:24:45.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 --2- 172.21.15.173:0/805135633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff64103120 0x7eff64103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:45.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 --2- 172.21.15.173:0/805135633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff64104bf0 0x7eff6410b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:45.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 --2- 172.21.15.173:0/805135633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104250 0x7eff64104660 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:45.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 -- 172.21.15.173:0/805135633 >> 172.21.15.173:0/805135633 conn(0x7eff640fe760 msgr2=0x7eff64100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:24:45.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 -- 172.21.15.173:0/805135633 shutdown_connections 2024-04-25T16:24:45.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.659+0000 7eff68ca0700 1 -- 172.21.15.173:0/805135633 wait complete. 2024-04-25T16:24:45.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff68ca0700 1 Processor -- start 2024-04-25T16:24:45.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff68ca0700 1 -- start start 2024-04-25T16:24:45.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff68ca0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff64103120 0x7eff6419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:45.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff68ca0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff64104250 0x7eff6419ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:45.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff68ca0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104bf0 0x7eff641a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:45.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff68ca0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410eaa0 con 0x7eff64103120 2024-04-25T16:24:45.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff68ca0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410e920 con 0x7eff64104bf0 2024-04-25T16:24:45.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff68ca0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7eff6410ec20 con 0x7eff64104250 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff62d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104bf0 0x7eff641a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff62d9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104bf0 0x7eff641a1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58214/0 (socket says 172.21.15.173:58214) 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff62d9d700 1 -- 172.21.15.173:0/3876621424 learned_addr learned my addr 172.21.15.173:0/3876621424 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff61d9b700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff64104250 0x7eff6419ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff6259c700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff64103120 0x7eff6419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff62d9d700 1 -- 172.21.15.173:0/3876621424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff64104250 msgr2=0x7eff6419ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff62d9d700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff64104250 0x7eff6419ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff62d9d700 1 -- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff64103120 msgr2=0x7eff6419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.660+0000 7eff62d9d700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff64103120 0x7eff6419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.661+0000 7eff62d9d700 1 -- 172.21.15.173:0/3876621424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff641a2320 con 0x7eff64104bf0 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.661+0000 7eff6259c700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff64103120 0x7eff6419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.661+0000 7eff62d9d700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104bf0 0x7eff641a1ca0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7eff5c007ee0 tx=0x7eff5c00d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.661+0000 7eff537fe700 1 -- 172.21.15.173:0/3876621424 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c00dbd0 con 0x7eff64104bf0 2024-04-25T16:24:45.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.661+0000 7eff537fe700 1 -- 172.21.15.173:0/3876621424 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7eff5c00dd70 con 0x7eff64104bf0 2024-04-25T16:24:45.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.661+0000 7eff537fe700 1 -- 172.21.15.173:0/3876621424 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0058d0 con 0x7eff64104bf0 2024-04-25T16:24:45.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.661+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff641a2610 con 0x7eff64104bf0 2024-04-25T16:24:45.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.661+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff641a2b50 con 0x7eff64104bf0 2024-04-25T16:24:45.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.662+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff64051190 con 0x7eff64104bf0 2024-04-25T16:24:45.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.663+0000 7eff537fe700 1 -- 172.21.15.173:0/3876621424 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7eff5c01a020 con 0x7eff64104bf0 2024-04-25T16:24:45.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.664+0000 7eff537fe700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7eff4c0afb70 0x7eff4c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:24:45.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.664+0000 7eff537fe700 1 -- 172.21.15.173:0/3876621424 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7eff5c0dcf60 con 0x7eff64104bf0 2024-04-25T16:24:45.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.664+0000 7eff6259c700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7eff4c0afb70 0x7eff4c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:24:45.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.665+0000 7eff6259c700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7eff4c0afb70 0x7eff4c0b2080 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7eff64104150 tx=0x7eff54005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:24:45.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.668+0000 7eff537fe700 1 -- 172.21.15.173:0/3876621424 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff5c010070 con 0x7eff64104bf0 2024-04-25T16:24:45.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:45 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi098 2024-04-25T16:24:45.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:45 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi098 to 17088M 2024-04-25T16:24:45.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:45 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi098 2024-04-25T16:24:45.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:45 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi098 to 17088M 2024-04-25T16:24:45.875 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:24:45.873+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7eff64100830 con 0x7eff4c0afb70 2024-04-25T16:24:46.666 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:46 smithi173 ceph-mon[32556]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T16:24:46.666 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:46 smithi173 ceph-mon[32556]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:46.666 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:46 smithi173 ceph-mon[32556]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:46.666 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:46.666 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:46.666 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:46.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:46 smithi203 ceph-mon[33115]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T16:24:46.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:46 smithi203 ceph-mon[33115]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:46.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:46 smithi203 ceph-mon[33115]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:46.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:46.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:46.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:46.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:46 smithi098 ceph-mon[29162]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T16:24:46.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:46 smithi098 ceph-mon[29162]: pgmap v48: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:46.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:46 smithi098 ceph-mon[29162]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:24:46.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:24:46.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:24:46.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:49.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:48 smithi203 ceph-mon[33115]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:49.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:48 smithi203 ceph-mon[33115]: from='client.? 172.21.15.173:0/2321983736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]: dispatch 2024-04-25T16:24:49.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:48 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]: dispatch 2024-04-25T16:24:49.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:48 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]': finished 2024-04-25T16:24:49.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:48 smithi203 ceph-mon[33115]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T16:24:49.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:24:49.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:48 smithi098 ceph-mon[29162]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:49.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:48 smithi098 ceph-mon[29162]: from='client.? 172.21.15.173:0/2321983736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]: dispatch 2024-04-25T16:24:49.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:48 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]: dispatch 2024-04-25T16:24:49.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:48 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]': finished 2024-04-25T16:24:49.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:48 smithi098 ceph-mon[29162]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T16:24:49.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:24:49.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:48 smithi173 ceph-mon[32556]: pgmap v49: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:49.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:48 smithi173 ceph-mon[32556]: from='client.? 172.21.15.173:0/2321983736' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]: dispatch 2024-04-25T16:24:49.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:48 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]: dispatch 2024-04-25T16:24:49.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:48 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d963fea-ee1e-4f05-af8e-31a28d82c49c"}]': finished 2024-04-25T16:24:49.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:48 smithi173 ceph-mon[32556]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T16:24:49.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:24:50.167 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:49 smithi173 ceph-mon[32556]: from='client.? 172.21.15.173:0/852631604' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:50.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:49 smithi203 ceph-mon[33115]: from='client.? 172.21.15.173:0/852631604' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:50.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:49 smithi098 ceph-mon[29162]: from='client.? 172.21.15.173:0/852631604' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:24:51.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:50 smithi203 ceph-mon[33115]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:51.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:50 smithi098 ceph-mon[29162]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:51.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:50 smithi173 ceph-mon[32556]: pgmap v51: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:53.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:52 smithi173 ceph-mon[32556]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:52 smithi203 ceph-mon[33115]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:52 smithi098 ceph-mon[29162]: pgmap v52: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:55.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:54 smithi173 ceph-mon[32556]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:55.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:54 smithi203 ceph-mon[33115]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:55.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:54 smithi098 ceph-mon[29162]: pgmap v53: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:56.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T16:24:56.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:56.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T16:24:56.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:56.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T16:24:56.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:24:57.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:56 smithi173 ceph-mon[32556]: Deploying daemon osd.1 on smithi173 2024-04-25T16:24:57.016 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:56 smithi173 ceph-mon[32556]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:57.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:56 smithi203 ceph-mon[33115]: Deploying daemon osd.1 on smithi173 2024-04-25T16:24:57.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:56 smithi203 ceph-mon[33115]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:57.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:56 smithi098 ceph-mon[29162]: Deploying daemon osd.1 on smithi173 2024-04-25T16:24:57.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:56 smithi098 ceph-mon[29162]: pgmap v54: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:59.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:24:58 smithi203 ceph-mon[33115]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:59.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:24:58 smithi173 ceph-mon[32556]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:24:59.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:24:58 smithi098 ceph-mon[29162]: pgmap v55: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:01.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:00 smithi203 ceph-mon[33115]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:01.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:00 smithi098 ceph-mon[29162]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:01.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:00 smithi173 ceph-mon[32556]: pgmap v56: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:02.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:02.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:02.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:02.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:02 smithi203 ceph-mon[33115]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:02.459 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:02.460 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:02.460 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:02.460 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:02 smithi173 ceph-mon[32556]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:02.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:02.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:02.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:02.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:02 smithi098 ceph-mon[29162]: pgmap v57: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:04.640 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 1 on host 'smithi173' 2024-04-25T16:25:04.641 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.638+0000 7eff537fe700 1 -- 172.21.15.173:0/3876621424 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7eff64100830 con 0x7eff4c0afb70 2024-04-25T16:25:04.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7eff4c0afb70 msgr2=0x7eff4c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:04.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7eff4c0afb70 0x7eff4c0b2080 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7eff64104150 tx=0x7eff54005f90 comp rx=0 tx=0).stop 2024-04-25T16:25:04.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104bf0 msgr2=0x7eff641a1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:04.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104bf0 0x7eff641a1ca0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7eff5c007ee0 tx=0x7eff5c00d3b0 comp rx=0 tx=0).stop 2024-04-25T16:25:04.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 shutdown_connections 2024-04-25T16:25:04.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7eff4c0afb70 0x7eff4c0b2080 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:04.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff64104bf0 0x7eff641a1ca0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:04.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7eff64103120 0x7eff6419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:04.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 --2- 172.21.15.173:0/3876621424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff64104250 0x7eff6419ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:04.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 >> 172.21.15.173:0/3876621424 conn(0x7eff640fe760 msgr2=0x7eff64100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:04.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 shutdown_connections 2024-04-25T16:25:04.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:04.641+0000 7eff68ca0700 1 -- 172.21.15.173:0/3876621424 wait complete. 2024-04-25T16:25:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:05.197 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:05.197 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.197 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.197 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:05.197 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:05.197 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.197 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.197 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.198 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:05.223 DEBUG:teuthology.orchestra.run.smithi173:osd.1> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T16:25:05.225 INFO:tasks.cephadm:Deploying osd.2 on smithi203 with /dev/vg_nvme/lv_4... 2024-04-25T16:25:05.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T16:25:05.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: pgmap v58: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:05.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:05.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:05.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:05.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:05.470 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:25:06.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:05 smithi203 ceph-mon[33115]: from='osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:25:06.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:05 smithi203 ceph-mon[33115]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:25:06.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:05 smithi098 ceph-mon[29162]: from='osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:25:06.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:05 smithi098 ceph-mon[29162]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:25:06.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:05 smithi173 ceph-mon[32556]: from='osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:25:06.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:05 smithi173 ceph-mon[32556]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T16:25:07.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:06 smithi203 ceph-mon[33115]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:07.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:06 smithi203 ceph-mon[33115]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T16:25:07.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:06 smithi203 ceph-mon[33115]: from='osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:25:07.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:06 smithi203 ceph-mon[33115]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T16:25:07.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:07.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:06 smithi203 ceph-mon[33115]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:25:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:06 smithi098 ceph-mon[29162]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:06 smithi098 ceph-mon[29162]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T16:25:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:06 smithi098 ceph-mon[29162]: from='osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:25:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:06 smithi098 ceph-mon[29162]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T16:25:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:06 smithi098 ceph-mon[29162]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:25:07.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:06 smithi173 ceph-mon[32556]: pgmap v59: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:07.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:06 smithi173 ceph-mon[32556]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T16:25:07.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:06 smithi173 ceph-mon[32556]: from='osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:25:07.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:06 smithi173 ceph-mon[32556]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T16:25:07.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:07.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:06 smithi173 ceph-mon[32556]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:25:07.361 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T16:25:07.361 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T16:25:07.362 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-25T16:25:07.362 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-25T16:25:07.362 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125584 s, 835 MB/s 2024-04-25T16:25:07.362 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-25T16:25:07.925 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:07 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1[37167]: 2024-04-25T16:25:07.660+0000 7f5df71cb700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T16:25:07.926 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:07 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1[37167]: 2024-04-25T16:25:07.666+0000 7f5deef78700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:25:08.028 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:07 smithi203 ceph-mon[33115]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:25:08.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:07 smithi203 ceph-mon[33115]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T16:25:08.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:08.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:08.029 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:07 smithi203 ceph-mon[33115]: from='osd.1 ' entity='osd.1' 2024-04-25T16:25:08.056 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_4 2024-04-25T16:25:08.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:07 smithi098 ceph-mon[29162]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:25:08.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:07 smithi098 ceph-mon[29162]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T16:25:08.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:08.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:08.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:07 smithi098 ceph-mon[29162]: from='osd.1 ' entity='osd.1' 2024-04-25T16:25:08.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:07 smithi173 ceph-mon[32556]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:25:08.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:07 smithi173 ceph-mon[32556]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T16:25:08.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:08.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:08.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:07 smithi173 ceph-mon[32556]: from='osd.1 ' entity='osd.1' 2024-04-25T16:25:08.298 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:25:09.010 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:25:09.010 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:25:09.010 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:09.010 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:09.010 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.010 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.011 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:09.011 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.011 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:09.011 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:09.011 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.011 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] boot 2024-04-25T16:25:09.011 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T16:25:09.011 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:09.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:25:09.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:25:09.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:09.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] boot 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T16:25:09.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:09.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:25:09.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:25:09.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: pgmap v62: 0 pgs: ; 0 B data, 35 MiB used, 89 GiB / 89 GiB avail 2024-04-25T16:25:09.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:09.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:09.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:09.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:09.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:09.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: osd.1 [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] boot 2024-04-25T16:25:09.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T16:25:09.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T16:25:09.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 -- 172.21.15.203:0/1407672075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4105370 msgr2=0x7fc3a4105780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:09.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/1407672075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4105370 0x7fc3a4105780 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc38c009990 tx=0x7fc38c009ca0 comp rx=0 tx=0).stop 2024-04-25T16:25:09.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 -- 172.21.15.203:0/1407672075 shutdown_connections 2024-04-25T16:25:09.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/1407672075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4105d10 0x7fc3a410c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/1407672075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a4104240 0x7fc3a4104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/1407672075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4105370 0x7fc3a4105780 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 -- 172.21.15.203:0/1407672075 >> 172.21.15.203:0/1407672075 conn(0x7fc3a40ff8a0 msgr2=0x7fc3a4101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 -- 172.21.15.203:0/1407672075 shutdown_connections 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.742+0000 7fc3aa06e700 1 -- 172.21.15.203:0/1407672075 wait complete. 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3aa06e700 1 Processor -- start 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3aa06e700 1 -- start start 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3aa06e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4104240 0x7fc3a419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3aa06e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4105370 0x7fc3a419ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3aa06e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a4105d10 0x7fc3a41a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3aa06e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a410fba0 con 0x7fc3a4105d10 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3aa06e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a410fa20 con 0x7fc3a4105370 2024-04-25T16:25:09.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3aa06e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a410fd20 con 0x7fc3a4104240 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3a37fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4104240 0x7fc3a419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3a37fe700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4104240 0x7fc3a419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:53640/0 (socket says 172.21.15.203:53640) 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3a37fe700 1 -- 172.21.15.203:0/3095052537 learned_addr learned my addr 172.21.15.203:0/3095052537 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc3a3fff700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a4105d10 0x7fc3a41a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.743+0000 7fc39affd700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4105370 0x7fc3a419ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a37fe700 1 -- 172.21.15.203:0/3095052537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4105370 msgr2=0x7fc3a419ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a37fe700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4105370 0x7fc3a419ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a37fe700 1 -- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a4105d10 msgr2=0x7fc3a41a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a37fe700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a4105d10 0x7fc3a41a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a37fe700 1 -- 172.21.15.203:0/3095052537 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a41a22a0 con 0x7fc3a4104240 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a37fe700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4104240 0x7fc3a419a740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c970 tx=0x7fc39400cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a17fa700 1 -- 172.21.15.203:0/3095052537 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394004220 con 0x7fc3a4104240 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a17fa700 1 -- 172.21.15.203:0/3095052537 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc3940043c0 con 0x7fc3a4104240 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3a17fa700 1 -- 172.21.15.203:0/3095052537 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394003b60 con 0x7fc3a4104240 2024-04-25T16:25:09.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a41a2590 con 0x7fc3a4104240 2024-04-25T16:25:09.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.744+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a41a2b20 con 0x7fc3a4104240 2024-04-25T16:25:09.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.745+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a4051190 con 0x7fc3a4104240 2024-04-25T16:25:09.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.746+0000 7fc3a17fa700 1 -- 172.21.15.203:0/3095052537 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7fc394003d00 con 0x7fc3a4104240 2024-04-25T16:25:09.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.747+0000 7fc3a17fa700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3900afb70 0x7fc3900b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:09.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.747+0000 7fc3a17fa700 1 -- 172.21.15.203:0/3095052537 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fc3940dce00 con 0x7fc3a4104240 2024-04-25T16:25:09.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.747+0000 7fc39affd700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3900afb70 0x7fc3900b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:09.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.748+0000 7fc39affd700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3900afb70 0x7fc3900b2080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00b5c0 tx=0x7fc38c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:09.753 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.752+0000 7fc3a17fa700 1 -- 172.21.15.203:0/3095052537 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc394014030 con 0x7fc3a4104240 2024-04-25T16:25:09.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:09.958+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc3a4063ad0 con 0x7fc3900afb70 2024-04-25T16:25:10.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:09 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi173 2024-04-25T16:25:10.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:09 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-25T16:25:10.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:09 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi173 2024-04-25T16:25:10.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:09 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-25T16:25:10.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:09 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi173 2024-04-25T16:25:10.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:09 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi173 to 17088M 2024-04-25T16:25:11.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:10 smithi203 ceph-mon[33115]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:11.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:10 smithi203 ceph-mon[33115]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T16:25:11.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:11.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:11.171 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:11.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:10 smithi098 ceph-mon[29162]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:11.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:10 smithi098 ceph-mon[29162]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T16:25:11.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:11.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:11.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:11.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:10 smithi173 ceph-mon[32556]: pgmap v64: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:11.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:10 smithi173 ceph-mon[32556]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T16:25:11.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:11.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:11.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:12.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:11 smithi203 ceph-mon[33115]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:12.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:11 smithi098 ceph-mon[29162]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:12.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:11 smithi173 ceph-mon[32556]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:13.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:12 smithi203 ceph-mon[33115]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:13.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:12 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/1991491337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]: dispatch 2024-04-25T16:25:13.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:12 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]: dispatch 2024-04-25T16:25:13.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:12 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]': finished 2024-04-25T16:25:13.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:12 smithi203 ceph-mon[33115]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T16:25:13.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:12 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:12 smithi098 ceph-mon[29162]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:12 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/1991491337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]: dispatch 2024-04-25T16:25:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:12 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]: dispatch 2024-04-25T16:25:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:12 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]': finished 2024-04-25T16:25:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:12 smithi098 ceph-mon[29162]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T16:25:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:12 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:13.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:12 smithi173 ceph-mon[32556]: pgmap v66: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:13.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:12 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/1991491337' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]: dispatch 2024-04-25T16:25:13.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:12 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]: dispatch 2024-04-25T16:25:13.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:12 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea05c891-ec27-45e1-9226-2f3d5752a73f"}]': finished 2024-04-25T16:25:13.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:12 smithi173 ceph-mon[32556]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T16:25:13.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:12 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:14.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:13 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/1939901185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:14.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:13 smithi098 ceph-mon[29162]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:14.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:13 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/1939901185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:14.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:13 smithi173 ceph-mon[32556]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:14.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:13 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/1939901185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:14.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:13 smithi203 ceph-mon[33115]: pgmap v68: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:17.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:16 smithi203 ceph-mon[33115]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:17.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:16 smithi098 ceph-mon[29162]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:17.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:16 smithi173 ceph-mon[32556]: pgmap v69: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:18.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:18 smithi203 ceph-mon[33115]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:18 smithi098 ceph-mon[29162]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:19.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:18 smithi173 ceph-mon[32556]: pgmap v70: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:20.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:19 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T16:25:20.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:19 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:20.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T16:25:20.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:20.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:19 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T16:25:20.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:19 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:21.182 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:20 smithi173 ceph-mon[32556]: Deploying daemon osd.2 on smithi203 2024-04-25T16:25:21.183 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:20 smithi173 ceph-mon[32556]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:21.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:20 smithi203 ceph-mon[33115]: Deploying daemon osd.2 on smithi203 2024-04-25T16:25:21.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:20 smithi203 ceph-mon[33115]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:20 smithi098 ceph-mon[29162]: Deploying daemon osd.2 on smithi203 2024-04-25T16:25:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:20 smithi098 ceph-mon[29162]: pgmap v71: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:23.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:22 smithi203 ceph-mon[33115]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:22 smithi098 ceph-mon[29162]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:23.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:22 smithi173 ceph-mon[32556]: pgmap v72: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:25.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:24 smithi203 ceph-mon[33115]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:25.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:24 smithi098 ceph-mon[29162]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:25.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:24 smithi173 ceph-mon[32556]: pgmap v73: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:26.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:26.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:26.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:26.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:26 smithi203 ceph-mon[33115]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:26.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:26.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:26.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:26.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:26 smithi098 ceph-mon[29162]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:26.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:26.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:26.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:26.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:26 smithi173 ceph-mon[32556]: pgmap v74: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:28.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.324+0000 7fc3a17fa700 1 -- 172.21.15.203:0/3095052537 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc3a4063ad0 con 0x7fc3900afb70 2024-04-25T16:25:28.327 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 2 on host 'smithi203' 2024-04-25T16:25:28.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3900afb70 msgr2=0x7fc3900b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:28.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3900afb70 0x7fc3900b2080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00b5c0 tx=0x7fc38c01a040 comp rx=0 tx=0).stop 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4104240 msgr2=0x7fc3a419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4104240 0x7fc3a419a740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c970 tx=0x7fc39400cc80 comp rx=0 tx=0).stop 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 shutdown_connections 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3900afb70 0x7fc3900b2080 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc3a4104240 0x7fc3a419a740 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc3a4105d10 0x7fc3a41a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 --2- 172.21.15.203:0/3095052537 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3a4105370 0x7fc3a419ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 >> 172.21.15.203:0/3095052537 conn(0x7fc3a40ff8a0 msgr2=0x7fc3a41085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 shutdown_connections 2024-04-25T16:25:28.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:25:28.326+0000 7fc3aa06e700 1 -- 172.21.15.203:0/3095052537 wait complete. 2024-04-25T16:25:28.924 DEBUG:teuthology.orchestra.run.smithi203:osd.2> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T16:25:28.927 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_3... 2024-04-25T16:25:28.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T16:25:29.170 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:29.191 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2[38294]: 2024-04-25T16:25:28.982+0000 7ff34c21d7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T16:25:29.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:29.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:29.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:29.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:29.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: pgmap v75: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:29.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:30.089 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:29 smithi203 ceph-mon[33115]: from='osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:25:30.089 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:29 smithi203 ceph-mon[33115]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:25:30.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:29 smithi098 ceph-mon[29162]: from='osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:25:30.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:29 smithi098 ceph-mon[29162]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:25:30.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:29 smithi173 ceph-mon[32556]: from='osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:25:30.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:29 smithi173 ceph-mon[32556]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T16:25:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T16:25:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T16:25:31.136 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-25T16:25:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-25T16:25:31.136 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125492 s, 836 MB/s 2024-04-25T16:25:31.136 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-25T16:25:31.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:30 smithi203 ceph-mon[33115]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:31.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:30 smithi203 ceph-mon[33115]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T16:25:31.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:30 smithi203 ceph-mon[33115]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T16:25:31.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:30 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:31.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:30 smithi203 ceph-mon[33115]: from='osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:25:31.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:30 smithi203 ceph-mon[33115]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:25:31.190 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:30 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2[38294]: 2024-04-25T16:25:30.919+0000 7ff341882700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T16:25:31.190 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:30 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2[38294]: 2024-04-25T16:25:30.924+0000 7ff33d637700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:25:31.192 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:30 smithi098 ceph-mon[29162]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:31.192 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:30 smithi098 ceph-mon[29162]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T16:25:31.192 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:30 smithi098 ceph-mon[29162]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T16:25:31.192 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:30 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:31.193 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:30 smithi098 ceph-mon[29162]: from='osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:25:31.193 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:30 smithi098 ceph-mon[29162]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:25:31.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:30 smithi173 ceph-mon[32556]: pgmap v76: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:31.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:30 smithi173 ceph-mon[32556]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T16:25:31.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:30 smithi173 ceph-mon[32556]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T16:25:31.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:30 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:31.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:30 smithi173 ceph-mon[32556]: from='osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:25:31.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:30 smithi173 ceph-mon[32556]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:25:31.749 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-04-25T16:25:32.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:31 smithi098 ceph-mon[29162]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:25:32.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:31 smithi098 ceph-mon[29162]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T16:25:32.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:32.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:32.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:32.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:32.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:32.012 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:25:32.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:31 smithi203 ceph-mon[33115]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:25:32.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:31 smithi203 ceph-mon[33115]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T16:25:32.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:32.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:32.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:32.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:32.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:31 smithi173 ceph-mon[32556]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:25:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:31 smithi173 ceph-mon[32556]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T16:25:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:32.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:33.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:25:33.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:25:33.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:33.154 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi203 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] boot 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:33.155 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi203 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] boot 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T16:25:33.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:33.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:33.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:33.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: pgmap v79: 0 pgs: ; 0 B data, 65 MiB used, 179 GiB / 179 GiB avail 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi203 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi203 to 17088M 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: osd.2 [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] boot 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:33.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 -- 172.21.15.98:0/1085951464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078108d90 msgr2=0x7f807810b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 --2- 172.21.15.98:0/1085951464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078108d90 0x7f807810b2a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f807000b0a0 tx=0x7f807000b3b0 comp rx=0 tx=0).stop 2024-04-25T16:25:33.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 -- 172.21.15.98:0/1085951464 shutdown_connections 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 --2- 172.21.15.98:0/1085951464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8078108d90 0x7f807810b2a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 --2- 172.21.15.98:0/1085951464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f80781039a0 0x7f8078105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 --2- 172.21.15.98:0/1085951464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8078106300 0x7f8078108720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 -- 172.21.15.98:0/1085951464 >> 172.21.15.98:0/1085951464 conn(0x7f80780fda60 msgr2=0x7f80780fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 -- 172.21.15.98:0/1085951464 shutdown_connections 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.637+0000 7f807da31700 1 -- 172.21.15.98:0/1085951464 wait complete. 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f807da31700 1 Processor -- start 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f807da31700 1 -- start start 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f807da31700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80781039a0 0x7f807819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f807da31700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8078106300 0x7f807819ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f807da31700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8078108d90 0x7f80781a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f807da31700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f807810fb40 con 0x7f80781039a0 2024-04-25T16:25:33.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f807da31700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f807810f9c0 con 0x7f8078108d90 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f807da31700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f807810fcc0 con 0x7f8078106300 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f8076ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80781039a0 0x7f807819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f8076ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80781039a0 0x7f807819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53160/0 (socket says 172.21.15.98:53160) 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f8076ffd700 1 -- 172.21.15.98:0/4093562329 learned_addr learned my addr 172.21.15.98:0/4093562329 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f80767fc700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8078106300 0x7f807819ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.638+0000 7f80777fe700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8078108d90 0x7f80781a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f8076ffd700 1 -- 172.21.15.98:0/4093562329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8078106300 msgr2=0x7f807819ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f8076ffd700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8078106300 0x7f807819ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f8076ffd700 1 -- 172.21.15.98:0/4093562329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8078108d90 msgr2=0x7f80781a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f8076ffd700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8078108d90 0x7f80781a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f8076ffd700 1 -- 172.21.15.98:0/4093562329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80781a21b0 con 0x7f80781039a0 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f8076ffd700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80781039a0 0x7f807819a650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f806800e970 tx=0x7f806800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:33.642 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f807ca2f700 1 -- 172.21.15.98:0/4093562329 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806800cbd0 con 0x7f80781039a0 2024-04-25T16:25:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f807ca2f700 1 -- 172.21.15.98:0/4093562329 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f806800cd70 con 0x7f80781039a0 2024-04-25T16:25:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80781a24a0 con 0x7f80781039a0 2024-04-25T16:25:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f807ca2f700 1 -- 172.21.15.98:0/4093562329 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8068005570 con 0x7f80781039a0 2024-04-25T16:25:33.643 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.639+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80781a29e0 con 0x7f80781039a0 2024-04-25T16:25:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.640+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8078051190 con 0x7f80781039a0 2024-04-25T16:25:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.642+0000 7f807ca2f700 1 -- 172.21.15.98:0/4093562329 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f8068004750 con 0x7f80781039a0 2024-04-25T16:25:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.642+0000 7f807ca2f700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f80600afaf0 0x7f80600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.642+0000 7f80767fc700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f80600afaf0 0x7f80600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.642+0000 7f807ca2f700 1 -- 172.21.15.98:0/4093562329 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 2851+0+0 (secure 0 0 0) 0x7f80680dd9c0 con 0x7f80781039a0 2024-04-25T16:25:33.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.643+0000 7f80767fc700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f80600afaf0 0x7f80600b2000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f807819b700 tx=0x7f806c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:33.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.646+0000 7f807ca2f700 1 -- 172.21.15.98:0/4093562329 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8068053ff0 con 0x7f80781039a0 2024-04-25T16:25:33.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:33.854+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8078063ad0 con 0x7f80600afaf0 2024-04-25T16:25:34.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:33 smithi098 ceph-mon[29162]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T16:25:34.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:33 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:34.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:33 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:34.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:33 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:34.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:33 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:25:34.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:33 smithi173 ceph-mon[32556]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T16:25:34.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:33 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:34.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:33 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:34.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:33 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:34.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:33 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:25:34.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:33 smithi203 ceph-mon[33115]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T16:25:34.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:33 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:34.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:33 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:34.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:33 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:34.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:33 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:25:35.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:34 smithi098 ceph-mon[29162]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:35.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:34 smithi098 ceph-mon[29162]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:35.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T16:25:35.012 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:34 smithi098 ceph-mon[29162]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T16:25:35.013 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:25:35.242 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:34 smithi203 ceph-mon[33115]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:35.242 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:34 smithi203 ceph-mon[33115]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:35.242 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T16:25:35.242 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:34 smithi203 ceph-mon[33115]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T16:25:35.242 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:25:35.242 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42339]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:25:35.243 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42339]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.243 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42339]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.243 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42339]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.243 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42342]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:25:35.243 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42342]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.243 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42342]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.243 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42342]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:34 smithi173 ceph-mon[32556]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:25:35.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:34 smithi173 ceph-mon[32556]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:35.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T16:25:35.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:34 smithi173 ceph-mon[32556]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T16:25:35.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:25:35.267 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41311]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:25:35.268 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41311]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.268 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41311]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.268 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41311]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.268 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41314]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:25:35.268 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41314]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.268 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41314]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.268 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41314]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.422 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48383]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:25:35.422 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48383]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48383]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48383]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48386]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:25:35.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48386]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48386]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.423 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48386]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.423 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48374]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:25:35.423 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48374]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.423 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48374]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.423 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48374]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.423 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48377]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:25:35.424 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48377]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.424 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48377]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.424 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:25:35 smithi098 sudo[48377]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42345]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:25:35.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42345]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42345]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42345]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42348]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:25:35.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42348]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42348]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 sudo[42348]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41317]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T16:25:35.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41317]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41317]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41317]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:35.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41320]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T16:25:35.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41320]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T16:25:35.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41320]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T16:25:35.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 sudo[41320]: pam_unix(sudo:session): session closed for user root 2024-04-25T16:25:36.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:25:36.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T16:25:36.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.005 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.006 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:35 smithi098 ceph-mon[29162]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:36.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:36.207+0000 7f807ca2f700 1 -- 172.21.15.98:0/4093562329 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f80680a3760 con 0x7f80781039a0 2024-04-25T16:25:36.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:25:36.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T16:25:36.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:35 smithi173 ceph-mon[32556]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T16:25:36.441 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:35 smithi203 ceph-mon[33115]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:36 smithi098 ceph-mon[29162]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T16:25:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:36 smithi098 ceph-mon[29162]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T16:25:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4047132351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb6d74f9-c730-4a72-a11f-141d71e4ba71"}]: dispatch 2024-04-25T16:25:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4047132351' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb6d74f9-c730-4a72-a11f-141d71e4ba71"}]': finished 2024-04-25T16:25:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:36 smithi098 ceph-mon[29162]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T16:25:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:36 smithi173 ceph-mon[32556]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T16:25:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:36 smithi173 ceph-mon[32556]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T16:25:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4047132351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb6d74f9-c730-4a72-a11f-141d71e4ba71"}]: dispatch 2024-04-25T16:25:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4047132351' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb6d74f9-c730-4a72-a11f-141d71e4ba71"}]': finished 2024-04-25T16:25:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:36 smithi173 ceph-mon[32556]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T16:25:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:36 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:37.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:36 smithi203 ceph-mon[33115]: osdmap e22: 3 total, 3 up, 3 in 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:36 smithi203 ceph-mon[33115]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4047132351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fb6d74f9-c730-4a72-a11f-141d71e4ba71"}]: dispatch 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4047132351' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fb6d74f9-c730-4a72-a11f-141d71e4ba71"}]': finished 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:36 smithi203 ceph-mon[33115]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T16:25:37.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:36 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:38.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:37 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2816221709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:38.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:37 smithi098 ceph-mon[29162]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:38.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:37 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2816221709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:38.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:37 smithi173 ceph-mon[32556]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:38.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:37 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2816221709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:25:38.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:37 smithi203 ceph-mon[33115]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:41.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:40 smithi203 ceph-mon[33115]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:41.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:40 smithi098 ceph-mon[29162]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:40 smithi173 ceph-mon[32556]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:43.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:42 smithi098 ceph-mon[29162]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:43.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:42 smithi203 ceph-mon[33115]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:43.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:42 smithi173 ceph-mon[32556]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:44.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:43 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T16:25:44.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:43 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:44.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:43 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T16:25:44.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:43 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:44.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:43 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T16:25:44.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:43 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:45.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:44 smithi098 ceph-mon[29162]: Deploying daemon osd.3 on smithi098 2024-04-25T16:25:45.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:44 smithi098 ceph-mon[29162]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:45.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:44 smithi203 ceph-mon[33115]: Deploying daemon osd.3 on smithi098 2024-04-25T16:25:45.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:44 smithi203 ceph-mon[33115]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:45.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:44 smithi173 ceph-mon[32556]: Deploying daemon osd.3 on smithi098 2024-04-25T16:25:45.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:44 smithi173 ceph-mon[32556]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:47.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:46 smithi098 ceph-mon[29162]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:47.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:46 smithi203 ceph-mon[33115]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:47.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:46 smithi173 ceph-mon[32556]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:49.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:48 smithi203 ceph-mon[33115]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:49.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:48 smithi098 ceph-mon[29162]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:49.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:48 smithi173 ceph-mon[32556]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:50.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:50.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:50.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:50.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:50 smithi098 ceph-mon[29162]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:50.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:50.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:50.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:50.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:50 smithi173 ceph-mon[32556]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:50.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:50.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:50.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:50.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:50 smithi203 ceph-mon[33115]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:52 smithi203 ceph-mon[33115]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:53.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:53.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:53.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:52 smithi098 ceph-mon[29162]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:52 smithi173 ceph-mon[32556]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:53.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.411+0000 7f807ca2f700 1 -- 172.21.15.98:0/4093562329 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8078063ad0 con 0x7f80600afaf0 2024-04-25T16:25:53.413 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-04-25T16:25:53.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f80600afaf0 msgr2=0x7f80600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f80600afaf0 0x7f80600b2000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f807819b700 tx=0x7f806c00b410 comp rx=0 tx=0).stop 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80781039a0 msgr2=0x7f807819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80781039a0 0x7f807819a650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f806800e970 tx=0x7f806800ec80 comp rx=0 tx=0).stop 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 shutdown_connections 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f80600afaf0 0x7f80600b2000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8078108d90 0x7f80781a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f80781039a0 0x7f807819a650 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 --2- 172.21.15.98:0/4093562329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8078106300 0x7f807819ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.413+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 >> 172.21.15.98:0/4093562329 conn(0x7f80780fda60 msgr2=0x7f8078104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.414+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 shutdown_connections 2024-04-25T16:25:53.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:25:53.414+0000 7f807da31700 1 -- 172.21.15.98:0/4093562329 wait complete. 2024-04-25T16:25:54.146 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T16:25:54.149 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_3... 2024-04-25T16:25:54.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T16:25:54.393 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:25:54.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:54.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:54.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:54.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:54 smithi173 ceph-mon[32556]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T16:25:54.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:54 smithi173 ceph-mon[32556]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:54.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:54.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:54.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:54.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:54 smithi098 ceph-mon[29162]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T16:25:54.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:54 smithi098 ceph-mon[29162]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:54.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:54.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:54.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:25:54.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:54 smithi203 ceph-mon[33115]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T16:25:54.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:54 smithi203 ceph-mon[33115]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:55.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:55 smithi098 ceph-mon[29162]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T16:25:55.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:55 smithi098 ceph-mon[29162]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T16:25:55.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:55 smithi098 ceph-mon[29162]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:25:55.621 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:55 smithi173 ceph-mon[32556]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T16:25:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:55 smithi173 ceph-mon[32556]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T16:25:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:55 smithi173 ceph-mon[32556]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:25:55.737 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:55.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:55 smithi203 ceph-mon[33115]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T16:25:55.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:55 smithi203 ceph-mon[33115]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T16:25:55.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:55 smithi203 ceph-mon[33115]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:25:55.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.405 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3[50507]: 2024-04-25T16:25:56.101+0000 7f2fba210700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T16:25:56.405 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3[50507]: 2024-04-25T16:25:56.108+0000 7f2fb1fbd700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:25:56.502 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T16:25:56.502 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T16:25:56.502 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-25T16:25:56.502 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-25T16:25:56.502 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112587 s, 931 MB/s 2024-04-25T16:25:56.502 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] boot 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T16:25:56.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] boot 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T16:25:56.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:25:56.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:25:56.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:25:56.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: osdmap e25: 4 total, 3 up, 4 in 2024-04-25T16:25:56.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:56.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-04-25T16:25:56.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: from='osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667]' entity='osd.3' 2024-04-25T16:25:56.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: osd.3 [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] boot 2024-04-25T16:25:56.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T16:25:56.941 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T16:25:57.212 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-04-25T16:25:57.456 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:25:58.603 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi098 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi098 to 8544M 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T16:25:58.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:58 smithi173 ceph-mon[32556]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:25:58.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi098 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi098 to 8544M 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:58.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T16:25:58.691 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:58 smithi203 ceph-mon[33115]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:25:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi098 2024-04-25T16:25:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:25:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi098 to 8544M 2024-04-25T16:25:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:58.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:25:58.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:25:58.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T16:25:58.755 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:58 smithi098 ceph-mon[29162]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:25:59.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 -- 172.21.15.173:0/3225203302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 msgr2=0x7fea88103530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:59.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 --2- 172.21.15.173:0/3225203302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 0x7fea88103530 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fea78009990 tx=0x7fea78009ca0 comp rx=0 tx=0).stop 2024-04-25T16:25:59.150 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 -- 172.21.15.173:0/3225203302 shutdown_connections 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 --2- 172.21.15.173:0/3225203302 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea88104bf0 0x7fea8810b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 --2- 172.21.15.173:0/3225203302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 0x7fea88103530 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 --2- 172.21.15.173:0/3225203302 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea88104250 0x7fea88104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 -- 172.21.15.173:0/3225203302 >> 172.21.15.173:0/3225203302 conn(0x7fea880fe760 msgr2=0x7fea88100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 -- 172.21.15.173:0/3225203302 shutdown_connections 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 -- 172.21.15.173:0/3225203302 wait complete. 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.148+0000 7fea8e131700 1 Processor -- start 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea8e131700 1 -- start start 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea8e131700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 0x7fea8819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea8e131700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea88104250 0x7fea8819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea8e131700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea88104bf0 0x7fea881a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea8e131700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fea8810ea10 con 0x7fea88104bf0 2024-04-25T16:25:59.151 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea8e131700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fea8810e890 con 0x7fea88103120 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea8e131700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fea8810eb90 con 0x7fea88104250 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 0x7fea8819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 0x7fea8819a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:58276/0 (socket says 172.21.15.173:58276) 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 -- 172.21.15.173:0/3523987189 learned_addr learned my addr 172.21.15.173:0/3523987189 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea87fff700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea88104bf0 0x7fea881a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea86ffd700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea88104250 0x7fea8819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 -- 172.21.15.173:0/3523987189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea88104250 msgr2=0x7fea8819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea88104250 0x7fea8819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 -- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea88104bf0 msgr2=0x7fea881a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea88104bf0 0x7fea881a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 -- 172.21.15.173:0/3523987189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea881a2210 con 0x7fea88103120 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea86ffd700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea88104250 0x7fea8819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:25:59.152 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.149+0000 7fea877fe700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 0x7fea8819a740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fea780098e0 tx=0x7fea78009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.150+0000 7fea84ff9700 1 -- 172.21.15.173:0/3523987189 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78027070 con 0x7fea88103120 2024-04-25T16:25:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.150+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea881a24a0 con 0x7fea88103120 2024-04-25T16:25:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.150+0000 7fea84ff9700 1 -- 172.21.15.173:0/3523987189 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea78004210 con 0x7fea88103120 2024-04-25T16:25:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.150+0000 7fea84ff9700 1 -- 172.21.15.173:0/3523987189 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea78022e50 con 0x7fea88103120 2024-04-25T16:25:59.153 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.150+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea881a28f0 con 0x7fea88103120 2024-04-25T16:25:59.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.151+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea88051190 con 0x7fea88103120 2024-04-25T16:25:59.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.152+0000 7fea84ff9700 1 -- 172.21.15.173:0/3523987189 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fea7802a070 con 0x7fea88103120 2024-04-25T16:25:59.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.152+0000 7fea84ff9700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea700afd20 0x7fea700b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:25:59.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.153+0000 7fea84ff9700 1 -- 172.21.15.173:0/3523987189 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fea780defd0 con 0x7fea88103120 2024-04-25T16:25:59.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.153+0000 7fea86ffd700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea700afd20 0x7fea700b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:25:59.154 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.153+0000 7fea86ffd700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea700afd20 0x7fea700b2230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea7c005fd0 tx=0x7fea7c005de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:25:59.156 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.156+0000 7fea84ff9700 1 -- 172.21.15.173:0/3523987189 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea78031020 con 0x7fea88103120 2024-04-25T16:25:59.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:25:59.362+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fea881008c0 con 0x7fea700afd20 2024-04-25T16:25:59.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:59 smithi098 ceph-mon[29162]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T16:25:59.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:59.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:59.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:25:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:59.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:59 smithi203 ceph-mon[33115]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T16:25:59.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:59.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:59.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:25:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:25:59.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:59 smithi173 ceph-mon[32556]: osdmap e28: 4 total, 4 up, 4 in 2024-04-25T16:25:59.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:25:59.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:25:59.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:25:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:00.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:00 smithi203 ceph-mon[33115]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:00.690 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:00 smithi203 ceph-mon[33115]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T16:26:00.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:00 smithi098 ceph-mon[29162]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:00.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:00 smithi098 ceph-mon[29162]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T16:26:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:00 smithi173 ceph-mon[32556]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:00 smithi173 ceph-mon[32556]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 120 MiB used, 357 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-25T16:26:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:02 smithi203 ceph-mon[33115]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:26:03.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:02 smithi203 ceph-mon[33115]: from='client.? 172.21.15.173:0/1204048266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]: dispatch 2024-04-25T16:26:03.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:02 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]: dispatch 2024-04-25T16:26:03.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:02 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]': finished 2024-04-25T16:26:03.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:02 smithi203 ceph-mon[33115]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T16:26:03.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:03.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:02 smithi203 ceph-mon[33115]: from='client.? 172.21.15.173:0/937497560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:02 smithi098 ceph-mon[29162]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:26:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:02 smithi098 ceph-mon[29162]: from='client.? 172.21.15.173:0/1204048266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]: dispatch 2024-04-25T16:26:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:02 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]: dispatch 2024-04-25T16:26:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:02 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]': finished 2024-04-25T16:26:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:02 smithi098 ceph-mon[29162]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T16:26:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:02 smithi098 ceph-mon[29162]: from='client.? 172.21.15.173:0/937497560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:03.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:02 smithi173 ceph-mon[32556]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:26:03.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:02 smithi173 ceph-mon[32556]: from='client.? 172.21.15.173:0/1204048266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]: dispatch 2024-04-25T16:26:03.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:02 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]: dispatch 2024-04-25T16:26:03.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:02 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7375471d-1a21-428b-855f-8d9cd5f5e5fb"}]': finished 2024-04-25T16:26:03.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:02 smithi173 ceph-mon[32556]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T16:26:03.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:03.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:02 smithi173 ceph-mon[32556]: from='client.? 172.21.15.173:0/937497560' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:04 smithi203 ceph-mon[33115]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T16:26:05.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:04 smithi098 ceph-mon[29162]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T16:26:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:04 smithi173 ceph-mon[32556]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-25T16:26:07.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:06 smithi203 ceph-mon[33115]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:26:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:06 smithi098 ceph-mon[29162]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:26:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:06 smithi173 ceph-mon[32556]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:26:09.183 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:08 smithi173 ceph-mon[32556]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T16:26:09.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:08 smithi203 ceph-mon[33115]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T16:26:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:08 smithi098 ceph-mon[29162]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T16:26:10.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T16:26:10.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T16:26:10.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:10.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T16:26:10.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:11.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:10 smithi203 ceph-mon[33115]: Deploying daemon osd.4 on smithi173 2024-04-25T16:26:11.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:10 smithi203 ceph-mon[33115]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:11.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:10 smithi098 ceph-mon[29162]: Deploying daemon osd.4 on smithi173 2024-04-25T16:26:11.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:10 smithi098 ceph-mon[29162]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:11.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:10 smithi173 ceph-mon[32556]: Deploying daemon osd.4 on smithi173 2024-04-25T16:26:11.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:10 smithi173 ceph-mon[32556]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:13.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:12 smithi098 ceph-mon[29162]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:13.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:12 smithi173 ceph-mon[32556]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:13.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:12 smithi203 ceph-mon[33115]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:15.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:14 smithi098 ceph-mon[29162]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:15.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:14 smithi173 ceph-mon[32556]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:15.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:14 smithi203 ceph-mon[33115]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:16.407 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:16.408 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:16.408 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:16.408 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:16 smithi173 ceph-mon[32556]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:16.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:16.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:16.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:16.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:16 smithi203 ceph-mon[33115]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:16.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:16.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:16.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:16.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:16 smithi098 ceph-mon[29162]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:18.882 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.880+0000 7fea84ff9700 1 -- 172.21.15.173:0/3523987189 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fea881008c0 con 0x7fea700afd20 2024-04-25T16:26:18.882 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea700afd20 msgr2=0x7fea700b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea700afd20 0x7fea700b2230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fea7c005fd0 tx=0x7fea7c005de0 comp rx=0 tx=0).stop 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 msgr2=0x7fea8819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 0x7fea8819a740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fea780098e0 tx=0x7fea78009f90 comp rx=0 tx=0).stop 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 shutdown_connections 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea700afd20 0x7fea700b2230 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fea88104bf0 0x7fea881a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea88103120 0x7fea8819a740 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:18.884 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 --2- 172.21.15.173:0/3523987189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea88104250 0x7fea8819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:18.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 >> 172.21.15.173:0/3523987189 conn(0x7fea880fe760 msgr2=0x7fea881001b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:18.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 shutdown_connections 2024-04-25T16:26:18.885 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:26:18.883+0000 7fea8e131700 1 -- 172.21.15.173:0/3523987189 wait complete. 2024-04-25T16:26:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:18 smithi203 ceph-mon[33115]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.213 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.213 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.213 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:19.213 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:19.213 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:18 smithi173 ceph-mon[32556]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:19.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:18 smithi098 ceph-mon[29162]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:19.560 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T16:26:19.563 INFO:tasks.cephadm:Deploying osd.5 on smithi203 with /dev/vg_nvme/lv_3... 2024-04-25T16:26:19.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T16:26:19.808 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:26:20.154 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:19 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:20.154 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:19 smithi203 ceph-mon[33115]: from='osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:26:20.154 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:19 smithi203 ceph-mon[33115]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:26:20.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:20.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:19 smithi098 ceph-mon[29162]: from='osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:26:20.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:19 smithi098 ceph-mon[29162]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:26:20.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:19 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:20.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:19 smithi173 ceph-mon[32556]: from='osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:26:20.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:19 smithi173 ceph-mon[32556]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T16:26:21.170 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:20 smithi173 ceph-mon[32556]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T16:26:21.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:20 smithi173 ceph-mon[32556]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:21.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:20 smithi173 ceph-mon[32556]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T16:26:21.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:20 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:21.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:20 smithi173 ceph-mon[32556]: from='osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:26:21.171 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:20 smithi173 ceph-mon[32556]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:26:21.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:20 smithi203 ceph-mon[33115]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T16:26:21.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:20 smithi203 ceph-mon[33115]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:21.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:20 smithi203 ceph-mon[33115]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T16:26:21.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:20 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:21.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:20 smithi203 ceph-mon[33115]: from='osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:26:21.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:20 smithi203 ceph-mon[33115]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:26:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:20 smithi098 ceph-mon[29162]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T16:26:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:20 smithi098 ceph-mon[29162]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:20 smithi098 ceph-mon[29162]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T16:26:21.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:20 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:21.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:20 smithi098 ceph-mon[29162]: from='osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:26:21.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:20 smithi098 ceph-mon[29162]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:26:21.803 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T16:26:21.803 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T16:26:21.803 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-25T16:26:21.803 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-25T16:26:21.803 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140763 s, 745 MB/s 2024-04-25T16:26:21.803 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-25T16:26:21.888 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:26:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4[44106]: 2024-04-25T16:26:21.632+0000 7f5037577700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T16:26:21.889 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:26:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4[44106]: 2024-04-25T16:26:21.638+0000 7f502f324700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:26:22.131 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:21 smithi203 ceph-mon[33115]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:26:22.131 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:21 smithi203 ceph-mon[33115]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T16:26:22.131 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:21 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:22.131 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:21 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:22.131 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:21 smithi203 ceph-mon[33115]: from='osd.4 ' entity='osd.4' 2024-04-25T16:26:22.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:21 smithi098 ceph-mon[29162]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:26:22.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:21 smithi098 ceph-mon[29162]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T16:26:22.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:22.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:22.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:21 smithi098 ceph-mon[29162]: from='osd.4 ' entity='osd.4' 2024-04-25T16:26:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:21 smithi173 ceph-mon[32556]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:26:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:21 smithi173 ceph-mon[32556]: osdmap e31: 5 total, 4 up, 5 in 2024-04-25T16:26:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:21 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:21 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:22.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:21 smithi173 ceph-mon[32556]: from='osd.4 ' entity='osd.4' 2024-04-25T16:26:22.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_3 2024-04-25T16:26:22.698 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:26:23.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:26:23.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] boot 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:23.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] boot 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:23.237 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:23.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:23.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:26:23.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:26:23.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: osd.4 [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] boot 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:24.138 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:23 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi173 2024-04-25T16:26:24.138 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:23 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-25T16:26:24.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:23 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi173 2024-04-25T16:26:24.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:23 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-25T16:26:24.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:23 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi173 2024-04-25T16:26:24.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:23 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi173 to 8544M 2024-04-25T16:26:24.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 -- 172.21.15.203:0/1843033921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374105880 msgr2=0x7f4374105c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:24.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 --2- 172.21.15.203:0/1843033921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374105880 0x7f4374105c90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4370009990 tx=0x7f4370009ca0 comp rx=0 tx=0).stop 2024-04-25T16:26:24.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 -- 172.21.15.203:0/1843033921 shutdown_connections 2024-04-25T16:26:24.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 --2- 172.21.15.203:0/1843033921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374105880 0x7f4374105c90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:24.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 --2- 172.21.15.203:0/1843033921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43741048b0 0x7f437410c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:24.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 --2- 172.21.15.203:0/1843033921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4374103e30 0x7f4374104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:24.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 -- 172.21.15.203:0/1843033921 >> 172.21.15.203:0/1843033921 conn(0x7f43740ff8a0 msgr2=0x7f4374101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:24.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 -- 172.21.15.203:0/1843033921 shutdown_connections 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.386+0000 7f437c227700 1 -- 172.21.15.203:0/1843033921 wait complete. 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437c227700 1 Processor -- start 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437c227700 1 -- start start 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437c227700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374103e30 0x7f4374078440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437c227700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43741048b0 0x7f4374076af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437c227700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4374105880 0x7f4374077030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437c227700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f437410fc50 con 0x7f4374105880 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437c227700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f437410fad0 con 0x7f43741048b0 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437c227700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f437410fdd0 con 0x7f4374103e30 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f4379fc3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374103e30 0x7f4374078440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f4379fc3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374103e30 0x7f4374078440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:45784/0 (socket says 172.21.15.203:45784) 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f4379fc3700 1 -- 172.21.15.203:0/2919505275 learned_addr learned my addr 172.21.15.203:0/2919505275 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:26:24.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f43797c2700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43741048b0 0x7f4374076af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.387+0000 7f437a7c4700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4374105880 0x7f4374077030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f4379fc3700 1 -- 172.21.15.203:0/2919505275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43741048b0 msgr2=0x7f4374076af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f4379fc3700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43741048b0 0x7f4374076af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f4379fc3700 1 -- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4374105880 msgr2=0x7f4374077030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f4379fc3700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4374105880 0x7f4374077030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f4379fc3700 1 -- 172.21.15.203:0/2919505275 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4374077790 con 0x7f4374103e30 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f4379fc3700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374103e30 0x7f4374078440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43700098e0 tx=0x7f437000b9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f436affd700 1 -- 172.21.15.203:0/2919505275 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4370026070 con 0x7f4374103e30 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43741a4920 con 0x7f4374103e30 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f436affd700 1 -- 172.21.15.203:0/2919505275 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4370003d20 con 0x7f4374103e30 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f436affd700 1 -- 172.21.15.203:0/2919505275 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437000fbf0 con 0x7f4374103e30 2024-04-25T16:26:24.391 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.388+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43741a4d70 con 0x7f4374103e30 2024-04-25T16:26:24.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.389+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4374051190 con 0x7f4374103e30 2024-04-25T16:26:24.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.390+0000 7f436affd700 1 -- 172.21.15.203:0/2919505275 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4370029070 con 0x7f4374103e30 2024-04-25T16:26:24.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.391+0000 7f436affd700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43600afc00 0x7f43600b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:24.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.391+0000 7f436affd700 1 -- 172.21.15.203:0/2919505275 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f43700de900 con 0x7f4374103e30 2024-04-25T16:26:24.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.391+0000 7f43797c2700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43600afc00 0x7f43600b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:24.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.392+0000 7f43797c2700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43600afc00 0x7f43600b2110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4374077df0 tx=0x7f4364009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:26:24.396 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.396+0000 7f436affd700 1 -- 172.21.15.203:0/2919505275 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4370013070 con 0x7f4374103e30 2024-04-25T16:26:24.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:24.600+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f437402e790 con 0x7f43600afc00 2024-04-25T16:26:25.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:24 smithi203 ceph-mon[33115]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:25.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:24 smithi203 ceph-mon[33115]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T16:26:25.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:24 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:25.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:24 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:25.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:24 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:25.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:24 smithi098 ceph-mon[29162]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:25.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:24 smithi098 ceph-mon[29162]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T16:26:25.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:25.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:25.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:24 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:25.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:24 smithi173 ceph-mon[32556]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:25.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:24 smithi173 ceph-mon[32556]: osdmap e33: 5 total, 5 up, 5 in 2024-04-25T16:26:25.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:24 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:25.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:24 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:25.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:24 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:26.181 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:25 smithi203 ceph-mon[33115]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:26.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:25 smithi098 ceph-mon[29162]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:26.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:25 smithi173 ceph-mon[32556]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:27.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:26 smithi203 ceph-mon[33115]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:27.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:26 smithi098 ceph-mon[29162]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:27.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:26 smithi173 ceph-mon[32556]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:28.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:27 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/2769113841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]: dispatch 2024-04-25T16:26:28.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:27 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]: dispatch 2024-04-25T16:26:28.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:27 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]': finished 2024-04-25T16:26:28.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:27 smithi203 ceph-mon[33115]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T16:26:28.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:27 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:28.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:27 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/2769113841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]: dispatch 2024-04-25T16:26:28.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:27 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]: dispatch 2024-04-25T16:26:28.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:27 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]': finished 2024-04-25T16:26:28.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:27 smithi098 ceph-mon[29162]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T16:26:28.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:27 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:28.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:27 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/2769113841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]: dispatch 2024-04-25T16:26:28.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:27 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]: dispatch 2024-04-25T16:26:28.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:27 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f"}]': finished 2024-04-25T16:26:28.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:27 smithi173 ceph-mon[32556]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T16:26:28.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:27 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:29.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:28 smithi203 ceph-mon[33115]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:29.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:28 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/1953690034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:29.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:28 smithi098 ceph-mon[29162]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:29.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:28 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/1953690034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:29.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:28 smithi173 ceph-mon[32556]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:29.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:28 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/1953690034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:31.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:30 smithi203 ceph-mon[33115]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:31.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:30 smithi098 ceph-mon[29162]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:31.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:30 smithi173 ceph-mon[32556]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:33.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:32 smithi173 ceph-mon[32556]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:33.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:32 smithi203 ceph-mon[33115]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:33.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:32 smithi098 ceph-mon[29162]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:35.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:34 smithi203 ceph-mon[33115]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:35.204 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T16:26:35.204 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:34 smithi098 ceph-mon[29162]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T16:26:35.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:35.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:34 smithi173 ceph-mon[32556]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:35.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T16:26:35.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:36.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:35 smithi203 ceph-mon[33115]: Deploying daemon osd.5 on smithi203 2024-04-25T16:26:36.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:35 smithi098 ceph-mon[29162]: Deploying daemon osd.5 on smithi203 2024-04-25T16:26:36.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:35 smithi173 ceph-mon[32556]: Deploying daemon osd.5 on smithi203 2024-04-25T16:26:37.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:36 smithi203 ceph-mon[33115]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:37.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:36 smithi098 ceph-mon[29162]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:37.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:36 smithi173 ceph-mon[32556]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:38 smithi203 ceph-mon[33115]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:39.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:38 smithi098 ceph-mon[29162]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:39.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:38 smithi173 ceph-mon[32556]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:41.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:40 smithi203 ceph-mon[33115]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:41.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:41.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:41.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:41.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:40 smithi098 ceph-mon[29162]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:41.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:41.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:41.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:40 smithi173 ceph-mon[32556]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:41.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:43.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:42 smithi203 ceph-mon[33115]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:42 smithi098 ceph-mon[29162]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:43.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:42 smithi173 ceph-mon[32556]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:44.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.065+0000 7f436affd700 1 -- 172.21.15.203:0/2919505275 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f437402e790 con 0x7f43600afc00 2024-04-25T16:26:44.066 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 5 on host 'smithi203' 2024-04-25T16:26:44.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43600afc00 msgr2=0x7f43600b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:44.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43600afc00 0x7f43600b2110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4374077df0 tx=0x7f4364009260 comp rx=0 tx=0).stop 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374103e30 msgr2=0x7f4374078440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374103e30 0x7f4374078440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43700098e0 tx=0x7f437000b9e0 comp rx=0 tx=0).stop 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 shutdown_connections 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43600afc00 0x7f43600b2110 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4374103e30 0x7f4374078440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4374105880 0x7f4374077030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 --2- 172.21.15.203:0/2919505275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43741048b0 0x7f4374076af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 >> 172.21.15.203:0/2919505275 conn(0x7f43740ff8a0 msgr2=0x7f43741085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 shutdown_connections 2024-04-25T16:26:44.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:26:44.069+0000 7f437c227700 1 -- 172.21.15.203:0/2919505275 wait complete. 2024-04-25T16:26:44.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:44.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:44.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:44 smithi173 ceph-mon[32556]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:44.407 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.408 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:44 smithi203 ceph-mon[33115]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:44.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:44.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:44.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:44.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:44 smithi098 ceph-mon[29162]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:44.702 DEBUG:teuthology.orchestra.run.smithi203:osd.5> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T16:26:44.704 INFO:tasks.cephadm:Deploying osd.6 on smithi098 with /dev/vg_nvme/lv_2... 2024-04-25T16:26:44.704 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T16:26:44.950 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:26:45.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:45 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:45.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:45 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:45.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:45 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:45.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:45 smithi173 ceph-mon[32556]: from='osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:26:45.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:45 smithi173 ceph-mon[32556]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:26:45.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:45.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:45.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:45 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:45.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:45 smithi098 ceph-mon[29162]: from='osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:26:45.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:45 smithi098 ceph-mon[29162]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:26:45.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:45 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:45.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:45 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:45.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:45 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:26:45.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:45 smithi203 ceph-mon[33115]: from='osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:26:45.440 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:45 smithi203 ceph-mon[33115]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T16:26:46.379 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:46 smithi098 ceph-mon[29162]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T16:26:46.379 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:46 smithi098 ceph-mon[29162]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T16:26:46.379 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:46 smithi098 ceph-mon[29162]: from='osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:26:46.379 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:46.379 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:46 smithi098 ceph-mon[29162]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:26:46.379 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:46 smithi098 ceph-mon[29162]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:46.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:46 smithi203 ceph-mon[33115]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T16:26:46.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:46 smithi203 ceph-mon[33115]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T16:26:46.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:46 smithi203 ceph-mon[33115]: from='osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:26:46.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:46.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:46 smithi203 ceph-mon[33115]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:26:46.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:46 smithi203 ceph-mon[33115]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:46.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:46 smithi173 ceph-mon[32556]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T16:26:46.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:46 smithi173 ceph-mon[32556]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T16:26:46.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:46 smithi173 ceph-mon[32556]: from='osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:26:46.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:46.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:46 smithi173 ceph-mon[32556]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:26:46.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:46 smithi173 ceph-mon[32556]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:46.925 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T16:26:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T16:26:46.926 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-25T16:26:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-25T16:26:46.926 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122858 s, 853 MB/s 2024-04-25T16:26:46.926 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-25T16:26:47.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:47 smithi203 ceph-mon[33115]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:26:47.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:47 smithi203 ceph-mon[33115]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T16:26:47.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:47 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:47.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:47 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:47.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:47 smithi203 ceph-mon[33115]: from='osd.5 ' entity='osd.5' 2024-04-25T16:26:47.190 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:26:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:26:46.793+0000 7fa44329e700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T16:26:47.190 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:26:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:26:46.801+0000 7fa43b04b700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:26:47.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:47 smithi098 ceph-mon[29162]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:26:47.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:47 smithi098 ceph-mon[29162]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T16:26:47.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:47 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:47.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:47 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:47.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:47 smithi098 ceph-mon[29162]: from='osd.5 ' entity='osd.5' 2024-04-25T16:26:47.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:47 smithi173 ceph-mon[32556]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:26:47.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:47 smithi173 ceph-mon[32556]: osdmap e36: 6 total, 5 up, 6 in 2024-04-25T16:26:47.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:47 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:47.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:47 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:47.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:47 smithi173 ceph-mon[32556]: from='osd.5 ' entity='osd.5' 2024-04-25T16:26:47.528 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-04-25T16:26:47.768 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:26:48.062 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] boot 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:48.063 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:26:48.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:26:48.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:48.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] boot 2024-04-25T16:26:48.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T16:26:48.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:48.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:48.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:48.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:48.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:48.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:48.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: osd.5 [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] boot 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:26:48.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:48.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:48.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:26:48.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:26:49.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:49 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi203 2024-04-25T16:26:49.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:49 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-25T16:26:49.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:49 smithi098 ceph-mon[29162]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T16:26:49.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:49 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi203 2024-04-25T16:26:49.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:49 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-25T16:26:49.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:49 smithi203 ceph-mon[33115]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T16:26:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 -- 172.21.15.98:0/177290390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281028b0 msgr2=0x7fbd28102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/177290390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281028b0 0x7fbd28102cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbd18009990 tx=0x7fbd18009ca0 comp rx=0 tx=0).stop 2024-04-25T16:26:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 -- 172.21.15.98:0/177290390 shutdown_connections 2024-04-25T16:26:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/177290390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd28104380 0x7fbd2810ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/177290390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281028b0 0x7fbd28102cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:49.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/177290390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd281039e0 0x7fbd28103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 -- 172.21.15.98:0/177290390 >> 172.21.15.98:0/177290390 conn(0x7fbd280fdef0 msgr2=0x7fbd281003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 -- 172.21.15.98:0/177290390 shutdown_connections 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.443+0000 7fbd2cffc700 1 -- 172.21.15.98:0/177290390 wait complete. 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2cffc700 1 Processor -- start 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2cffc700 1 -- start start 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2cffc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd281028b0 0x7fbd280ffef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2cffc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281039e0 0x7fbd2819f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2cffc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd28104380 0x7fbd281a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2cffc700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810e1a0 con 0x7fbd281039e0 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2cffc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810e020 con 0x7fbd281028b0 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2cffc700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2810e320 con 0x7fbd28104380 2024-04-25T16:26:49.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd25d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281039e0 0x7fbd2819f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd25d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281039e0 0x7fbd2819f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43102/0 (socket says 172.21.15.98:43102) 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd25d9b700 1 -- 172.21.15.98:0/1366207389 learned_addr learned my addr 172.21.15.98:0/1366207389 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd26d9d700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd28104380 0x7fbd281a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd2659c700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd281028b0 0x7fbd280ffef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd25d9b700 1 -- 172.21.15.98:0/1366207389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd28104380 msgr2=0x7fbd281a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd25d9b700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd28104380 0x7fbd281a1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd25d9b700 1 -- 172.21.15.98:0/1366207389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd281028b0 msgr2=0x7fbd280ffef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.444+0000 7fbd25d9b700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd281028b0 0x7fbd280ffef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.445+0000 7fbd25d9b700 1 -- 172.21.15.98:0/1366207389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd281a2290 con 0x7fbd281039e0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.445+0000 7fbd25d9b700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281039e0 0x7fbd2819f4f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00c920 tx=0x7fbd1c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.445+0000 7fbd177fe700 1 -- 172.21.15.98:0/1366207389 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c007b00 con 0x7fbd281039e0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.445+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd281a2580 con 0x7fbd281039e0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.445+0000 7fbd177fe700 1 -- 172.21.15.98:0/1366207389 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbd1c004cc0 con 0x7fbd281039e0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.445+0000 7fbd177fe700 1 -- 172.21.15.98:0/1366207389 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0053f0 con 0x7fbd281039e0 2024-04-25T16:26:49.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.445+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd281a2b10 con 0x7fbd281039e0 2024-04-25T16:26:49.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.446+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd28051190 con 0x7fbd281039e0 2024-04-25T16:26:49.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.447+0000 7fbd177fe700 1 -- 172.21.15.98:0/1366207389 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbd1c004750 con 0x7fbd281039e0 2024-04-25T16:26:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.448+0000 7fbd177fe700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbd100afb80 0x7fbd100b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:26:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.448+0000 7fbd2659c700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbd100afb80 0x7fbd100b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:26:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.448+0000 7fbd177fe700 1 -- 172.21.15.98:0/1366207389 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4533+0+0 (secure 0 0 0) 0x7fbd1c0ddcc0 con 0x7fbd281039e0 2024-04-25T16:26:49.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.448+0000 7fbd2659c700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbd100afb80 0x7fbd100b2090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbd18003820 tx=0x7fbd1801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:26:49.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.452+0000 7fbd177fe700 1 -- 172.21.15.98:0/1366207389 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd1c014030 con 0x7fbd281039e0 2024-04-25T16:26:49.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:49 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi203 2024-04-25T16:26:49.517 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:49 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi203 to 8544M 2024-04-25T16:26:49.518 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:49 smithi173 ceph-mon[32556]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T16:26:49.660 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:26:49.659+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fbd28063ad0 con 0x7fbd100afb80 2024-04-25T16:26:50.313 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:50 smithi098 ceph-mon[29162]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T16:26:50.313 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:50 smithi098 ceph-mon[29162]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:50.313 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:50.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:50.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:50.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:50 smithi098 ceph-mon[29162]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:50.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:50 smithi203 ceph-mon[33115]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T16:26:50.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:50 smithi203 ceph-mon[33115]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:50.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:50.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:50.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:50.341 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:50 smithi203 ceph-mon[33115]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:50.417 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:50 smithi173 ceph-mon[32556]: osdmap e39: 6 total, 6 up, 6 in 2024-04-25T16:26:50.417 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:50 smithi173 ceph-mon[32556]: from='client.14370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:26:50.417 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:26:50.417 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:26:50.417 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:26:50.417 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:50 smithi173 ceph-mon[32556]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:53.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:52 smithi098 ceph-mon[29162]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:53.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:52 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3658051205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89aa247d-c76d-4fa9-a368-bc45954b8b3f"}]: dispatch 2024-04-25T16:26:53.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:52 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3658051205' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89aa247d-c76d-4fa9-a368-bc45954b8b3f"}]': finished 2024-04-25T16:26:53.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:52 smithi098 ceph-mon[29162]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T16:26:53.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:26:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:52 smithi203 ceph-mon[33115]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:52 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3658051205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89aa247d-c76d-4fa9-a368-bc45954b8b3f"}]: dispatch 2024-04-25T16:26:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:52 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3658051205' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89aa247d-c76d-4fa9-a368-bc45954b8b3f"}]': finished 2024-04-25T16:26:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:52 smithi203 ceph-mon[33115]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T16:26:53.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:26:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:52 smithi173 ceph-mon[32556]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:52 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3658051205' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "89aa247d-c76d-4fa9-a368-bc45954b8b3f"}]: dispatch 2024-04-25T16:26:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:52 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3658051205' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "89aa247d-c76d-4fa9-a368-bc45954b8b3f"}]': finished 2024-04-25T16:26:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:52 smithi173 ceph-mon[32556]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T16:26:53.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:26:54.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:53 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1613213657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:53 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1613213657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:53 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1613213657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:26:55.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:54 smithi203 ceph-mon[33115]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:55.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:54 smithi098 ceph-mon[29162]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:55.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:54 smithi173 ceph-mon[32556]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:26:57.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:56 smithi098 ceph-mon[29162]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T16:26:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:56 smithi173 ceph-mon[32556]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T16:26:57.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:56 smithi203 ceph-mon[33115]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-25T16:26:59.160 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:58 smithi098 ceph-mon[29162]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-25T16:26:59.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:58 smithi173 ceph-mon[32556]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-25T16:26:59.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:58 smithi203 ceph-mon[33115]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-25T16:27:00.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T16:27:00.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:26:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:00.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T16:27:00.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:26:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:00.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T16:27:00.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:26:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:01.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:00 smithi098 ceph-mon[29162]: Deploying daemon osd.6 on smithi098 2024-04-25T16:27:01.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:00 smithi098 ceph-mon[29162]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:00 smithi173 ceph-mon[32556]: Deploying daemon osd.6 on smithi098 2024-04-25T16:27:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:00 smithi173 ceph-mon[32556]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:01.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:00 smithi203 ceph-mon[33115]: Deploying daemon osd.6 on smithi098 2024-04-25T16:27:01.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:00 smithi203 ceph-mon[33115]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:02 smithi098 ceph-mon[29162]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:02 smithi173 ceph-mon[32556]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:03.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:02 smithi203 ceph-mon[33115]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:27:05.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:04 smithi098 ceph-mon[29162]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-25T16:27:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:04 smithi173 ceph-mon[32556]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-25T16:27:05.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:04 smithi203 ceph-mon[33115]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-25T16:27:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:07.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:07.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:07.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:06 smithi098 ceph-mon[29162]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T16:27:07.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:07.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:07.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:07.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:06 smithi203 ceph-mon[33115]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T16:27:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:07.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:06 smithi173 ceph-mon[32556]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T16:27:09.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:08 smithi203 ceph-mon[33115]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:08 smithi098 ceph-mon[29162]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:08 smithi173 ceph-mon[32556]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:09.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.686+0000 7fbd177fe700 1 -- 172.21.15.98:0/1366207389 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbd28063ad0 con 0x7fbd100afb80 2024-04-25T16:27:09.688 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 6 on host 'smithi098' 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbd100afb80 msgr2=0x7fbd100b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbd100afb80 0x7fbd100b2090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbd18003820 tx=0x7fbd1801a040 comp rx=0 tx=0).stop 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281039e0 msgr2=0x7fbd2819f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281039e0 0x7fbd2819f4f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00c920 tx=0x7fbd1c00cc30 comp rx=0 tx=0).stop 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 shutdown_connections 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbd100afb80 0x7fbd100b2090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd28104380 0x7fbd281a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd281028b0 0x7fbd280ffef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:09.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 --2- 172.21.15.98:0/1366207389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbd281039e0 0x7fbd2819f4f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:09.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 >> 172.21.15.98:0/1366207389 conn(0x7fbd280fdef0 msgr2=0x7fbd28107640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:09.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 shutdown_connections 2024-04-25T16:27:09.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:27:09.689+0000 7fbd2cffc700 1 -- 172.21.15.98:0/1366207389 wait complete. 2024-04-25T16:27:10.346 DEBUG:teuthology.orchestra.run.smithi098:osd.6> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T16:27:10.348 INFO:tasks.cephadm:Deploying osd.7 on smithi173 with /dev/vg_nvme/lv_2... 2024-04-25T16:27:10.348 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T16:27:10.594 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:27:10.915 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.916 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.916 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:10 smithi173 ceph-mon[32556]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T16:27:10.916 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:10 smithi173 ceph-mon[32556]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:10.916 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.916 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:10 smithi203 ceph-mon[33115]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T16:27:10.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:10 smithi203 ceph-mon[33115]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:10.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.955 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:10 smithi098 ceph-mon[29162]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T16:27:10.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:10 smithi098 ceph-mon[29162]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:10.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:10.956 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:11.934 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:11 smithi173 ceph-mon[32556]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T16:27:11.934 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:11 smithi173 ceph-mon[32556]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T16:27:11.934 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:11 smithi173 ceph-mon[32556]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:27:11.934 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:11.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:11 smithi203 ceph-mon[33115]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T16:27:11.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:11 smithi203 ceph-mon[33115]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T16:27:11.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:11 smithi203 ceph-mon[33115]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:27:11.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:12.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:11 smithi098 ceph-mon[29162]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T16:27:12.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:11 smithi098 ceph-mon[29162]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T16:27:12.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:11 smithi098 ceph-mon[29162]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:27:12.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:12.000 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:27:11 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:27:11.652+0000 7f6f64a34700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T16:27:12.000 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:27:11 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:27:11.664+0000 7f6f607e9700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:27:12.687 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T16:27:12.687 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T16:27:12.688 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-25T16:27:12.688 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-25T16:27:12.688 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124308 s, 844 MB/s 2024-04-25T16:27:12.688 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 2024-04-25T16:27:12.688 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-25T16:27:12.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:12 smithi173 ceph-mon[32556]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:27:12.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:12 smithi173 ceph-mon[32556]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T16:27:12.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:12 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:12.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:12 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:12.766 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:12 smithi173 ceph-mon[32556]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:12.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:12 smithi203 ceph-mon[33115]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:27:12.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:12 smithi203 ceph-mon[33115]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T16:27:12.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:12 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:12.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:12 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:12.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:12 smithi203 ceph-mon[33115]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:13.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:12 smithi098 ceph-mon[29162]: from='osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:27:13.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:12 smithi098 ceph-mon[29162]: osdmap e42: 7 total, 6 up, 7 in 2024-04-25T16:27:13.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:12 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:13.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:12 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:13.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:12 smithi098 ceph-mon[29162]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-04-25T16:27:13.299 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-04-25T16:27:13.536 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:27:13.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:13 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:27:13.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:13 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:27:13.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:13 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:13.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:13 smithi203 ceph-mon[33115]: osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] boot 2024-04-25T16:27:13.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:13 smithi203 ceph-mon[33115]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T16:27:13.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:13 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:14.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:13 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:27:14.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:13 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:27:14.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:13 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:14.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:13 smithi098 ceph-mon[29162]: osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] boot 2024-04-25T16:27:14.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:13 smithi098 ceph-mon[29162]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T16:27:14.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:13 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:13 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:27:14.007 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:13 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:27:14.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:13 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:14.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:13 smithi173 ceph-mon[32556]: osd.6 [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] boot 2024-04-25T16:27:14.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:13 smithi173 ceph-mon[32556]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T16:27:14.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:13 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T16:27:14.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T16:27:14.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi098 2024-04-25T16:27:14.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:14.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:14.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:14.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi098 to 5696M 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:14.940 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T16:27:14.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi098 2024-04-25T16:27:14.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:14.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:14.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:14.996 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi098 to 5696M 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:14.997 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:15.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: osdmap e44: 7 total, 7 up, 7 in 2024-04-25T16:27:15.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi098 2024-04-25T16:27:15.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:15.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:15.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:15.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi098 to 5696M 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:15.004 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.202+0000 7f7356088700 1 -- 172.21.15.173:0/1310154947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 msgr2=0x7f7350102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.202+0000 7f7356088700 1 --2- 172.21.15.173:0/1310154947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 0x7f7350102610 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7344009990 tx=0x7f7344009ca0 comp rx=0 tx=0).stop 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.202+0000 7f7356088700 1 -- 172.21.15.173:0/1310154947 shutdown_connections 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.202+0000 7f7356088700 1 --2- 172.21.15.173:0/1310154947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f73501057b0 0x7f7350105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.202+0000 7f7356088700 1 --2- 172.21.15.173:0/1310154947 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7350102b50 0x7f735010c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.202+0000 7f7356088700 1 --2- 172.21.15.173:0/1310154947 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 0x7f7350102610 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.203+0000 7f7356088700 1 -- 172.21.15.173:0/1310154947 >> 172.21.15.173:0/1310154947 conn(0x7f73500fded0 msgr2=0x7f7350100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.203+0000 7f7356088700 1 -- 172.21.15.173:0/1310154947 shutdown_connections 2024-04-25T16:27:15.204 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.203+0000 7f7356088700 1 -- 172.21.15.173:0/1310154947 wait complete. 2024-04-25T16:27:15.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.203+0000 7f7356088700 1 Processor -- start 2024-04-25T16:27:15.205 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.203+0000 7f7356088700 1 -- start start 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f7356088700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 0x7f735019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f7356088700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7350102b50 0x7f735019ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f7356088700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73501057b0 0x7f73501a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f7356088700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f735010fba0 con 0x7f73501057b0 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f7356088700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f735010fa20 con 0x7f7350102200 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f7356088700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f735010fd20 con 0x7f7350102b50 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 0x7f735019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 0x7f735019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:45386/0 (socket says 172.21.15.173:45386) 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 -- 172.21.15.173:0/1510901694 learned_addr learned my addr 172.21.15.173:0/1510901694 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734effd700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7350102b50 0x7f735019ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 -- 172.21.15.173:0/1510901694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7350102b50 msgr2=0x7f735019ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7350102b50 0x7f735019ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:15.207 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 -- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73501057b0 msgr2=0x7f73501a1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734ffff700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73501057b0 0x7f73501a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73501057b0 0x7f73501a1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 -- 172.21.15.173:0/1510901694 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73501a21f0 con 0x7f7350102200 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734f7fe700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 0x7f735019a720 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f734000c970 tx=0x7f734000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734cff9700 1 -- 172.21.15.173:0/1510901694 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340007b00 con 0x7f7350102200 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734cff9700 1 -- 172.21.15.173:0/1510901694 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7340007ca0 con 0x7f7350102200 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73501a24e0 con 0x7f7350102200 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.204+0000 7f734cff9700 1 -- 172.21.15.173:0/1510901694 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340018660 con 0x7f7350102200 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.205+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73501a2aa0 con 0x7f7350102200 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.205+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7350051190 con 0x7f7350102200 2024-04-25T16:27:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.208+0000 7f734cff9700 1 -- 172.21.15.173:0/1510901694 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f734000f450 con 0x7f7350102200 2024-04-25T16:27:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.208+0000 7f734cff9700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afb80 0x7f73380b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.209+0000 7f734cff9700 1 -- 172.21.15.173:0/1510901694 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7f73400ddf00 con 0x7f7350102200 2024-04-25T16:27:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.209+0000 7f734effd700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afb80 0x7f73380b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.209+0000 7f734effd700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afb80 0x7f73380b2090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f734400b5c0 tx=0x7f734401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:15.213 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.212+0000 7f734cff9700 1 -- 172.21.15.173:0/1510901694 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7340014030 con 0x7f7350102200 2024-04-25T16:27:15.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:15.418+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7350063ad0 con 0x7f73380afb80 2024-04-25T16:27:16.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:15 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:16.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:15 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:16.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:15 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:16.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:15 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:16.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:15 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:16.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:15 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:16.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:15 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:16.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:15 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:16.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:15 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:17.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:16 smithi173 ceph-mon[32556]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:17.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:16 smithi173 ceph-mon[32556]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:17.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:16 smithi203 ceph-mon[33115]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:17.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:16 smithi203 ceph-mon[33115]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:17.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:16 smithi098 ceph-mon[29162]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:17.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:16 smithi098 ceph-mon[29162]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:18 smithi203 ceph-mon[33115]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:18 smithi203 ceph-mon[33115]: from='client.? 172.21.15.173:0/2114298687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]: dispatch 2024-04-25T16:27:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:18 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]: dispatch 2024-04-25T16:27:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:18 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]': finished 2024-04-25T16:27:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:18 smithi203 ceph-mon[33115]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T16:27:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:18 smithi203 ceph-mon[33115]: from='client.? 172.21.15.173:0/2275440746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:18 smithi098 ceph-mon[29162]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:18 smithi098 ceph-mon[29162]: from='client.? 172.21.15.173:0/2114298687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]: dispatch 2024-04-25T16:27:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:18 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]: dispatch 2024-04-25T16:27:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:18 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]': finished 2024-04-25T16:27:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:18 smithi098 ceph-mon[29162]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T16:27:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:18 smithi098 ceph-mon[29162]: from='client.? 172.21.15.173:0/2275440746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:18 smithi173 ceph-mon[32556]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 213 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:18 smithi173 ceph-mon[32556]: from='client.? 172.21.15.173:0/2114298687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]: dispatch 2024-04-25T16:27:19.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:18 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]: dispatch 2024-04-25T16:27:19.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:18 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4a90448-6dcb-4dd1-aec5-cc423090bdf0"}]': finished 2024-04-25T16:27:19.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:18 smithi173 ceph-mon[32556]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T16:27:19.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:19.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:18 smithi173 ceph-mon[32556]: from='client.? 172.21.15.173:0/2275440746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:21.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:20 smithi203 ceph-mon[33115]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:21.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:20 smithi098 ceph-mon[29162]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:21.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:20 smithi173 ceph-mon[32556]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:23.194 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:22 smithi173 ceph-mon[32556]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:23.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:22 smithi098 ceph-mon[29162]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:22 smithi203 ceph-mon[33115]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:25.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:24 smithi098 ceph-mon[29162]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:25.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:24 smithi173 ceph-mon[32556]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:25.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:24 smithi203 ceph-mon[33115]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:26.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:25 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T16:27:26.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:25 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:26.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:25 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T16:27:26.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:25 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:26.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:25 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T16:27:26.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:25 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:27.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:26 smithi098 ceph-mon[29162]: Deploying daemon osd.7 on smithi173 2024-04-25T16:27:27.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:26 smithi098 ceph-mon[29162]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:27.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:26 smithi173 ceph-mon[32556]: Deploying daemon osd.7 on smithi173 2024-04-25T16:27:27.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:26 smithi173 ceph-mon[32556]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:27.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:26 smithi203 ceph-mon[33115]: Deploying daemon osd.7 on smithi173 2024-04-25T16:27:27.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:26 smithi203 ceph-mon[33115]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:29.134 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:28 smithi173 ceph-mon[32556]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:29.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:28 smithi098 ceph-mon[29162]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:29.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:28 smithi203 ceph-mon[33115]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:31.152 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:30 smithi173 ceph-mon[32556]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:31.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:30 smithi098 ceph-mon[29162]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:31.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:30 smithi203 ceph-mon[33115]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:32.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:32.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:32.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:32.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:32 smithi203 ceph-mon[33115]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:32.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:32.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:32.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:32.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:32 smithi098 ceph-mon[29162]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:32.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:32.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:32.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:32.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:32 smithi173 ceph-mon[32556]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:34.931 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 7 on host 'smithi173' 2024-04-25T16:27:34.932 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.931+0000 7f734cff9700 1 -- 172.21.15.173:0/1510901694 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7350063ad0 con 0x7f73380afb80 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afb80 msgr2=0x7f73380b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afb80 0x7f73380b2090 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f734400b5c0 tx=0x7f734401a040 comp rx=0 tx=0).stop 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 msgr2=0x7f735019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 0x7f735019a720 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f734000c970 tx=0x7f734000cc80 comp rx=0 tx=0).stop 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 shutdown_connections 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afb80 0x7f73380b2090 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f73501057b0 0x7f73501a1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7350102200 0x7f735019a720 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 --2- 172.21.15.173:0/1510901694 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7350102b50 0x7f735019ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 >> 172.21.15.173:0/1510901694 conn(0x7f73500fded0 msgr2=0x7f7350108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:34.934 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 shutdown_connections 2024-04-25T16:27:34.935 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:27:34.934+0000 7f7356088700 1 -- 172.21.15.173:0/1510901694 wait complete. 2024-04-25T16:27:35.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:35.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:35.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:35.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:35.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:35.222 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:35.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:35.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:35.505 DEBUG:teuthology.orchestra.run.smithi173:osd.7> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T16:27:35.507 INFO:tasks.cephadm:Deploying osd.8 on smithi203 with /dev/vg_nvme/lv_2... 2024-04-25T16:27:35.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T16:27:35.751 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:27:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:35 smithi203 ceph-mon[33115]: from='osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:27:36.157 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:35 smithi203 ceph-mon[33115]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:27:36.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:35 smithi098 ceph-mon[29162]: from='osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:27:36.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:35 smithi098 ceph-mon[29162]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:27:36.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:35 smithi173 ceph-mon[32556]: from='osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:27:36.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:35 smithi173 ceph-mon[32556]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T16:27:37.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:36 smithi203 ceph-mon[33115]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:37.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:36 smithi203 ceph-mon[33115]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T16:27:37.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:36 smithi203 ceph-mon[33115]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T16:27:37.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:36 smithi203 ceph-mon[33115]: from='osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:27:37.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:36 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:37.119 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:36 smithi203 ceph-mon[33115]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:27:37.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:36 smithi098 ceph-mon[29162]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:37.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:36 smithi098 ceph-mon[29162]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T16:27:37.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:36 smithi098 ceph-mon[29162]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T16:27:37.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:36 smithi098 ceph-mon[29162]: from='osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:27:37.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:36 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:37.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:36 smithi098 ceph-mon[29162]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:27:37.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:36 smithi173 ceph-mon[32556]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:37.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:36 smithi173 ceph-mon[32556]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T16:27:37.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:36 smithi173 ceph-mon[32556]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T16:27:37.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:36 smithi173 ceph-mon[32556]: from='osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:27:37.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:36 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:37.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:36 smithi173 ceph-mon[32556]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:27:37.266 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:27:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:27:36.955+0000 7fd1289d0700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T16:27:37.266 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:27:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:27:36.965+0000 7fd124785700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:27:37.704 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T16:27:37.704 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T16:27:37.704 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-25T16:27:37.704 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-25T16:27:37.705 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124827 s, 840 MB/s 2024-04-25T16:27:37.705 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-25T16:27:38.231 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:37 smithi173 ceph-mon[32556]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:27:38.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:37 smithi173 ceph-mon[32556]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T16:27:38.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:38.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:38.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:37 smithi098 ceph-mon[29162]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:27:38.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:37 smithi098 ceph-mon[29162]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T16:27:38.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:38.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:38.301 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:37 smithi203 ceph-mon[33115]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:27:38.301 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:37 smithi203 ceph-mon[33115]: osdmap e47: 8 total, 7 up, 8 in 2024-04-25T16:27:38.301 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:38.301 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:38.332 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_2 2024-04-25T16:27:38.591 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] boot 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:39.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:39.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:38 smithi203 ceph-mon[33115]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T16:27:39.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:27:39.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:27:39.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:39.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] boot 2024-04-25T16:27:39.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T16:27:39.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:38 smithi098 ceph-mon[29162]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T16:27:39.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:27:39.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:27:39.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 214 MiB used, 626 GiB / 626 GiB avail 2024-04-25T16:27:39.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: osd.7 [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] boot 2024-04-25T16:27:39.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T16:27:39.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T16:27:39.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:39.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:38 smithi173 ceph-mon[32556]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T16:27:40.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.174+0000 7f629182d700 1 -- 172.21.15.203:0/3468552708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 msgr2=0x7f628c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:40.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.174+0000 7f629182d700 1 --2- 172.21.15.203:0/3468552708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 0x7f628c1045f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f627c009990 tx=0x7f627c009ca0 comp rx=0 tx=0).stop 2024-04-25T16:27:40.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.174+0000 7f629182d700 1 -- 172.21.15.203:0/3468552708 shutdown_connections 2024-04-25T16:27:40.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.174+0000 7f629182d700 1 --2- 172.21.15.203:0/3468552708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 0x7f628c1045f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:40.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.174+0000 7f629182d700 1 --2- 172.21.15.203:0/3468552708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628c105cb0 0x7f628c10c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:40.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.174+0000 7f629182d700 1 --2- 172.21.15.203:0/3468552708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c105310 0x7f628c105720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:40.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.174+0000 7f629182d700 1 -- 172.21.15.203:0/3468552708 >> 172.21.15.203:0/3468552708 conn(0x7f628c0ff840 msgr2=0x7f628c101cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:40.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.174+0000 7f629182d700 1 -- 172.21.15.203:0/3468552708 shutdown_connections 2024-04-25T16:27:40.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.175+0000 7f629182d700 1 -- 172.21.15.203:0/3468552708 wait complete. 2024-04-25T16:27:40.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.175+0000 7f629182d700 1 Processor -- start 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.175+0000 7f629182d700 1 -- start start 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629182d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 0x7f628c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629182d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628c105310 0x7f628c19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629182d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c105cb0 0x7f628c1a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629182d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f628c10fb00 con 0x7f628c105cb0 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629182d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f628c10f980 con 0x7f628c105310 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629182d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f628c10fc80 con 0x7f628c1041e0 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 0x7f628c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 0x7f628c19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:45020/0 (socket says 172.21.15.203:45020) 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 -- 172.21.15.203:0/455221523 learned_addr learned my addr 172.21.15.203:0/455221523 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628a7fc700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628c105310 0x7f628c19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:40.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628b7fe700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c105cb0 0x7f628c1a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 -- 172.21.15.203:0/455221523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628c105310 msgr2=0x7f628c19ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628c105310 0x7f628c19ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 -- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c105cb0 msgr2=0x7f628c1a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c105cb0 0x7f628c1a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 -- 172.21.15.203:0/455221523 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c1a2260 con 0x7f628c1041e0 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f628affd700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 0x7f628c19a700 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f627c006010 tx=0x7f627c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629082b700 1 -- 172.21.15.203:0/455221523 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c003c10 con 0x7f628c1041e0 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628c1a24f0 con 0x7f628c1041e0 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629082b700 1 -- 172.21.15.203:0/455221523 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f627c004830 con 0x7f628c1041e0 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629082b700 1 -- 172.21.15.203:0/455221523 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c005370 con 0x7f628c1041e0 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.176+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f628c1a2940 con 0x7f628c1041e0 2024-04-25T16:27:40.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.177+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f628c051190 con 0x7f628c1041e0 2024-04-25T16:27:40.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.179+0000 7f629082b700 1 -- 172.21.15.203:0/455221523 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f627c0049e0 con 0x7f628c1041e0 2024-04-25T16:27:40.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.179+0000 7f629082b700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f62740afb80 0x7f62740b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:27:40.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.180+0000 7f629082b700 1 -- 172.21.15.203:0/455221523 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7f627c0df530 con 0x7f628c1041e0 2024-04-25T16:27:40.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.180+0000 7f628a7fc700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f62740afb80 0x7f62740b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:27:40.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:39 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi173 2024-04-25T16:27:40.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:39 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-25T16:27:40.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:39 smithi203 ceph-mon[33115]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:40.182 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:39 smithi203 ceph-mon[33115]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T16:27:40.182 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.180+0000 7f628a7fc700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f62740afb80 0x7f62740b2090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6280005950 tx=0x7f62800058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:27:40.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.184+0000 7f629082b700 1 -- 172.21.15.203:0/455221523 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f627c013070 con 0x7f628c1041e0 2024-04-25T16:27:40.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:39 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi173 2024-04-25T16:27:40.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:39 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-25T16:27:40.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:39 smithi098 ceph-mon[29162]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:40.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:39 smithi098 ceph-mon[29162]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T16:27:40.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:39 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi173 2024-04-25T16:27:40.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:39 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi173 to 5696M 2024-04-25T16:27:40.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:39 smithi173 ceph-mon[32556]: pgmap v175: 1 pgs: 1 remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:40.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:39 smithi173 ceph-mon[32556]: osdmap e50: 8 total, 8 up, 8 in 2024-04-25T16:27:40.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:40.393+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f628c063ad0 con 0x7f62740afb80 2024-04-25T16:27:41.240 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:40 smithi203 ceph-mon[33115]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:41.240 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:41.240 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:41.240 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:41.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:40 smithi098 ceph-mon[29162]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:41.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:41.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:41.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:40 smithi173 ceph-mon[32556]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:27:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:27:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:27:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:42.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:41 smithi098 ceph-mon[29162]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:42.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:41 smithi173 ceph-mon[32556]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:42.320 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:41 smithi203 ceph-mon[33115]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:43.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:43 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/4100983236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]: dispatch 2024-04-25T16:27:43.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:43 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]: dispatch 2024-04-25T16:27:43.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:43 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]': finished 2024-04-25T16:27:43.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:43 smithi098 ceph-mon[29162]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T16:27:43.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:43 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:27:43.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:43 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/4100983236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]: dispatch 2024-04-25T16:27:43.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:43 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]: dispatch 2024-04-25T16:27:43.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:43 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]': finished 2024-04-25T16:27:43.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:43 smithi173 ceph-mon[32556]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T16:27:43.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:43 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:27:43.567 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:43 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/4100983236' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]: dispatch 2024-04-25T16:27:43.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:43 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]: dispatch 2024-04-25T16:27:43.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:43 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "921fce6a-8fac-4547-b46a-b177454e3b55"}]': finished 2024-04-25T16:27:43.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:43 smithi203 ceph-mon[33115]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T16:27:43.568 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:43 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:27:44.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:44 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/1492477694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:44.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:44 smithi173 ceph-mon[32556]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T16:27:44.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:44 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/1492477694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:44.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:44 smithi203 ceph-mon[33115]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T16:27:44.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:44 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/1492477694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:27:44.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:44 smithi098 ceph-mon[29162]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-04-25T16:27:47.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:46 smithi203 ceph-mon[33115]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s, 0 objects/s recovering 2024-04-25T16:27:47.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:46 smithi098 ceph-mon[29162]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s, 0 objects/s recovering 2024-04-25T16:27:47.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:46 smithi173 ceph-mon[32556]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 166 KiB/s, 0 objects/s recovering 2024-04-25T16:27:49.237 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:48 smithi203 ceph-mon[33115]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T16:27:49.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:48 smithi098 ceph-mon[29162]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T16:27:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:48 smithi173 ceph-mon[32556]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-25T16:27:51.231 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:50 smithi203 ceph-mon[33115]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T16:27:51.232 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T16:27:51.232 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:51.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:50 smithi098 ceph-mon[29162]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T16:27:51.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T16:27:51.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:50 smithi173 ceph-mon[32556]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T16:27:51.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:27:52.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:51 smithi203 ceph-mon[33115]: Deploying daemon osd.8 on smithi203 2024-04-25T16:27:52.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:51 smithi098 ceph-mon[29162]: Deploying daemon osd.8 on smithi203 2024-04-25T16:27:52.254 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:51 smithi173 ceph-mon[32556]: Deploying daemon osd.8 on smithi203 2024-04-25T16:27:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:52 smithi203 ceph-mon[33115]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T16:27:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:52 smithi098 ceph-mon[29162]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T16:27:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:52 smithi173 ceph-mon[32556]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-25T16:27:55.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:54 smithi098 ceph-mon[29162]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:55.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:54 smithi173 ceph-mon[32556]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:55.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:54 smithi203 ceph-mon[33115]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:57.094 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:56 smithi203 ceph-mon[33115]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:57.095 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:57.095 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:57.095 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:57.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:56 smithi098 ceph-mon[29162]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:57.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:57.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:57.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:56 smithi173 ceph-mon[32556]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:27:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:27:59.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:27:58 smithi203 ceph-mon[33115]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:59.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:27:58 smithi098 ceph-mon[29162]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:59.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:27:58 smithi173 ceph-mon[32556]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:27:59.909 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 8 on host 'smithi203' 2024-04-25T16:27:59.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.908+0000 7f629082b700 1 -- 172.21.15.203:0/455221523 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f628c063ad0 con 0x7f62740afb80 2024-04-25T16:27:59.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f62740afb80 msgr2=0x7f62740b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:59.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f62740afb80 0x7f62740b2090 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6280005950 tx=0x7f62800058e0 comp rx=0 tx=0).stop 2024-04-25T16:27:59.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 msgr2=0x7f628c19a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:27:59.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 0x7f628c19a700 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f627c006010 tx=0x7f627c0047b0 comp rx=0 tx=0).stop 2024-04-25T16:27:59.911 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 shutdown_connections 2024-04-25T16:27:59.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f62740afb80 0x7f62740b2090 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:59.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f628c1041e0 0x7f628c19a700 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:59.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f628c105cb0 0x7f628c1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:59.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 --2- 172.21.15.203:0/455221523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f628c105310 0x7f628c19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:27:59.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 >> 172.21.15.203:0/455221523 conn(0x7f628c0ff840 msgr2=0x7f628c108540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:27:59.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 shutdown_connections 2024-04-25T16:27:59.912 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:27:59.911+0000 7f629182d700 1 -- 172.21.15.203:0/455221523 wait complete. 2024-04-25T16:28:00.549 DEBUG:teuthology.orchestra.run.smithi203:osd.8> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T16:28:00.552 INFO:tasks.cephadm:Deploying osd.9 on smithi098 with /dev/vg_nvme/lv_1... 2024-04-25T16:28:00.552 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T16:28:00.799 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:28:01.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:00 smithi098 ceph-mon[29162]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:28:01.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:00 smithi098 ceph-mon[29162]: from='osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:28:01.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:00 smithi098 ceph-mon[29162]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:28:01.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.139 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:00 smithi203 ceph-mon[33115]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:28:01.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:00 smithi203 ceph-mon[33115]: from='osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:28:01.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:00 smithi203 ceph-mon[33115]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:28:01.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:00 smithi173 ceph-mon[32556]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:28:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:00 smithi173 ceph-mon[32556]: from='osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:28:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:00 smithi173 ceph-mon[32556]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T16:28:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:01.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:02.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:01 smithi173 ceph-mon[32556]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T16:28:02.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:01 smithi173 ceph-mon[32556]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T16:28:02.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:02.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:01 smithi173 ceph-mon[32556]: from='osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:28:02.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:01 smithi173 ceph-mon[32556]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:28:02.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:01 smithi098 ceph-mon[29162]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T16:28:02.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:01 smithi098 ceph-mon[29162]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T16:28:02.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:01 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:02.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:01 smithi098 ceph-mon[29162]: from='osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:28:02.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:01 smithi098 ceph-mon[29162]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:28:02.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:01 smithi203 ceph-mon[33115]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T16:28:02.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:01 smithi203 ceph-mon[33115]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T16:28:02.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:01 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:02.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:01 smithi203 ceph-mon[33115]: from='osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:28:02.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:01 smithi203 ceph-mon[33115]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:28:02.905 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:28:02.632+0000 7fea2fe3f700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T16:28:02.905 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:28:02.645+0000 7fea27bec700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:28:02.906 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T16:28:02.906 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T16:28:02.906 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-04-25T16:28:02.906 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-04-25T16:28:02.906 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135229 s, 775 MB/s 2024-04-25T16:28:02.906 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-04-25T16:28:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:28:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:28:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T16:28:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: from='osd.8 ' entity='osd.8' 2024-04-25T16:28:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] boot 2024-04-25T16:28:03.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T16:28:03.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:28:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:28:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T16:28:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: from='osd.8 ' entity='osd.8' 2024-04-25T16:28:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] boot 2024-04-25T16:28:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T16:28:03.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: osdmap e53: 9 total, 8 up, 9 in 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: from='osd.8 ' entity='osd.8' 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: osd.8 [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] boot 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T16:28:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T16:28:03.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-04-25T16:28:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:28:04.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:03 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:28:04.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:03 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:28:04.083 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:03 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:28:04.083 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:03 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:28:04.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:03 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:28:04.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:03 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:28:05.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:05.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T16:28:05.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:05.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:05.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: pgmap v192: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T16:28:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:05.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:05.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:05.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:05.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:05.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:05.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 -- 172.21.15.98:0/574041045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424103e30 msgr2=0x7fe424104240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/574041045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424103e30 0x7fe424104240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe418009990 tx=0x7fe418009ca0 comp rx=0 tx=0).stop 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 -- 172.21.15.98:0/574041045 shutdown_connections 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/574041045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe424105880 0x7fe424105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/574041045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe4241048b0 0x7fe42410c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/574041045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424103e30 0x7fe424104240 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 -- 172.21.15.98:0/574041045 >> 172.21.15.98:0/574041045 conn(0x7fe4240ff8a0 msgr2=0x7fe424101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 -- 172.21.15.98:0/574041045 shutdown_connections 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 -- 172.21.15.98:0/574041045 wait complete. 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 Processor -- start 2024-04-25T16:28:05.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.613+0000 7fe42a4e6700 1 -- start start 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe42a4e6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424103e30 0x7fe42419a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe42a4e6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4241048b0 0x7fe42419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe42a4e6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424105880 0x7fe4241a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe42a4e6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe42410fce0 con 0x7fe424105880 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe42a4e6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe42410fb60 con 0x7fe424103e30 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe42a4e6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe42410fe60 con 0x7fe4241048b0 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424105880 0x7fe4241a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424105880 0x7fe4241a1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:59800/0 (socket says 172.21.15.98:59800) 2024-04-25T16:28:05.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 -- 172.21.15.98:0/4208962016 learned_addr learned my addr 172.21.15.98:0/4208962016 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:28:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe4237fe700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4241048b0 0x7fe42419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe423fff700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424103e30 0x7fe42419a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 -- 172.21.15.98:0/4208962016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4241048b0 msgr2=0x7fe42419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4241048b0 0x7fe42419add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 -- 172.21.15.98:0/4208962016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424103e30 msgr2=0x7fe42419a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424103e30 0x7fe42419a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 -- 172.21.15.98:0/4208962016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4241a23f0 con 0x7fe424105880 2024-04-25T16:28:05.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe4237fe700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4241048b0 0x7fe42419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:28:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe428a83700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424105880 0x7fe4241a1d70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe41c007ee0 tx=0x7fe41c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe4217fa700 1 -- 172.21.15.98:0/4208962016 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c00dbd0 con 0x7fe424105880 2024-04-25T16:28:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe4217fa700 1 -- 172.21.15.98:0/4208962016 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe41c00dd70 con 0x7fe424105880 2024-04-25T16:28:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.614+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4241a26e0 con 0x7fe424105880 2024-04-25T16:28:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.615+0000 7fe4217fa700 1 -- 172.21.15.98:0/4208962016 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe41c0058d0 con 0x7fe424105880 2024-04-25T16:28:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.615+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241a2c50 con 0x7fe424105880 2024-04-25T16:28:05.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.616+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe424051190 con 0x7fe424105880 2024-04-25T16:28:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.617+0000 7fe4217fa700 1 -- 172.21.15.98:0/4208962016 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fe41c01a020 con 0x7fe424105880 2024-04-25T16:28:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.617+0000 7fe4217fa700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe40c0afb80 0x7fe40c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.618+0000 7fe423fff700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe40c0afb80 0x7fe40c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:05.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.618+0000 7fe4217fa700 1 -- 172.21.15.98:0/4208962016 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7fe41c0ddb20 con 0x7fe424105880 2024-04-25T16:28:05.620 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.618+0000 7fe423fff700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe40c0afb80 0x7fe40c0b2090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe414006fd0 tx=0x7fe414008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:05.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.622+0000 7fe4217fa700 1 -- 172.21.15.98:0/4208962016 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe41c010070 con 0x7fe424105880 2024-04-25T16:28:05.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:05.830+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe42402e8b0 con 0x7fe40c0afb80 2024-04-25T16:28:06.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:05 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi203 2024-04-25T16:28:06.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:05 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-25T16:28:06.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:05 smithi203 ceph-mon[33115]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T16:28:06.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:05 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:06.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:05 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:06.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:05 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:05 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi203 2024-04-25T16:28:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:05 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-25T16:28:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:05 smithi098 ceph-mon[29162]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T16:28:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:05 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:05 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:05 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:06.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:05 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi203 2024-04-25T16:28:06.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:05 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi203 to 5696M 2024-04-25T16:28:06.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:05 smithi173 ceph-mon[32556]: osdmap e56: 9 total, 9 up, 9 in 2024-04-25T16:28:06.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:05 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:06.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:05 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:06.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:05 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:07.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:06 smithi203 ceph-mon[33115]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:07.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:06 smithi203 ceph-mon[33115]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:07.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:06 smithi098 ceph-mon[29162]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:07.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:06 smithi098 ceph-mon[29162]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:06 smithi173 ceph-mon[32556]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:06 smithi173 ceph-mon[32556]: pgmap v195: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:09.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:08 smithi203 ceph-mon[33115]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:09.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:08 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/155664881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53d9c2d6-bded-41da-a30b-7b4d2c44b55a"}]: dispatch 2024-04-25T16:28:09.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:08 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/155664881' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d9c2d6-bded-41da-a30b-7b4d2c44b55a"}]': finished 2024-04-25T16:28:09.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:08 smithi203 ceph-mon[33115]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T16:28:09.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:08 smithi098 ceph-mon[29162]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:08 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/155664881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53d9c2d6-bded-41da-a30b-7b4d2c44b55a"}]: dispatch 2024-04-25T16:28:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:08 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/155664881' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d9c2d6-bded-41da-a30b-7b4d2c44b55a"}]': finished 2024-04-25T16:28:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:08 smithi098 ceph-mon[29162]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T16:28:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:08 smithi173 ceph-mon[32556]: pgmap v196: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:08 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/155664881' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "53d9c2d6-bded-41da-a30b-7b4d2c44b55a"}]: dispatch 2024-04-25T16:28:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:08 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/155664881' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "53d9c2d6-bded-41da-a30b-7b4d2c44b55a"}]': finished 2024-04-25T16:28:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:08 smithi173 ceph-mon[32556]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T16:28:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:10.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:09 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2298330314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:28:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:09 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2298330314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:28:10.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:09 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2298330314' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:28:11.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:10 smithi098 ceph-mon[29162]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:28:11.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:10 smithi173 ceph-mon[32556]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:28:11.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:10 smithi203 ceph-mon[33115]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T16:28:13.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:12 smithi098 ceph-mon[29162]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:28:13.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:12 smithi173 ceph-mon[32556]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:28:13.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:12 smithi203 ceph-mon[33115]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T16:28:15.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:14 smithi098 ceph-mon[29162]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T16:28:15.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:14 smithi173 ceph-mon[32556]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T16:28:15.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:14 smithi203 ceph-mon[33115]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-04-25T16:28:17.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:16 smithi098 ceph-mon[29162]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:28:17.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T16:28:17.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:16 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:17.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:16 smithi173 ceph-mon[32556]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:28:17.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T16:28:17.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:16 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:17.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:16 smithi203 ceph-mon[33115]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:28:17.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T16:28:17.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:16 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:18.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:17 smithi098 ceph-mon[29162]: Deploying daemon osd.9 on smithi098 2024-04-25T16:28:18.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:17 smithi173 ceph-mon[32556]: Deploying daemon osd.9 on smithi098 2024-04-25T16:28:18.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:17 smithi203 ceph-mon[33115]: Deploying daemon osd.9 on smithi098 2024-04-25T16:28:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:18 smithi098 ceph-mon[29162]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:28:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:18 smithi173 ceph-mon[32556]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:28:19.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:18 smithi203 ceph-mon[33115]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:28:21.234 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:20 smithi098 ceph-mon[29162]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:21.240 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:20 smithi203 ceph-mon[33115]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:21.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:20 smithi173 ceph-mon[32556]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:23.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:22 smithi098 ceph-mon[29162]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:23.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:23.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:23.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:22 smithi173 ceph-mon[32556]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:23.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:23.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:23.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:22 smithi203 ceph-mon[33115]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:25.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:24 smithi098 ceph-mon[29162]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:25.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:24 smithi173 ceph-mon[32556]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:25.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:24 smithi203 ceph-mon[33115]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:26.438 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 9 on host 'smithi098' 2024-04-25T16:28:26.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.437+0000 7fe4217fa700 1 -- 172.21.15.98:0/4208962016 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe42402e8b0 con 0x7fe40c0afb80 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe40c0afb80 msgr2=0x7fe40c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe40c0afb80 0x7fe40c0b2090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe414006fd0 tx=0x7fe414008040 comp rx=0 tx=0).stop 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424105880 msgr2=0x7fe4241a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424105880 0x7fe4241a1d70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe41c007ee0 tx=0x7fe41c00d3b0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 shutdown_connections 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe40c0afb80 0x7fe40c0b2090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe424105880 0x7fe4241a1d70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe424103e30 0x7fe42419a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 --2- 172.21.15.98:0/4208962016 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4241048b0 0x7fe42419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.439+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 >> 172.21.15.98:0/4208962016 conn(0x7fe4240ff8a0 msgr2=0x7fe424108610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.440+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 shutdown_connections 2024-04-25T16:28:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:28:26.440+0000 7fe42a4e6700 1 -- 172.21.15.98:0/4208962016 wait complete. 2024-04-25T16:28:26.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:26.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:26.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:26 smithi203 ceph-mon[33115]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:26.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:26 smithi203 ceph-mon[33115]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T16:28:26.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:26.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:26.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:26 smithi098 ceph-mon[29162]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:26.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:26 smithi098 ceph-mon[29162]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T16:28:26.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:26.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:26.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:26.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:26 smithi173 ceph-mon[32556]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:26.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:26 smithi173 ceph-mon[32556]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T16:28:27.141 DEBUG:teuthology.orchestra.run.smithi098:osd.9> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T16:28:27.143 INFO:tasks.cephadm:Deploying osd.10 on smithi173 with /dev/vg_nvme/lv_1... 2024-04-25T16:28:27.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T16:28:27.392 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:28:27.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:27 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:27.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:27 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:27.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:27 smithi173 ceph-mon[32556]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T16:28:27.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:27 smithi173 ceph-mon[32556]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T16:28:27.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:27 smithi173 ceph-mon[32556]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:28:27.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:27 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:27.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:27 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:27.628 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:27.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:27.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-mon[29162]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T16:28:27.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-mon[29162]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T16:28:27.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-mon[29162]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:28:27.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:27.629 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:27.629 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:28:27.408+0000 7f2bc3246700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T16:28:27.629 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:28:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:28:27.418+0000 7f2bbeffb700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:28:27.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:27 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:27.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:27 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:27.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:27 smithi203 ceph-mon[33115]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T16:28:27.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:27 smithi203 ceph-mon[33115]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T16:28:27.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:27 smithi203 ceph-mon[33115]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-04-25T16:28:27.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:27 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:27.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:27 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:28.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:28 smithi098 ceph-mon[29162]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:28:28.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:28 smithi098 ceph-mon[29162]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T16:28:28.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:28.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:28.685 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:28 smithi098 ceph-mon[29162]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:28.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:28 smithi203 ceph-mon[33115]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:28:28.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:28 smithi203 ceph-mon[33115]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T16:28:28.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:28.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:28.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:28 smithi203 ceph-mon[33115]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:28.739 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:28 smithi173 ceph-mon[32556]: from='osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-04-25T16:28:28.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:28 smithi173 ceph-mon[32556]: osdmap e59: 10 total, 9 up, 10 in 2024-04-25T16:28:28.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:28.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:28.740 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:28 smithi173 ceph-mon[32556]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-04-25T16:28:29.330 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T16:28:29.330 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T16:28:29.330 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-04-25T16:28:29.330 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-04-25T16:28:29.331 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01225 s, 856 MB/s 2024-04-25T16:28:29.331 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-04-25T16:28:29.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:29 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:28:29.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:29 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:28:29.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:29 smithi098 ceph-mon[29162]: osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] boot 2024-04-25T16:28:29.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:29 smithi098 ceph-mon[29162]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T16:28:29.594 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:29 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:29.671 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:29 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:28:29.671 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:29 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:28:29.671 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:29 smithi173 ceph-mon[32556]: osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] boot 2024-04-25T16:28:29.671 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:29 smithi173 ceph-mon[32556]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T16:28:29.671 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:29 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:29.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:29 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:28:29.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:29 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:28:29.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:29 smithi203 ceph-mon[33115]: osd.9 [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] boot 2024-04-25T16:28:29.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:29 smithi203 ceph-mon[33115]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T16:28:29.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:29 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T16:28:29.957 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-04-25T16:28:30.195 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:28:30.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:30 smithi173 ceph-mon[32556]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T16:28:30.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:30 smithi173 ceph-mon[32556]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:30.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:30 smithi203 ceph-mon[33115]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T16:28:30.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:30 smithi203 ceph-mon[33115]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:30.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:30 smithi098 ceph-mon[29162]: osdmap e61: 10 total, 10 up, 10 in 2024-04-25T16:28:30.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:30 smithi098 ceph-mon[29162]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi098 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi098 to 4272M 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.658 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:31.659 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:31.659 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi098 2024-04-25T16:28:31.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi098 to 4272M 2024-04-25T16:28:31.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:31.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:31.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 -- 172.21.15.173:0/3180796246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5201096a0 msgr2=0x7fd52010bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 --2- 172.21.15.173:0/3180796246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5201096a0 0x7fd52010bac0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd514009990 tx=0x7fd514009ca0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 -- 172.21.15.173:0/3180796246 shutdown_connections 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 --2- 172.21.15.173:0/3180796246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd52010c090 0x7fd52010e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 --2- 172.21.15.173:0/3180796246 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd52006cbb0 0x7fd520109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 --2- 172.21.15.173:0/3180796246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5201096a0 0x7fd52010bac0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 -- 172.21.15.173:0/3180796246 >> 172.21.15.173:0/3180796246 conn(0x7fd5200fe710 msgr2=0x7fd520100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 -- 172.21.15.173:0/3180796246 shutdown_connections 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.909+0000 7fd526593700 1 -- 172.21.15.173:0/3180796246 wait complete. 2024-04-25T16:28:31.911 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.910+0000 7fd526593700 1 Processor -- start 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.910+0000 7fd526593700 1 -- start start 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.910+0000 7fd526593700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd52006cbb0 0x7fd52019c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.910+0000 7fd526593700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5201096a0 0x7fd52019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.910+0000 7fd526593700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd52010c090 0x7fd5201a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.910+0000 7fd526593700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd520111ce0 con 0x7fd52006cbb0 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.910+0000 7fd526593700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd520111b60 con 0x7fd52010c090 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.910+0000 7fd526593700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd520111e60 con 0x7fd5201096a0 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd52010c090 0x7fd5201a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd52010c090 0x7fd5201a3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:49792/0 (socket says 172.21.15.173:49792) 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 -- 172.21.15.173:0/2161109256 learned_addr learned my addr 172.21.15.173:0/2161109256 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd51f7fe700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5201096a0 0x7fd52019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd51ffff700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd52006cbb0 0x7fd52019c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:31.912 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 -- 172.21.15.173:0/2161109256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5201096a0 msgr2=0x7fd52019ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:31.913 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5201096a0 0x7fd52019ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.913 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 -- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd52006cbb0 msgr2=0x7fd52019c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:31.913 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd52006cbb0 0x7fd52019c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:31.913 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 -- 172.21.15.173:0/2161109256 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5201a43b0 con 0x7fd52010c090 2024-04-25T16:28:31.913 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd524b30700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd52010c090 0x7fd5201a3dc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd51800ef30 tx=0x7fd51800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:31.913 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd51d7fa700 1 -- 172.21.15.173:0/2161109256 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5180092e0 con 0x7fd52010c090 2024-04-25T16:28:31.913 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5201a4640 con 0x7fd52010c090 2024-04-25T16:28:31.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd51d7fa700 1 -- 172.21.15.173:0/2161109256 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd518004bb0 con 0x7fd52010c090 2024-04-25T16:28:31.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd51d7fa700 1 -- 172.21.15.173:0/2161109256 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd518004020 con 0x7fd52010c090 2024-04-25T16:28:31.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.911+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5201a4b80 con 0x7fd52010c090 2024-04-25T16:28:31.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.912+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd520051190 con 0x7fd52010c090 2024-04-25T16:28:31.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.914+0000 7fd51d7fa700 1 -- 172.21.15.173:0/2161109256 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd51801b030 con 0x7fd52010c090 2024-04-25T16:28:31.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.915+0000 7fd51d7fa700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5080afb80 0x7fd5080b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:31.914 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.915+0000 7fd51d7fa700 1 -- 172.21.15.173:0/2161109256 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7fd5180de880 con 0x7fd52010c090 2024-04-25T16:28:31.915 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.915+0000 7fd51ffff700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5080afb80 0x7fd5080b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:31.915 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.915+0000 7fd51ffff700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5080afb80 0x7fd5080b2090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd52006c8b0 tx=0x7fd51000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:31.918 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:31.918+0000 7fd51d7fa700 1 -- 172.21.15.173:0/2161109256 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd518022020 con 0x7fd52010c090 2024-04-25T16:28:31.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi098 2024-04-25T16:28:31.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi098 to 4272M 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:31.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:32.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:32.123+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fd520063ad0 con 0x7fd5080afb80 2024-04-25T16:28:32.741 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:32 smithi173 ceph-mon[32556]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:32.741 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:32.741 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:32.741 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:32 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:32.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:32 smithi098 ceph-mon[29162]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:32.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:32.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:32.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:32 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:32.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:32 smithi203 ceph-mon[33115]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:32.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:32.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:32.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:32 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:33.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:33 smithi098 ceph-mon[29162]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:33.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:33 smithi173 ceph-mon[32556]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:33.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:33 smithi203 ceph-mon[33115]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:34 smithi098 ceph-mon[29162]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:34.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:34 smithi173 ceph-mon[32556]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:34.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:34 smithi203 ceph-mon[33115]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:35.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:35 smithi098 ceph-mon[29162]: from='client.? 172.21.15.173:0/274095437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]: dispatch 2024-04-25T16:28:35.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:35 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]: dispatch 2024-04-25T16:28:35.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:35 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]': finished 2024-04-25T16:28:35.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:35 smithi098 ceph-mon[29162]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T16:28:35.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:35 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:35.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:35 smithi173 ceph-mon[32556]: from='client.? 172.21.15.173:0/274095437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]: dispatch 2024-04-25T16:28:35.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:35 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]: dispatch 2024-04-25T16:28:35.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:35 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]': finished 2024-04-25T16:28:35.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:35 smithi173 ceph-mon[32556]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T16:28:35.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:35 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:35.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:35 smithi203 ceph-mon[33115]: from='client.? 172.21.15.173:0/274095437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]: dispatch 2024-04-25T16:28:35.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:35 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]: dispatch 2024-04-25T16:28:35.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:35 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18d5642e-ef6c-41dd-8aa4-9fd5190d03f7"}]': finished 2024-04-25T16:28:35.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:35 smithi203 ceph-mon[33115]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T16:28:35.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:35 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:36.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.173:0/894663523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:28:36.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:36 smithi098 ceph-mon[29162]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:36.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.173:0/894663523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:28:36.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:36 smithi173 ceph-mon[32556]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:36.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.173:0/894663523' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:28:36.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:36 smithi203 ceph-mon[33115]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:39.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:38 smithi173 ceph-mon[32556]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:39.206 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:38 smithi203 ceph-mon[33115]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:39.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:38 smithi098 ceph-mon[29162]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:41.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:40 smithi098 ceph-mon[29162]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:41.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:40 smithi173 ceph-mon[32556]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:41.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:40 smithi203 ceph-mon[33115]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:42 smithi098 ceph-mon[29162]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T16:28:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:43.260 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:42 smithi173 ceph-mon[32556]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:43.260 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T16:28:43.260 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:43.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:42 smithi203 ceph-mon[33115]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:43.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T16:28:43.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:44.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:43 smithi098 ceph-mon[29162]: Deploying daemon osd.10 on smithi173 2024-04-25T16:28:44.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:43 smithi173 ceph-mon[32556]: Deploying daemon osd.10 on smithi173 2024-04-25T16:28:44.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:43 smithi203 ceph-mon[33115]: Deploying daemon osd.10 on smithi173 2024-04-25T16:28:45.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:44 smithi098 ceph-mon[29162]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:45.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:44 smithi173 ceph-mon[32556]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:45.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:44 smithi203 ceph-mon[33115]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:47.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:46 smithi098 ceph-mon[29162]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:47.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:46 smithi173 ceph-mon[32556]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:47.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:46 smithi203 ceph-mon[33115]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:48.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:48 smithi203 ceph-mon[33115]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:48.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:48 smithi098 ceph-mon[29162]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:48.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:48 smithi173 ceph-mon[32556]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:49.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:49.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:49.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:50.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:50 smithi098 ceph-mon[29162]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:50.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:50 smithi173 ceph-mon[32556]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:50 smithi203 ceph-mon[33115]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:52.091 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 10 on host 'smithi173' 2024-04-25T16:28:52.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.091+0000 7fd51d7fa700 1 -- 172.21.15.173:0/2161109256 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fd520063ad0 con 0x7fd5080afb80 2024-04-25T16:28:52.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.093+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5080afb80 msgr2=0x7fd5080b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:52.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.093+0000 7fd526593700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5080afb80 0x7fd5080b2090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd52006c8b0 tx=0x7fd51000a380 comp rx=0 tx=0).stop 2024-04-25T16:28:52.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.093+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd52010c090 msgr2=0x7fd5201a3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.093+0000 7fd526593700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd52010c090 0x7fd5201a3dc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd51800ef30 tx=0x7fd51800c3a0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.094+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 shutdown_connections 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.094+0000 7fd526593700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5080afb80 0x7fd5080b2090 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.094+0000 7fd526593700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd52010c090 0x7fd5201a3dc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.094+0000 7fd526593700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd52006cbb0 0x7fd52019c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.094+0000 7fd526593700 1 --2- 172.21.15.173:0/2161109256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5201096a0 0x7fd52019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.094+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 >> 172.21.15.173:0/2161109256 conn(0x7fd5200fe710 msgr2=0x7fd52010a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.094+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 shutdown_connections 2024-04-25T16:28:52.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-04-25T16:28:52.094+0000 7fd526593700 1 -- 172.21.15.173:0/2161109256 wait complete. 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:28:52.491 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:52 smithi203 ceph-mon[33115]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:28:52.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:52 smithi098 ceph-mon[29162]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:28:52.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:28:52.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:52 smithi173 ceph-mon[32556]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T16:28:52.738 DEBUG:teuthology.orchestra.run.smithi173:osd.10> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T16:28:52.740 INFO:tasks.cephadm:Deploying osd.11 on smithi203 with /dev/vg_nvme/lv_1... 2024-04-25T16:28:52.741 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T16:28:52.996 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:28:54.234 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:54 smithi173 ceph-mon[32556]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T16:28:54.234 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:54 smithi173 ceph-mon[32556]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T16:28:54.234 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:54.234 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:54 smithi173 ceph-mon[32556]: from='osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:28:54.234 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:54 smithi173 ceph-mon[32556]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:28:54.234 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:54 smithi173 ceph-mon[32556]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:54.381 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:54 smithi203 ceph-mon[33115]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T16:28:54.381 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:54 smithi203 ceph-mon[33115]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T16:28:54.382 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:54.382 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:54 smithi203 ceph-mon[33115]: from='osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:28:54.382 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:54 smithi203 ceph-mon[33115]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:28:54.382 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:54 smithi203 ceph-mon[33115]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:54.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:54 smithi098 ceph-mon[29162]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T16:28:54.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:54 smithi098 ceph-mon[29162]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T16:28:54.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:54.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:54 smithi098 ceph-mon[29162]: from='osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:28:54.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:54 smithi098 ceph-mon[29162]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-04-25T16:28:54.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:54 smithi098 ceph-mon[29162]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:55.090 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:28:54 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:28:54.806+0000 7f7999356700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T16:28:55.090 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:28:54 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:28:54.820+0000 7f7991103700 -1 osd.10 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:28:55.234 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:55 smithi203 ceph-mon[33115]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:28:55.234 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:55 smithi203 ceph-mon[33115]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T16:28:55.234 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.234 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.235 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:55 smithi203 ceph-mon[33115]: osdmap e65: 11 total, 10 up, 11 in 2024-04-25T16:28:55.235 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.235 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:55 smithi203 ceph-mon[33115]: from='osd.10 ' entity='osd.10' 2024-04-25T16:28:55.235 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T16:28:55.235 INFO:teuthology.orchestra.run.smithi203.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T16:28:55.235 INFO:teuthology.orchestra.run.smithi203.stderr: stderr: 10+0 records in 2024-04-25T16:28:55.235 INFO:teuthology.orchestra.run.smithi203.stderr:10+0 records out 2024-04-25T16:28:55.235 INFO:teuthology.orchestra.run.smithi203.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126595 s, 828 MB/s 2024-04-25T16:28:55.235 INFO:teuthology.orchestra.run.smithi203.stderr:--> Zapping successful for: 2024-04-25T16:28:55.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:55 smithi098 ceph-mon[29162]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:28:55.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:55 smithi098 ceph-mon[29162]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T16:28:55.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:55 smithi098 ceph-mon[29162]: osdmap e65: 11 total, 10 up, 11 in 2024-04-25T16:28:55.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:55 smithi098 ceph-mon[29162]: from='osd.10 ' entity='osd.10' 2024-04-25T16:28:55.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:55 smithi173 ceph-mon[32556]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-04-25T16:28:55.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:55 smithi173 ceph-mon[32556]: osdmap e64: 11 total, 10 up, 11 in 2024-04-25T16:28:55.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:55 smithi173 ceph-mon[32556]: osdmap e65: 11 total, 10 up, 11 in 2024-04-25T16:28:55.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:55.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:55 smithi173 ceph-mon[32556]: from='osd.10 ' entity='osd.10' 2024-04-25T16:28:55.983 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi203:vg_nvme/lv_1 2024-04-25T16:28:56.225 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:28:56.348 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:56 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:28:56.348 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:56 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:28:56.348 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:56.348 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:56 smithi203 ceph-mon[33115]: osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] boot 2024-04-25T16:28:56.348 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:56 smithi203 ceph-mon[33115]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T16:28:56.348 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:56.348 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:56 smithi203 ceph-mon[33115]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:56.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:56 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:28:56.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:56 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:28:56.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:56.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:56 smithi098 ceph-mon[29162]: osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] boot 2024-04-25T16:28:56.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:56 smithi098 ceph-mon[29162]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T16:28:56.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:56.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:56 smithi098 ceph-mon[29162]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:56.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:56 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:28:56.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:56 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:28:56.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:56.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:56 smithi173 ceph-mon[32556]: osd.10 [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] boot 2024-04-25T16:28:56.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:56 smithi173 ceph-mon[32556]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T16:28:56.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T16:28:56.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:56 smithi173 ceph-mon[32556]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi173 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:57.309 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.310 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:57 smithi203 ceph-mon[33115]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi173 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:57.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:57.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:57 smithi098 ceph-mon[29162]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi173 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi173 to 4272M 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:57.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:28:57.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:28:57.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:57 smithi173 ceph-mon[32556]: osdmap e67: 11 total, 11 up, 11 in 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 -- 172.21.15.203:0/3289421145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0109fb0 msgr2=0x7f29c010c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 --2- 172.21.15.203:0/3289421145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0109fb0 0x7f29c010c3d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f29b800b0a0 tx=0x7f29b800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 -- 172.21.15.203:0/3289421145 shutdown_connections 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 --2- 172.21.15.203:0/3289421145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0109fb0 0x7f29c010c3d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 --2- 172.21.15.203:0/3289421145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c0104cf0 0x7f29c0107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 --2- 172.21.15.203:0/3289421145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29c0107650 0x7f29c0109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 -- 172.21.15.203:0/3289421145 >> 172.21.15.203:0/3289421145 conn(0x7f29c00fe730 msgr2=0x7f29c0100be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 -- 172.21.15.203:0/3289421145 shutdown_connections 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.884+0000 7f29c590c700 1 -- 172.21.15.203:0/3289421145 wait complete. 2024-04-25T16:28:57.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29c590c700 1 Processor -- start 2024-04-25T16:28:57.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29c590c700 1 -- start start 2024-04-25T16:28:57.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29c590c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c0104cf0 0x7f29c0104550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:57.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29c590c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0107650 0x7f29c0102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:57.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29c590c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29c0109fb0 0x7f29c0103140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29c590c700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29c010faf0 con 0x7f29c0104cf0 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29c590c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f29c010f970 con 0x7f29c0109fb0 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29c590c700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f29c010fc70 con 0x7f29c0107650 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.885+0000 7f29b67fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0107650 0x7f29c0102c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29b67fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0107650 0x7f29c0102c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:40492/0 (socket says 172.21.15.203:40492) 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29b67fc700 1 -- 172.21.15.203:0/2272118312 learned_addr learned my addr 172.21.15.203:0/2272118312 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29beffd700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c0104cf0 0x7f29c0104550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29bf7fe700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29c0109fb0 0x7f29c0103140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29b67fc700 1 -- 172.21.15.203:0/2272118312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29c0109fb0 msgr2=0x7f29c0103140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29b67fc700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29c0109fb0 0x7f29c0103140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29b67fc700 1 -- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c0104cf0 msgr2=0x7f29c0104550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29b67fc700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c0104cf0 0x7f29c0104550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:28:57.889 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29b67fc700 1 -- 172.21.15.203:0/2272118312 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c01038a0 con 0x7f29c0107650 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29beffd700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c0104cf0 0x7f29c0104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29b67fc700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0107650 0x7f29c0102c00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f29a800c920 tx=0x7f29a800cce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29c490a700 1 -- 172.21.15.203:0/2272118312 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a8007b00 con 0x7f29c0107650 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29c490a700 1 -- 172.21.15.203:0/2272118312 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29a8004cc0 con 0x7f29c0107650 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29c490a700 1 -- 172.21.15.203:0/2272118312 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29a80054f0 con 0x7f29c0107650 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c01a4710 con 0x7f29c0107650 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.886+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c01a4bf0 con 0x7f29c0107650 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.887+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c0051190 con 0x7f29c0107650 2024-04-25T16:28:57.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.890+0000 7f29c490a700 1 -- 172.21.15.203:0/2272118312 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f29a8007ca0 con 0x7f29c0107650 2024-04-25T16:28:57.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.890+0000 7f29c490a700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29ac0afb80 0x7f29ac0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:28:57.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.890+0000 7f29c490a700 1 -- 172.21.15.203:0/2272118312 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6694+0+0 (secure 0 0 0) 0x7f29a80de590 con 0x7f29c0107650 2024-04-25T16:28:57.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.891+0000 7f29beffd700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29ac0afb80 0x7f29ac0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:28:57.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.891+0000 7f29beffd700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29ac0afb80 0x7f29ac0b2090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f29b0005d90 tx=0x7f29b0005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:28:57.895 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:57.894+0000 7f29c490a700 1 -- 172.21.15.203:0/2272118312 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29a8014030 con 0x7f29c0107650 2024-04-25T16:28:58.105 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:28:58.104+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f29c0063ad0 con 0x7f29ac0afb80 2024-04-25T16:28:58.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:58 smithi203 ceph-mon[33115]: osdmap e68: 11 total, 11 up, 11 in 2024-04-25T16:28:58.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:58 smithi203 ceph-mon[33115]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:28:58.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:58.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:58.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:58.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:58 smithi098 ceph-mon[29162]: osdmap e68: 11 total, 11 up, 11 in 2024-04-25T16:28:58.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:58 smithi098 ceph-mon[29162]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:28:58.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:58.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:58.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:58.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:58 smithi173 ceph-mon[32556]: osdmap e68: 11 total, 11 up, 11 in 2024-04-25T16:28:58.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:58 smithi173 ceph-mon[32556]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:28:58.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T16:28:58.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T16:28:58.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:28:59.654 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:28:59 smithi098 ceph-mon[29162]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:59.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:28:59 smithi203 ceph-mon[33115]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:28:59.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:28:59 smithi173 ceph-mon[32556]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi203:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:29:00.674 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:00 smithi098 ceph-mon[29162]: pgmap v234: 1 pgs: 1 activating+degraded; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T16:29:00.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:00 smithi203 ceph-mon[33115]: pgmap v234: 1 pgs: 1 activating+degraded; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T16:29:00.774 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:00 smithi173 ceph-mon[32556]: pgmap v234: 1 pgs: 1 activating+degraded; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T16:29:01.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:01 smithi203 ceph-mon[33115]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-25T16:29:01.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:01 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/371223106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]: dispatch 2024-04-25T16:29:01.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:01 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]: dispatch 2024-04-25T16:29:01.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:01 smithi203 ceph-mon[33115]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]': finished 2024-04-25T16:29:01.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:01 smithi203 ceph-mon[33115]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T16:29:01.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:01 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:01.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:01 smithi098 ceph-mon[29162]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-25T16:29:01.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:01 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/371223106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]: dispatch 2024-04-25T16:29:01.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:01 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]: dispatch 2024-04-25T16:29:01.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:01 smithi098 ceph-mon[29162]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]': finished 2024-04-25T16:29:01.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:01 smithi098 ceph-mon[29162]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T16:29:01.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:01 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:01.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:01 smithi173 ceph-mon[32556]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-25T16:29:01.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:01 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/371223106' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]: dispatch 2024-04-25T16:29:01.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:01 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]: dispatch 2024-04-25T16:29:01.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:01 smithi173 ceph-mon[32556]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09b2102c-57de-477e-9a48-008e84d76db9"}]': finished 2024-04-25T16:29:01.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:01 smithi173 ceph-mon[32556]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T16:29:01.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:02.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:02 smithi203 ceph-mon[33115]: from='client.? 172.21.15.203:0/2653497941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:29:02.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:02 smithi203 ceph-mon[33115]: pgmap v236: 1 pgs: 1 activating+degraded; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T16:29:02.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:02 smithi098 ceph-mon[29162]: from='client.? 172.21.15.203:0/2653497941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:29:02.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:02 smithi098 ceph-mon[29162]: pgmap v236: 1 pgs: 1 activating+degraded; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T16:29:02.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:02 smithi173 ceph-mon[32556]: from='client.? 172.21.15.203:0/2653497941' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T16:29:02.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:02 smithi173 ceph-mon[32556]: pgmap v236: 1 pgs: 1 activating+degraded; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-04-25T16:29:04.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:04 smithi173 ceph-mon[32556]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T16:29:04.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:04 smithi173 ceph-mon[32556]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-25T16:29:04.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:04 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:29:04.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:04 smithi203 ceph-mon[33115]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T16:29:04.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:04 smithi203 ceph-mon[33115]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-25T16:29:04.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:04 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:29:04.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:04 smithi098 ceph-mon[29162]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-25T16:29:04.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:04 smithi098 ceph-mon[29162]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-25T16:29:04.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:04 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:29:07.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:06 smithi098 ceph-mon[29162]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T16:29:07.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:06 smithi173 ceph-mon[32556]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T16:29:07.310 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:06 smithi203 ceph-mon[33115]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-25T16:29:09.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:08 smithi203 ceph-mon[33115]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:29:09.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T16:29:09.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:08 smithi098 ceph-mon[29162]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:29:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T16:29:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:08 smithi173 ceph-mon[32556]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:29:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T16:29:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:09 smithi098 ceph-mon[29162]: Deploying daemon osd.11 on smithi203 2024-04-25T16:29:10.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:09 smithi173 ceph-mon[32556]: Deploying daemon osd.11 on smithi203 2024-04-25T16:29:10.398 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:09 smithi203 ceph-mon[33115]: Deploying daemon osd.11 on smithi203 2024-04-25T16:29:11.110 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:10 smithi203 ceph-mon[33115]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:29:11.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:10 smithi098 ceph-mon[29162]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:29:11.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:10 smithi173 ceph-mon[32556]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T16:29:13.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:12 smithi203 ceph-mon[33115]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T16:29:13.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:12 smithi098 ceph-mon[29162]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T16:29:13.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:12 smithi173 ceph-mon[32556]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 53 KiB/s, 0 objects/s recovering 2024-04-25T16:29:15.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:14 smithi098 ceph-mon[29162]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T16:29:15.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:15.691 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:15.692 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:14 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:15.692 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:14 smithi203 ceph-mon[33115]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T16:29:15.692 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:15.693 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:15.693 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:14 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:15.693 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:14 smithi173 ceph-mon[32556]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-25T16:29:15.693 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:15.693 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:15.693 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:14 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:16.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:16 smithi203 ceph-mon[33115]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:16.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:16 smithi098 ceph-mon[29162]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:16.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:16 smithi173 ceph-mon[32556]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:18.876 INFO:teuthology.orchestra.run.smithi203.stdout:Created osd(s) 11 on host 'smithi203' 2024-04-25T16:29:18.876 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.876+0000 7f29c490a700 1 -- 172.21.15.203:0/2272118312 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f29c0063ad0 con 0x7f29ac0afb80 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.878+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29ac0afb80 msgr2=0x7f29ac0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.878+0000 7f29c590c700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29ac0afb80 0x7f29ac0b2090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f29b0005d90 tx=0x7f29b0005d20 comp rx=0 tx=0).stop 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.878+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0107650 msgr2=0x7f29c0102c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.878+0000 7f29c590c700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0107650 0x7f29c0102c00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f29a800c920 tx=0x7f29a800cce0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.879+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 shutdown_connections 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.879+0000 7f29c590c700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29ac0afb80 0x7f29ac0b2090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.879+0000 7f29c590c700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29c0109fb0 0x7f29c0103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.879+0000 7f29c590c700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29c0104cf0 0x7f29c0104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.879+0000 7f29c590c700 1 --2- 172.21.15.203:0/2272118312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29c0107650 0x7f29c0102c00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.879+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 >> 172.21.15.203:0/2272118312 conn(0x7f29c00fe730 msgr2=0x7f29c0105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.879+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 shutdown_connections 2024-04-25T16:29:18.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:18.879+0000 7f29c590c700 1 -- 172.21.15.203:0/2272118312 wait complete. 2024-04-25T16:29:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:19.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:29:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:29:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:18 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:29:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:29:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:18 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:19.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:19.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:18 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:19.535 DEBUG:teuthology.orchestra.run.smithi203:osd.11> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T16:29:19.537 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T16:29:19.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T16:29:19.783 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:20.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:19 smithi098 ceph-mon[29162]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T16:29:20.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:19 smithi098 ceph-mon[29162]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T16:29:20.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:19 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:20.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:19 smithi098 ceph-mon[29162]: from='osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:29:20.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:19 smithi098 ceph-mon[29162]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:29:20.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:19 smithi203 ceph-mon[33115]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T16:29:20.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:19 smithi203 ceph-mon[33115]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T16:29:20.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:19 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:20.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:19 smithi203 ceph-mon[33115]: from='osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:29:20.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:19 smithi203 ceph-mon[33115]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:29:20.189 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:29:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:29:19.975+0000 7f3cd4476700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T16:29:20.189 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:29:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:29:19.988+0000 7f3cd022b700 -1 osd.11 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T16:29:20.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:19 smithi173 ceph-mon[32556]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T16:29:20.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:19 smithi173 ceph-mon[32556]: osdmap e70: 12 total, 11 up, 12 in 2024-04-25T16:29:20.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:19 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:20.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:19 smithi173 ceph-mon[32556]: from='osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:29:20.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:19 smithi173 ceph-mon[32556]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]: dispatch 2024-04-25T16:29:21.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:20 smithi203 ceph-mon[33115]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:21.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:20 smithi203 ceph-mon[33115]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:29:21.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:20 smithi203 ceph-mon[33115]: osdmap e71: 12 total, 11 up, 12 in 2024-04-25T16:29:21.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:20 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:20 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.042 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:20 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:20 smithi098 ceph-mon[29162]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:21.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:20 smithi098 ceph-mon[29162]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:29:21.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:20 smithi098 ceph-mon[29162]: osdmap e71: 12 total, 11 up, 12 in 2024-04-25T16:29:21.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:20 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:20 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:20 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:20 smithi173 ceph-mon[32556]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:21.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:20 smithi173 ceph-mon[32556]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi203", "root=default"]}]': finished 2024-04-25T16:29:21.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:20 smithi173 ceph-mon[32556]: osdmap e71: 12 total, 11 up, 12 in 2024-04-25T16:29:21.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:20 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:20 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:20 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:21.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 -- 172.21.15.98:0/3761977461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c41048b0 msgr2=0x7f33c410c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 --2- 172.21.15.98:0/3761977461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c41048b0 0x7f33c410c5c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f33bc00b0a0 tx=0x7f33bc00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 -- 172.21.15.98:0/3761977461 shutdown_connections 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 --2- 172.21.15.98:0/3761977461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33c4105880 0x7f33c4105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 --2- 172.21.15.98:0/3761977461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c41048b0 0x7f33c410c5c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 --2- 172.21.15.98:0/3761977461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c4103e30 0x7f33c4104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 -- 172.21.15.98:0/3761977461 >> 172.21.15.98:0/3761977461 conn(0x7f33c40ff8a0 msgr2=0x7f33c4101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 -- 172.21.15.98:0/3761977461 shutdown_connections 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 -- 172.21.15.98:0/3761977461 wait complete. 2024-04-25T16:29:21.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 Processor -- start 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.595+0000 7f33cb312700 1 -- start start 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33cb312700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c4103e30 0x7f33c419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33cb312700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c41048b0 0x7f33c419ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33cb312700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33c4105880 0x7f33c41a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33cb312700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f33c410fd60 con 0x7f33c4103e30 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33cb312700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f33c410fbe0 con 0x7f33c41048b0 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33cb312700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f33c410fee0 con 0x7f33c4105880 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c4103e30 0x7f33c419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c4103e30 0x7f33c419a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32796/0 (socket says 172.21.15.98:32796) 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 -- 172.21.15.98:0/376407104 learned_addr learned my addr 172.21.15.98:0/376407104 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c98af700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33c4105880 0x7f33c41a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c88ad700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c41048b0 0x7f33c419ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 -- 172.21.15.98:0/376407104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33c4105880 msgr2=0x7f33c41a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33c4105880 0x7f33c41a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 -- 172.21.15.98:0/376407104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c41048b0 msgr2=0x7f33c419ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c41048b0 0x7f33c419ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 -- 172.21.15.98:0/376407104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c41a2470 con 0x7f33c4103e30 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33c90ae700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c4103e30 0x7f33c419a910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f33b400b4d0 tx=0x7f33b400b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.596+0000 7f33ba7fc700 1 -- 172.21.15.98:0/376407104 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4015020 con 0x7f33c4103e30 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.597+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c41a2760 con 0x7f33c4103e30 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.597+0000 7f33ba7fc700 1 -- 172.21.15.98:0/376407104 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f33b4011cc0 con 0x7f33c4103e30 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.597+0000 7f33ba7fc700 1 -- 172.21.15.98:0/376407104 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4010c30 con 0x7f33c4103e30 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.597+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c41a2d70 con 0x7f33c4103e30 2024-04-25T16:29:21.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.598+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33c4051190 con 0x7f33c4103e30 2024-04-25T16:29:21.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.599+0000 7f33ba7fc700 1 -- 172.21.15.98:0/376407104 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f33b4018020 con 0x7f33c4103e30 2024-04-25T16:29:21.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.599+0000 7f33ba7fc700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f33b00afb80 0x7f33b00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:21.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.600+0000 7f33c88ad700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f33b00afb80 0x7f33b00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:21.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.600+0000 7f33ba7fc700 1 -- 172.21.15.98:0/376407104 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f33b40de470 con 0x7f33c4103e30 2024-04-25T16:29:21.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.600+0000 7f33c88ad700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f33b00afb80 0x7f33b00b2090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f33c0006fd0 tx=0x7f33c0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:21.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.603+0000 7f33ba7fc700 1 -- 172.21.15.98:0/376407104 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33b401f020 con 0x7f33c4103e30 2024-04-25T16:29:21.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.803+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f33c402e850 con 0x7f33c4103e30 2024-04-25T16:29:21.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.803+0000 7f33ba7fc700 1 -- 172.21.15.98:0/376407104 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f33b40a74f0 con 0x7f33c4103e30 2024-04-25T16:29:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:21.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f33b00afb80 msgr2=0x7f33b00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f33b00afb80 0x7f33b00b2090 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f33c0006fd0 tx=0x7f33c0008040 comp rx=0 tx=0).stop 2024-04-25T16:29:21.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c4103e30 msgr2=0x7f33c419a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:21.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c4103e30 0x7f33c419a910 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f33b400b4d0 tx=0x7f33b400b7e0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 shutdown_connections 2024-04-25T16:29:21.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f33b00afb80 0x7f33b00b2090 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33c4105880 0x7f33c41a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f33c4103e30 0x7f33c419a910 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 --2- 172.21.15.98:0/376407104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c41048b0 0x7f33c419ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:21.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 >> 172.21.15.98:0/376407104 conn(0x7f33c40ff8a0 msgr2=0x7f33c4108610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:21.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 shutdown_connections 2024-04-25T16:29:21.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:21.805+0000 7f33cb312700 1 -- 172.21.15.98:0/376407104 wait complete. 2024-04-25T16:29:22.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:21 smithi098 ceph-mon[29162]: purged_snaps scrub starts 2024-04-25T16:29:22.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:21 smithi098 ceph-mon[29162]: purged_snaps scrub ok 2024-04-25T16:29:22.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:21 smithi098 ceph-mon[29162]: osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] boot 2024-04-25T16:29:22.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:21 smithi098 ceph-mon[29162]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T16:29:22.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:21 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:22.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:21 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/376407104' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T16:29:22.177 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:21 smithi098 ceph-mon[29162]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:22.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:21 smithi203 ceph-mon[33115]: purged_snaps scrub starts 2024-04-25T16:29:22.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:21 smithi203 ceph-mon[33115]: purged_snaps scrub ok 2024-04-25T16:29:22.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:21 smithi203 ceph-mon[33115]: osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] boot 2024-04-25T16:29:22.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:21 smithi203 ceph-mon[33115]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T16:29:22.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:21 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:22.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:21 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/376407104' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T16:29:22.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:21 smithi203 ceph-mon[33115]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:21 smithi173 ceph-mon[32556]: purged_snaps scrub starts 2024-04-25T16:29:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:21 smithi173 ceph-mon[32556]: purged_snaps scrub ok 2024-04-25T16:29:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:21 smithi173 ceph-mon[32556]: osd.11 [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] boot 2024-04-25T16:29:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:21 smithi173 ceph-mon[32556]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T16:29:22.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:21 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T16:29:22.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:21 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/376407104' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T16:29:22.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:21 smithi173 ceph-mon[32556]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-04-25T16:29:22.643 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1714062560,"num_in_osds":12,"osd_in_since":1714062541,"num_remapped_pgs":0} 2024-04-25T16:29:22.643 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T16:29:22.895 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:23.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T16:29:23.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: Detected new or changed devices on smithi203 2024-04-25T16:29:23.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:23.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:22 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: Detected new or changed devices on smithi203 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:23.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:22 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: Detected new or changed devices on smithi203 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi203 to 4272M 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:23.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:22 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:24.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:24 smithi098 ceph-mon[29162]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:24.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:24 smithi173 ceph-mon[32556]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:24.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:24 smithi203 ceph-mon[33115]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:24.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.553+0000 7f454c4f0700 1 -- 172.21.15.98:0/1059930814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 msgr2=0x7f45441037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.553+0000 7f454c4f0700 1 --2- 172.21.15.98:0/1059930814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 0x7f45441037a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f45340099c0 tx=0x7f4534009cd0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 -- 172.21.15.98:0/1059930814 shutdown_connections 2024-04-25T16:29:24.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 --2- 172.21.15.98:0/1059930814 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4544108610 0x7f454410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 --2- 172.21.15.98:0/1059930814 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 0x7f45441037a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 --2- 172.21.15.98:0/1059930814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4544107cc0 0x7f45441080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 -- 172.21.15.98:0/1059930814 >> 172.21.15.98:0/1059930814 conn(0x7f45440fec90 msgr2=0x7f45441010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:24.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 -- 172.21.15.98:0/1059930814 shutdown_connections 2024-04-25T16:29:24.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 -- 172.21.15.98:0/1059930814 wait complete. 2024-04-25T16:29:24.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 Processor -- start 2024-04-25T16:29:24.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.554+0000 7f454c4f0700 1 -- start start 2024-04-25T16:29:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454c4f0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 0x7f45441a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454c4f0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4544107cc0 0x7f45441a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454c4f0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4544108610 0x7f454419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454c4f0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45441130f0 con 0x7f4544103390 2024-04-25T16:29:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454c4f0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4544112f70 con 0x7f4544107cc0 2024-04-25T16:29:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454c4f0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4544113270 con 0x7f4544108610 2024-04-25T16:29:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 0x7f45441a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 0x7f45441a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32808/0 (socket says 172.21.15.98:32808) 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 -- 172.21.15.98:0/4139653939 learned_addr learned my addr 172.21.15.98:0/4139653939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454aa8d700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4544108610 0x7f454419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f4549a8b700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4544107cc0 0x7f45441a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 -- 172.21.15.98:0/4139653939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4544108610 msgr2=0x7f454419aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4544108610 0x7f454419aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 -- 172.21.15.98:0/4139653939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4544107cc0 msgr2=0x7f45441a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4544107cc0 0x7f45441a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 -- 172.21.15.98:0/4139653939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f454419b0d0 con 0x7f4544103390 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f4549a8b700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4544107cc0 0x7f45441a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.555+0000 7f454a28c700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 0x7f45441a0800 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4534006010 tx=0x7f45340045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.556+0000 7f453b7fe700 1 -- 172.21.15.98:0/4139653939 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534026070 con 0x7f4544103390 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.556+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f454419b360 con 0x7f4544103390 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.556+0000 7f453b7fe700 1 -- 172.21.15.98:0/4139653939 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f453400f460 con 0x7f4544103390 2024-04-25T16:29:24.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.556+0000 7f453b7fe700 1 -- 172.21.15.98:0/4139653939 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4534009e60 con 0x7f4544103390 2024-04-25T16:29:24.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.556+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45441a7690 con 0x7f4544103390 2024-04-25T16:29:24.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.556+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4544104840 con 0x7f4544103390 2024-04-25T16:29:24.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.559+0000 7f453b7fe700 1 -- 172.21.15.98:0/4139653939 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4534029070 con 0x7f4544103390 2024-04-25T16:29:24.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.559+0000 7f453b7fe700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45300afb80 0x7f45300b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:24.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.559+0000 7f4549a8b700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45300afb80 0x7f45300b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:24.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.560+0000 7f453b7fe700 1 -- 172.21.15.98:0/4139653939 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f45340df3a0 con 0x7f4544103390 2024-04-25T16:29:24.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.560+0000 7f4549a8b700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45300afb80 0x7f45300b2090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4540005950 tx=0x7f45400058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:24.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.565+0000 7f453b7fe700 1 -- 172.21.15.98:0/4139653939 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4534013070 con 0x7f4544103390 2024-04-25T16:29:24.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.760+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f454419bf50 con 0x7f4544103390 2024-04-25T16:29:24.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.762+0000 7f453b7fe700 1 -- 172.21.15.98:0/4139653939 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18467 (secure 0 0 0) 0x7f45340a84b0 con 0x7f4544103390 2024-04-25T16:29:24.762 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:24.762 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":73,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:29:21.989091+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45300afb80 msgr2=0x7f45300b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45300afb80 0x7f45300b2090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4540005950 tx=0x7f45400058e0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 msgr2=0x7f45441a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 0x7f45441a0800 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4534006010 tx=0x7f45340045e0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 shutdown_connections 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45300afb80 0x7f45300b2090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4544108610 0x7f454419aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4544103390 0x7f45441a0800 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 --2- 172.21.15.98:0/4139653939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4544107cc0 0x7f45441a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:24.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 >> 172.21.15.98:0/4139653939 conn(0x7f45440fec90 msgr2=0x7f454406b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:24.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 shutdown_connections 2024-04-25T16:29:24.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:24.764+0000 7f454c4f0700 1 -- 172.21.15.98:0/4139653939 wait complete. 2024-04-25T16:29:25.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:25 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4139653939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:25.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:25 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4139653939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:25.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:25 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4139653939' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:25.385 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:29:25.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T16:29:25.619 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:29:26.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:26 smithi098 ceph-mon[29162]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:26.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:26 smithi173 ceph-mon[32556]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:26.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:26 smithi203 ceph-mon[33115]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:27.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2629295018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec04109740 msgr2=0x7fec041110a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:27.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2629295018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec04109740 0x7fec041110a0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7febfc00b0d0 tx=0x7febfc00b3e0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2629295018 shutdown_connections 2024-04-25T16:29:27.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2629295018 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec04109740 0x7fec041110a0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2629295018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec041044c0 0x7fec041048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2629295018 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec04108df0 0x7fec04109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2629295018 >> 172.21.15.98:0/2629295018 conn(0x7fec040ffd90 msgr2=0x7fec041021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:27.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2629295018 shutdown_connections 2024-04-25T16:29:27.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.317+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2629295018 wait complete. 2024-04-25T16:29:27.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.318+0000 7fec0a8ba700 1 Processor -- start 2024-04-25T16:29:27.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.318+0000 7fec0a8ba700 1 -- start start 2024-04-25T16:29:27.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec0a8ba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec041044c0 0x7fec041a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:27.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec0a8ba700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec04108df0 0x7fec041a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:27.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec0a8ba700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec04109740 0x7fec0419aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec0a8ba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fec04114270 con 0x7fec041044c0 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec0a8ba700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fec041140f0 con 0x7fec04108df0 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec0a8ba700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fec041143f0 con 0x7fec04109740 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec041044c0 0x7fec041a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec041044c0 0x7fec041a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44394/0 (socket says 172.21.15.98:44394) 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 -- 172.21.15.98:0/2903546247 learned_addr learned my addr 172.21.15.98:0/2903546247 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec08e57700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec04109740 0x7fec0419aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec037fe700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec04108df0 0x7fec041a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 -- 172.21.15.98:0/2903546247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec04109740 msgr2=0x7fec0419aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec04109740 0x7fec0419aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 -- 172.21.15.98:0/2903546247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec04108df0 msgr2=0x7fec041a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec04108df0 0x7fec041a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 -- 172.21.15.98:0/2903546247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0419b050 con 0x7fec041044c0 2024-04-25T16:29:27.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.319+0000 7fec03fff700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec041044c0 0x7fec041a07f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7febf400c9b0 tx=0x7febf400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:27.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.320+0000 7fec017fa700 1 -- 172.21.15.98:0/2903546247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4007b00 con 0x7fec041044c0 2024-04-25T16:29:27.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.320+0000 7fec017fa700 1 -- 172.21.15.98:0/2903546247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7febf4007ca0 con 0x7fec041044c0 2024-04-25T16:29:27.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.320+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0419b340 con 0x7fec041044c0 2024-04-25T16:29:27.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.320+0000 7fec017fa700 1 -- 172.21.15.98:0/2903546247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4018660 con 0x7fec041044c0 2024-04-25T16:29:27.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.320+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec041a7680 con 0x7fec041044c0 2024-04-25T16:29:27.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.321+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec04051190 con 0x7fec041044c0 2024-04-25T16:29:27.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.322+0000 7fec017fa700 1 -- 172.21.15.98:0/2903546247 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7febf400f450 con 0x7fec041044c0 2024-04-25T16:29:27.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.323+0000 7fec017fa700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7febec0afb80 0x7febec0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:27.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.323+0000 7fec037fe700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7febec0afb80 0x7febec0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:27.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.323+0000 7fec017fa700 1 -- 172.21.15.98:0/2903546247 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7febf40dea80 con 0x7fec041044c0 2024-04-25T16:29:27.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.323+0000 7fec037fe700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7febec0afb80 0x7febec0b2090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7febf8005950 tx=0x7febf80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:27.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.327+0000 7fec017fa700 1 -- 172.21.15.98:0/2903546247 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7febf4014030 con 0x7fec041044c0 2024-04-25T16:29:27.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.522+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec0419bf40 con 0x7fec041044c0 2024-04-25T16:29:27.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.523+0000 7fec017fa700 1 -- 172.21.15.98:0/2903546247 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7febf40a7af0 con 0x7fec041044c0 2024-04-25T16:29:27.523 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:29:27.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7febec0afb80 msgr2=0x7febec0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:27.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7febec0afb80 0x7febec0b2090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7febf8005950 tx=0x7febf80058e0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec041044c0 msgr2=0x7fec041a07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:27.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec041044c0 0x7fec041a07f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7febf400c9b0 tx=0x7febf400ccc0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 shutdown_connections 2024-04-25T16:29:27.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7febec0afb80 0x7febec0b2090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec04109740 0x7fec0419aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fec041044c0 0x7fec041a07f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 --2- 172.21.15.98:0/2903546247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec04108df0 0x7fec041a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:27.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 >> 172.21.15.98:0/2903546247 conn(0x7fec040ffd90 msgr2=0x7fec041085b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:27.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 shutdown_connections 2024-04-25T16:29:27.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:27.525+0000 7fec0a8ba700 1 -- 172.21.15.98:0/2903546247 wait complete. 2024-04-25T16:29:27.843 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:27 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2903546247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:29:27.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:27 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2903546247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:29:28.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:27 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2903546247' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:29:28.249 INFO:tasks.cephadm:Adding mds.a on smithi098 2024-04-25T16:29:28.249 INFO:tasks.cephadm:Adding mds.d on smithi098 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.g on smithi098 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.j on smithi098 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.b on smithi173 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.e on smithi173 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.h on smithi173 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.k on smithi173 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.c on smithi203 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.f on smithi203 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.i on smithi203 2024-04-25T16:29:28.250 INFO:tasks.cephadm:Adding mds.l on smithi203 2024-04-25T16:29:28.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l' 2024-04-25T16:29:28.503 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:29:28.874 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:28 smithi203 ceph-mon[33115]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:29.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:28 smithi098 ceph-mon[29162]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:29.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:28 smithi173 ceph-mon[32556]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 -- 172.21.15.203:0/369893631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4104210 msgr2=0x7f25b4104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 --2- 172.21.15.203:0/369893631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4104210 0x7f25b4104620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009990 tx=0x7f25a4009ca0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 -- 172.21.15.203:0/369893631 shutdown_connections 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 --2- 172.21.15.203:0/369893631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4104210 0x7f25b4104620 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 --2- 172.21.15.203:0/369893631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25b4105ce0 0x7f25b410c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 --2- 172.21.15.203:0/369893631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4105340 0x7f25b4105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 -- 172.21.15.203:0/369893631 >> 172.21.15.203:0/369893631 conn(0x7f25b40ff890 msgr2=0x7f25b4101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 -- 172.21.15.203:0/369893631 shutdown_connections 2024-04-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.173+0000 7f25b951e700 1 -- 172.21.15.203:0/369893631 wait complete. 2024-04-25T16:29:30.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.174+0000 7f25b951e700 1 Processor -- start 2024-04-25T16:29:30.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.174+0000 7f25b951e700 1 -- start start 2024-04-25T16:29:30.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.174+0000 7f25b951e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25b4104210 0x7f25b41a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:30.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.174+0000 7f25b951e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4105340 0x7f25b41a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.174+0000 7f25b951e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4105ce0 0x7f25b41a8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.174+0000 7f25b951e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410fb10 con 0x7f25b4105ce0 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.174+0000 7f25b951e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410f990 con 0x7f25b4104210 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.174+0000 7f25b951e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f25b410fc90 con 0x7f25b4105340 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4105340 0x7f25b41a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4105340 0x7f25b41a12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:38276/0 (socket says 172.21.15.203:38276) 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25b2ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25b4104210 0x7f25b41a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 -- 172.21.15.203:0/1923213122 learned_addr learned my addr 172.21.15.203:0/1923213122 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25b37fe700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4105ce0 0x7f25b41a8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 -- 172.21.15.203:0/1923213122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25b4104210 msgr2=0x7f25b41a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25b4104210 0x7f25b41a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 -- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4105ce0 msgr2=0x7f25b41a8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4105ce0 0x7f25b41a8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 -- 172.21.15.203:0/1923213122 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b41a88e0 con 0x7f25b4105340 2024-04-25T16:29:30.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25b2ffd700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25b4104210 0x7f25b41a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:29:30.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25aa7fc700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4105340 0x7f25b41a12b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f259c00d950 tx=0x7f259c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:30.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25b0ff9700 1 -- 172.21.15.203:0/1923213122 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c016070 con 0x7f25b4105340 2024-04-25T16:29:30.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b41a8bd0 con 0x7f25b4105340 2024-04-25T16:29:30.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25b0ff9700 1 -- 172.21.15.203:0/1923213122 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f259c00fd20 con 0x7f25b4105340 2024-04-25T16:29:30.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.175+0000 7f25b0ff9700 1 -- 172.21.15.203:0/1923213122 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c00e3b0 con 0x7f25b4105340 2024-04-25T16:29:30.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.176+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b41a9160 con 0x7f25b4105340 2024-04-25T16:29:30.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.176+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b4051190 con 0x7f25b4105340 2024-04-25T16:29:30.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.178+0000 7f25b0ff9700 1 -- 172.21.15.203:0/1923213122 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f259c019070 con 0x7f25b4105340 2024-04-25T16:29:30.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.179+0000 7f25b0ff9700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f25a00afb80 0x7f25a00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:30.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.179+0000 7f25b0ff9700 1 -- 172.21.15.203:0/1923213122 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f259c0de5a0 con 0x7f25b4105340 2024-04-25T16:29:30.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.179+0000 7f25b2ffd700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f25a00afb80 0x7f25a00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:30.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.179+0000 7f25b2ffd700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f25a00afb80 0x7f25a00b2090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f25b41a2170 tx=0x7f25a401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:30.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.185+0000 7f25b0ff9700 1 -- 172.21.15.203:0/1923213122 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f259c0a7620 con 0x7f25b4105340 2024-04-25T16:29:30.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.392+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}) -- 0x7f25b4063bd0 con 0x7f25a00afb80 2024-04-25T16:29:30.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.401+0000 7f25b0ff9700 1 -- 172.21.15.203:0/1923213122 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f25b4063bd0 con 0x7f25a00afb80 2024-04-25T16:29:30.401 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mds.all update... 2024-04-25T16:29:30.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.403+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f25a00afb80 msgr2=0x7f25a00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:30.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.403+0000 7f25b951e700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f25a00afb80 0x7f25a00b2090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f25b41a2170 tx=0x7f25a401a040 comp rx=0 tx=0).stop 2024-04-25T16:29:30.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.403+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4105340 msgr2=0x7f25b41a12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:30.403 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.403+0000 7f25b951e700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4105340 0x7f25b41a12b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f259c00d950 tx=0x7f259c00dc60 comp rx=0 tx=0).stop 2024-04-25T16:29:30.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.403+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 shutdown_connections 2024-04-25T16:29:30.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.403+0000 7f25b951e700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f25a00afb80 0x7f25a00b2090 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.403+0000 7f25b951e700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f25b4105ce0 0x7f25b41a8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.404+0000 7f25b951e700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25b4104210 0x7f25b41a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.404+0000 7f25b951e700 1 --2- 172.21.15.203:0/1923213122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f25b4105340 0x7f25b41a12b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:30.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.404+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 >> 172.21.15.203:0/1923213122 conn(0x7f25b40ff890 msgr2=0x7f25b41065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:30.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.404+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 shutdown_connections 2024-04-25T16:29:30.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-25T16:29:30.404+0000 7f25b951e700 1 -- 172.21.15.203:0/1923213122 wait complete. 2024-04-25T16:29:31.069 DEBUG:teuthology.orchestra.run.smithi098:mds.a> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T16:29:31.072 DEBUG:teuthology.orchestra.run.smithi098:mds.d> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T16:29:31.122 DEBUG:teuthology.orchestra.run.smithi098:mds.g> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T16:29:31.168 DEBUG:teuthology.orchestra.run.smithi098:mds.j> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T16:29:31.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:30 smithi203 ceph-mon[33115]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:31.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:30 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:31.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:30 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:31.213 DEBUG:teuthology.orchestra.run.smithi173:mds.b> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T16:29:31.216 DEBUG:teuthology.orchestra.run.smithi173:mds.e> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T16:29:31.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:30 smithi098 ceph-mon[29162]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:31.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:30 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:31.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:30 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:31.235 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:30 smithi173 ceph-mon[32556]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:31.235 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:30 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:31.235 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:30 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:29:31.237 DEBUG:teuthology.orchestra.run.smithi173:mds.h> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T16:29:31.281 DEBUG:teuthology.orchestra.run.smithi173:mds.k> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T16:29:31.324 DEBUG:teuthology.orchestra.run.smithi203:mds.c> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T16:29:31.326 DEBUG:teuthology.orchestra.run.smithi203:mds.f> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T16:29:31.369 DEBUG:teuthology.orchestra.run.smithi203:mds.i> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T16:29:31.412 DEBUG:teuthology.orchestra.run.smithi203:mds.l> sudo journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T16:29:31.455 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T16:29:31.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:29:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 -- 172.21.15.98:0/2690350366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8108920 msgr2=0x7feaa810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 --2- 172.21.15.98:0/2690350366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8108920 0x7feaa810ad40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feaa0009ca0 tx=0x7feaa002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 -- 172.21.15.98:0/2690350366 shutdown_connections 2024-04-25T16:29:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 --2- 172.21.15.98:0/2690350366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8108920 0x7feaa810ad40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 --2- 172.21.15.98:0/2690350366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feaa8103c20 0x7feaa8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 --2- 172.21.15.98:0/2690350366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feaa81029e0 0x7feaa81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 -- 172.21.15.98:0/2690350366 >> 172.21.15.98:0/2690350366 conn(0x7feaa80fdbd0 msgr2=0x7feaa8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:31.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 -- 172.21.15.98:0/2690350366 shutdown_connections 2024-04-25T16:29:31.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.578+0000 7feaaecf9640 1 -- 172.21.15.98:0/2690350366 wait complete. 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaecf9640 1 Processor -- start 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaecf9640 1 -- start start 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaecf9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feaa81029e0 0x7feaa810d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaecf9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8103c20 0x7feaa810b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaecf9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feaa8108920 0x7feaa810be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaecf9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7feaa810f520 con 0x7feaa8103c20 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaecf9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7feaa810f3a0 con 0x7feaa81029e0 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaecf9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7feaa810f6a0 con 0x7feaa8108920 2024-04-25T16:29:31.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8103c20 0x7feaa810b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8103c20 0x7feaa810b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44404/0 (socket says 172.21.15.98:44404) 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 -- 172.21.15.98:0/1781806515 learned_addr learned my addr 172.21.15.98:0/1781806515 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaaca95640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feaa81029e0 0x7feaa810d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaad296640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feaa8108920 0x7feaa810be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 -- 172.21.15.98:0/1781806515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feaa8108920 msgr2=0x7feaa810be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feaa8108920 0x7feaa810be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 -- 172.21.15.98:0/1781806515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feaa81029e0 msgr2=0x7feaa810d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feaa81029e0 0x7feaa810d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 -- 172.21.15.98:0/1781806515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa810c5c0 con 0x7feaa8103c20 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.579+0000 7feaa7fff640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8103c20 0x7feaa810b950 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00d950 tx=0x7fea9c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.580+0000 7feaa5ffb640 1 -- 172.21.15.98:0/1781806515 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c017070 con 0x7feaa8103c20 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.580+0000 7feaa5ffb640 1 -- 172.21.15.98:0/1781806515 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fea9c009c30 con 0x7feaa8103c20 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.580+0000 7feaa5ffb640 1 -- 172.21.15.98:0/1781806515 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c00f620 con 0x7feaa8103c20 2024-04-25T16:29:31.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.580+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa81a41d0 con 0x7feaa8103c20 2024-04-25T16:29:31.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.580+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa81a4690 con 0x7feaa8103c20 2024-04-25T16:29:31.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.581+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaa8107730 con 0x7feaa8103c20 2024-04-25T16:29:31.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.582+0000 7feaa5ffb640 1 -- 172.21.15.98:0/1781806515 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fea9c010460 con 0x7feaa8103c20 2024-04-25T16:29:31.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.582+0000 7feaa5ffb640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea900afaf0 0x7fea900b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.583+0000 7feaaca95640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea900afaf0 0x7fea900b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.583+0000 7feaa5ffb640 1 -- 172.21.15.98:0/1781806515 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fea9c0de280 con 0x7feaa8103c20 2024-04-25T16:29:31.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.583+0000 7feaaca95640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea900afaf0 0x7fea900b2000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fea98005fd0 tx=0x7fea9800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:31.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.587+0000 7feaa5ffb640 1 -- 172.21.15.98:0/1781806515 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea9c009de0 con 0x7feaa8103c20 2024-04-25T16:29:31.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.764+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feaa80637a0 con 0x7feaa8103c20 2024-04-25T16:29:31.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.765+0000 7feaa5ffb640 1 -- 172.21.15.98:0/1781806515 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18467 (secure 0 0 0) 0x7fea9c0a6cb0 con 0x7feaa8103c20 2024-04-25T16:29:31.765 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:31.766 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":73,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:29:21.989091+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea900afaf0 msgr2=0x7fea900b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea900afaf0 0x7fea900b2000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fea98005fd0 tx=0x7fea9800b410 comp rx=0 tx=0).stop 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8103c20 msgr2=0x7feaa810b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8103c20 0x7feaa810b950 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00d950 tx=0x7fea9c00de10 comp rx=0 tx=0).stop 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 shutdown_connections 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fea900afaf0 0x7fea900b2000 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7feaa8108920 0x7feaa810be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7feaa8103c20 0x7feaa810b950 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 --2- 172.21.15.98:0/1781806515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feaa81029e0 0x7feaa810d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 >> 172.21.15.98:0/1781806515 conn(0x7feaa80fdbd0 msgr2=0x7feaa8104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 shutdown_connections 2024-04-25T16:29:31.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.767+0000 7feaaecf9640 1 -- 172.21.15.98:0/1781806515 wait complete. 2024-04-25T16:29:31.779 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:29:31.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:29:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 -- 172.21.15.98:0/2834572663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60104480 msgr2=0x7f9a6010abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 --2- 172.21.15.98:0/2834572663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60104480 0x7f9a6010abd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9a58009ca0 tx=0x7f9a5802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 -- 172.21.15.98:0/2834572663 shutdown_connections 2024-04-25T16:29:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 --2- 172.21.15.98:0/2834572663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60104480 0x7f9a6010abd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 --2- 172.21.15.98:0/2834572663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a60103960 0x7f9a60103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 --2- 172.21.15.98:0/2834572663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a60102f80 0x7f9a60103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 -- 172.21.15.98:0/2834572663 >> 172.21.15.98:0/2834572663 conn(0x7f9a600fd380 msgr2=0x7f9a600ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 -- 172.21.15.98:0/2834572663 shutdown_connections 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 -- 172.21.15.98:0/2834572663 wait complete. 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 Processor -- start 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.890+0000 7f9a678f2640 1 -- start start 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a678f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60102f80 0x7f9a60199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a678f2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a60103960 0x7f9a60199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a678f2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a60104480 0x7f9a601a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a678f2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6010e450 con 0x7f9a60102f80 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a678f2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6010e2d0 con 0x7f9a60104480 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a678f2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a6010e5d0 con 0x7f9a60103960 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60102f80 0x7f9a60199000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60102f80 0x7f9a60199000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44414/0 (socket says 172.21.15.98:44414) 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 -- 172.21.15.98:0/4294301139 learned_addr learned my addr 172.21.15.98:0/4294301139 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a65e8f640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a60104480 0x7f9a601a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a64e8d640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a60103960 0x7f9a60199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 -- 172.21.15.98:0/4294301139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a60103960 msgr2=0x7f9a60199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a60103960 0x7f9a60199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 -- 172.21.15.98:0/4294301139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a60104480 msgr2=0x7f9a601a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a60104480 0x7f9a601a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 -- 172.21.15.98:0/4294301139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a601a0af0 con 0x7f9a60102f80 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a6568e640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60102f80 0x7f9a60199000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009860 tx=0x7f9a5c009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a567fc640 1 -- 172.21.15.98:0/4294301139 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c01d070 con 0x7f9a60102f80 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a567fc640 1 -- 172.21.15.98:0/4294301139 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a5c007c30 con 0x7f9a60102f80 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a60101810 con 0x7f9a60102f80 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a567fc640 1 -- 172.21.15.98:0/4294301139 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c018cb0 con 0x7f9a60102f80 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.891+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a60101d60 con 0x7f9a60102f80 2024-04-25T16:29:31.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.892+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a6006a160 con 0x7f9a60102f80 2024-04-25T16:29:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.894+0000 7f9a567fc640 1 -- 172.21.15.98:0/4294301139 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9a5c020070 con 0x7f9a60102f80 2024-04-25T16:29:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.895+0000 7f9a567fc640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a4c0afaf0 0x7f9a4c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.895+0000 7f9a64e8d640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a4c0afaf0 0x7f9a4c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:31.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.895+0000 7f9a567fc640 1 -- 172.21.15.98:0/4294301139 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f9a5c0dfd80 con 0x7f9a60102f80 2024-04-25T16:29:31.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.896+0000 7f9a64e8d640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a4c0afaf0 0x7f9a4c0b2000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a50005fd0 tx=0x7f9a50005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:31.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:31.899+0000 7f9a567fc640 1 -- 172.21.15.98:0/4294301139 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a5c007de0 con 0x7f9a60102f80 2024-04-25T16:29:32.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.084+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a601a0cc0 con 0x7f9a60102f80 2024-04-25T16:29:32.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.084+0000 7f9a567fc640 1 -- 172.21.15.98:0/4294301139 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f9a5c0a87b0 con 0x7f9a60102f80 2024-04-25T16:29:32.084 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:29:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a4c0afaf0 msgr2=0x7f9a4c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:32.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a4c0afaf0 0x7f9a4c0b2000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a50005fd0 tx=0x7f9a50005e20 comp rx=0 tx=0).stop 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60102f80 msgr2=0x7f9a60199000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60102f80 0x7f9a60199000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009860 tx=0x7f9a5c009d20 comp rx=0 tx=0).stop 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 shutdown_connections 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a4c0afaf0 0x7f9a4c0b2000 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a60104480 0x7f9a601a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a60103960 0x7f9a60199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 --2- 172.21.15.98:0/4294301139 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a60102f80 0x7f9a60199000 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 >> 172.21.15.98:0/4294301139 conn(0x7f9a600fd380 msgr2=0x7f9a60106b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 shutdown_connections 2024-04-25T16:29:32.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.086+0000 7f9a678f2640 1 -- 172.21.15.98:0/4294301139 wait complete. 2024-04-25T16:29:32.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:29:32.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-25T16:29:32.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:32.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:32.096 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:32.097 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:31 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1781806515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:32.098 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:32.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/550507710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54102a00 msgr2=0x7f2c54103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:32.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/550507710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54102a00 0x7f2c54103700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009900 tx=0x7f2c4402a410 comp rx=0 tx=0).stop 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/550507710 shutdown_connections 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/550507710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c54108940 0x7f2c5410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/550507710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c54103c40 0x7f2c54104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/550507710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54102a00 0x7f2c54103700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/550507710 >> 172.21.15.98:0/550507710 conn(0x7f2c540fdbf0 msgr2=0x7f2c541000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/550507710 shutdown_connections 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.205+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/550507710 wait complete. 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c5a6b1640 1 Processor -- start 2024-04-25T16:29:32.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c5a6b1640 1 -- start start 2024-04-25T16:29:32.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c5a6b1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c54102a00 0x7f2c54199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:32.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c5a6b1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c54103c40 0x7f2c54199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:32.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c5a6b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54108940 0x7f2c541a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:32.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c5a6b1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5410e3e0 con 0x7f2c54108940 2024-04-25T16:29:32.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c5a6b1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5410e260 con 0x7f2c54103c40 2024-04-25T16:29:32.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c5a6b1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2c5410e560 con 0x7f2c54102a00 2024-04-25T16:29:32.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c58c4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54108940 0x7f2c541a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c58c4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54108940 0x7f2c541a0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44434/0 (socket says 172.21.15.98:44434) 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c58c4e640 1 -- 172.21.15.98:0/1802363803 learned_addr learned my addr 172.21.15.98:0/1802363803 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c537fe640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c54103c40 0x7f2c54199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c53fff640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c54102a00 0x7f2c54199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c58c4e640 1 -- 172.21.15.98:0/1802363803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c54102a00 msgr2=0x7f2c54199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c58c4e640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c54102a00 0x7f2c54199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c58c4e640 1 -- 172.21.15.98:0/1802363803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c54103c40 msgr2=0x7f2c54199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c58c4e640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c54103c40 0x7f2c54199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.206+0000 7f2c58c4e640 1 -- 172.21.15.98:0/1802363803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c541a0b90 con 0x7f2c54108940 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.207+0000 7f2c58c4e640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54108940 0x7f2c541a0520 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00cad0 tx=0x7f2c4c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.207+0000 7f2c517fa640 1 -- 172.21.15.98:0/1802363803 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00ae20 con 0x7f2c54108940 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.207+0000 7f2c517fa640 1 -- 172.21.15.98:0/1802363803 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c4c005020 con 0x7f2c54108940 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.207+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c541a0e80 con 0x7f2c54108940 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.207+0000 7f2c517fa640 1 -- 172.21.15.98:0/1802363803 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00f6b0 con 0x7f2c54108940 2024-04-25T16:29:32.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.207+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c541a1290 con 0x7f2c54108940 2024-04-25T16:29:32.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.208+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c54107750 con 0x7f2c54108940 2024-04-25T16:29:32.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.209+0000 7f2c517fa640 1 -- 172.21.15.98:0/1802363803 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2c4c0051d0 con 0x7f2c54108940 2024-04-25T16:29:32.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.210+0000 7f2c517fa640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c3c0afaf0 0x7f2c3c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:32.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.210+0000 7f2c53fff640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c3c0afaf0 0x7f2c3c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:32.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.210+0000 7f2c517fa640 1 -- 172.21.15.98:0/1802363803 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f2c4c0dead0 con 0x7f2c54108940 2024-04-25T16:29:32.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.210+0000 7f2c53fff640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c3c0afaf0 0x7f2c3c0b2000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c4402a930 tx=0x7f2c44005720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:32.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.213+0000 7f2c517fa640 1 -- 172.21.15.98:0/1802363803 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c4c014030 con 0x7f2c54108940 2024-04-25T16:29:32.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:29:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-25T16:29:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:31 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1781806515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.429+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c5402db50 con 0x7f2c54108940 2024-04-25T16:29:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.430+0000 7f2c517fa640 1 -- 172.21.15.98:0/1802363803 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f2c4c0a7b50 con 0x7f2c54108940 2024-04-25T16:29:32.430 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:32.430 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T16:29:32.430 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 1 2024-04-25T16:29:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c3c0afaf0 msgr2=0x7f2c3c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c3c0afaf0 0x7f2c3c0b2000 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2c4402a930 tx=0x7f2c44005720 comp rx=0 tx=0).stop 2024-04-25T16:29:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54108940 msgr2=0x7f2c541a0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54108940 0x7f2c541a0520 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00cad0 tx=0x7f2c4c00cf90 comp rx=0 tx=0).stop 2024-04-25T16:29:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 shutdown_connections 2024-04-25T16:29:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c3c0afaf0 0x7f2c3c0b2000 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c54108940 0x7f2c541a0520 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c54103c40 0x7f2c54199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 --2- 172.21.15.98:0/1802363803 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c54102a00 0x7f2c54199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 >> 172.21.15.98:0/1802363803 conn(0x7f2c540fdbf0 msgr2=0x7f2c54104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 shutdown_connections 2024-04-25T16:29:32.433 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:32.432+0000 7f2c5a6b1640 1 -- 172.21.15.98:0/1802363803 wait complete. 2024-04-25T16:29:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:29:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: Saving service mds.all spec with placement smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-25T16:29:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:29:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:32.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:32.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:32.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:31 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1781806515' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:29:33.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:32 smithi098 ceph-mon[29162]: Deploying daemon mds.l on smithi203 2024-04-25T16:29:33.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:32 smithi098 ceph-mon[29162]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:33.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:32 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4294301139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:29:33.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:32 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1802363803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:33.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:32 smithi173 ceph-mon[32556]: Deploying daemon mds.l on smithi203 2024-04-25T16:29:33.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:32 smithi173 ceph-mon[32556]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:33.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:32 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4294301139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:29:33.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:32 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1802363803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:33.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:32 smithi203 ceph-mon[33115]: Deploying daemon mds.l on smithi203 2024-04-25T16:29:33.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:32 smithi203 ceph-mon[33115]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:33.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:32 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4294301139' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:29:33.400 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:32 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1802363803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:33.400 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 systemd[1]: Starting Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:29:33.676 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 podman[63545]: 2024-04-25 16:29:33.303712787 +0000 UTC m=+0.018863325 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:29:33.676 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 podman[63545]: 2024-04-25 16:29:33.41168822 +0000 UTC m=+0.126838731 container create eb5325b0a0ffa330c1a4c214ade5b9b0cb728e2d5d6447bfc48cd958fc4d6043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:29:33.676 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 podman[63545]: 2024-04-25 16:29:33.639818062 +0000 UTC m=+0.354968584 container init eb5325b0a0ffa330c1a4c214ade5b9b0cb728e2d5d6447bfc48cd958fc4d6043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T16:29:33.676 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 podman[63545]: 2024-04-25 16:29:33.642362786 +0000 UTC m=+0.357513306 container start eb5325b0a0ffa330c1a4c214ade5b9b0cb728e2d5d6447bfc48cd958fc4d6043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T16:29:33.676 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 ceph-mds[63564]: -- 172.21.15.203:0/1243750524 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56457c0dba40 con 0x56457c123a80 2024-04-25T16:29:33.938 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: starting mds.l at 2024-04-25T16:29:33.938 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 bash[63545]: eb5325b0a0ffa330c1a4c214ade5b9b0cb728e2d5d6447bfc48cd958fc4d6043 2024-04-25T16:29:33.939 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:29:33 smithi203 systemd[1]: Started Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:34.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:34.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 -- 172.21.15.98:0/4092154790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78102040 msgr2=0x7fdb78102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 --2- 172.21.15.98:0/4092154790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78102040 0x7fdb78102450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009900 tx=0x7fdb6802a400 comp rx=0 tx=0).stop 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 -- 172.21.15.98:0/4092154790 shutdown_connections 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 --2- 172.21.15.98:0/4092154790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb78103580 0x7fdb78103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 --2- 172.21.15.98:0/4092154790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb78102a20 0x7fdb78102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 --2- 172.21.15.98:0/4092154790 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78102040 0x7fdb78102450 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 -- 172.21.15.98:0/4092154790 >> 172.21.15.98:0/4092154790 conn(0x7fdb780fdbf0 msgr2=0x7fdb781000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 -- 172.21.15.98:0/4092154790 shutdown_connections 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.564+0000 7fdb7d084640 1 -- 172.21.15.98:0/4092154790 wait complete. 2024-04-25T16:29:34.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.565+0000 7fdb7d084640 1 Processor -- start 2024-04-25T16:29:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.565+0000 7fdb7d084640 1 -- start start 2024-04-25T16:29:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.565+0000 7fdb7d084640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb78102040 0x7fdb7819c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.565+0000 7fdb7d084640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb78102a20 0x7fdb7819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.565+0000 7fdb7d084640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78103580 0x7fdb781a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.565+0000 7fdb7d084640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdb78111680 con 0x7fdb78103580 2024-04-25T16:29:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.565+0000 7fdb7d084640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdb78111500 con 0x7fdb78102a20 2024-04-25T16:29:34.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.565+0000 7fdb7d084640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdb78111800 con 0x7fdb78102040 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78103580 0x7fdb781a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78103580 0x7fdb781a37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44448/0 (socket says 172.21.15.98:44448) 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 -- 172.21.15.98:0/650864874 learned_addr learned my addr 172.21.15.98:0/650864874 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7659c640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb78102a20 0x7fdb7819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb76d9d640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb78102040 0x7fdb7819c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 -- 172.21.15.98:0/650864874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb78102040 msgr2=0x7fdb7819c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb78102040 0x7fdb7819c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 -- 172.21.15.98:0/650864874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb78102a20 msgr2=0x7fdb7819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb78102a20 0x7fdb7819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 -- 172.21.15.98:0/650864874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb781a3da0 con 0x7fdb78103580 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7759e640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78103580 0x7fdb781a37e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000cce0 tx=0x7fdb7000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb5ffff640 1 -- 172.21.15.98:0/650864874 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7000f450 con 0x7fdb78103580 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb5ffff640 1 -- 172.21.15.98:0/650864874 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdb7000fa80 con 0x7fdb78103580 2024-04-25T16:29:34.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb5ffff640 1 -- 172.21.15.98:0/650864874 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb70018ad0 con 0x7fdb78103580 2024-04-25T16:29:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb781a4090 con 0x7fdb78103580 2024-04-25T16:29:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.566+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb781a4590 con 0x7fdb78103580 2024-04-25T16:29:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.567+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb7810d4a0 con 0x7fdb78103580 2024-04-25T16:29:34.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.569+0000 7fdb5ffff640 1 -- 172.21.15.98:0/650864874 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fdb7000f5f0 con 0x7fdb78103580 2024-04-25T16:29:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.570+0000 7fdb5ffff640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdb600afaf0 0x7fdb600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.570+0000 7fdb5ffff640 1 -- 172.21.15.98:0/650864874 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fdb700defe0 con 0x7fdb78103580 2024-04-25T16:29:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.570+0000 7fdb76d9d640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdb600afaf0 0x7fdb600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:34.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.571+0000 7fdb76d9d640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdb600afaf0 0x7fdb600b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdb68000c00 tx=0x7fdb68005720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:34.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.574+0000 7fdb5ffff640 1 -- 172.21.15.98:0/650864874 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdb70014030 con 0x7fdb78103580 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:boot 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: fsmap 1 up:standby 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: Deploying daemon mds.k on smithi173 2024-04-25T16:29:34.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:34 smithi098 ceph-mon[29162]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:34.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:boot 2024-04-25T16:29:34.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: fsmap 1 up:standby 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: Deploying daemon mds.k on smithi173 2024-04-25T16:29:34.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:34 smithi173 ceph-mon[32556]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:34.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.789+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb7802db50 con 0x7fdb78103580 2024-04-25T16:29:34.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.790+0000 7fdb5ffff640 1 -- 172.21.15.98:0/650864874 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fdb700a80f0 con 0x7fdb78103580 2024-04-25T16:29:34.790 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:34.790 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T16:29:34.790 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 2 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdb600afaf0 msgr2=0x7fdb600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdb600afaf0 0x7fdb600b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fdb68000c00 tx=0x7fdb68005720 comp rx=0 tx=0).stop 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78103580 msgr2=0x7fdb781a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78103580 0x7fdb781a37e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000cce0 tx=0x7fdb7000a620 comp rx=0 tx=0).stop 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 shutdown_connections 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdb600afaf0 0x7fdb600b2000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdb78103580 0x7fdb781a37e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb78102a20 0x7fdb7819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 --2- 172.21.15.98:0/650864874 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb78102040 0x7fdb7819c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:34.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 >> 172.21.15.98:0/650864874 conn(0x7fdb780fdbf0 msgr2=0x7fdb7810a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:34.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 shutdown_connections 2024-04-25T16:29:34.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:34.792+0000 7fdb7d084640 1 -- 172.21.15.98:0/650864874 wait complete. 2024-04-25T16:29:34.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:boot 2024-04-25T16:29:34.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: fsmap 1 up:standby 2024-04-25T16:29:34.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:29:34.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:34.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:34.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:34.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:34.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: Deploying daemon mds.k on smithi173 2024-04-25T16:29:34.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:34 smithi203 ceph-mon[33115]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:35.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:35 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/650864874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:35.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:35 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/650864874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:36.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:35 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/650864874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:36.694 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:29:36 smithi173 systemd[1]: Started Ceph mds.k for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:36.804 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 -- 172.21.15.98:0/1277599414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a01029c0 msgr2=0x7f13a01036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:36.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 --2- 172.21.15.98:0/1277599414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a01029c0 0x7f13a01036c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f139c009900 tx=0x7f139c02a410 comp rx=0 tx=0).stop 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 -- 172.21.15.98:0/1277599414 shutdown_connections 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 --2- 172.21.15.98:0/1277599414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a0108900 0x7f13a010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 --2- 172.21.15.98:0/1277599414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13a0103c00 0x7f13a0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 --2- 172.21.15.98:0/1277599414 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a01029c0 0x7f13a01036c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 -- 172.21.15.98:0/1277599414 >> 172.21.15.98:0/1277599414 conn(0x7f13a00fdbd0 msgr2=0x7f13a0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 -- 172.21.15.98:0/1277599414 shutdown_connections 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 -- 172.21.15.98:0/1277599414 wait complete. 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 Processor -- start 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.916+0000 7f13a732d640 1 -- start start 2024-04-25T16:29:36.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a732d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13a01029c0 0x7f13a006a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a732d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a0103c00 0x7f13a019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a732d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a0108900 0x7f13a01a1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a732d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f13a010e370 con 0x7f13a0103c00 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a732d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f13a010e1f0 con 0x7f13a0108900 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a732d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f13a010e4f0 con 0x7f13a01029c0 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a0103c00 0x7f13a019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a0103c00 0x7f13a019eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38658/0 (socket says 172.21.15.98:38658) 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 -- 172.21.15.98:0/634801375 learned_addr learned my addr 172.21.15.98:0/634801375 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a50c9640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13a01029c0 0x7f13a006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a58ca640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a0108900 0x7f13a01a1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 -- 172.21.15.98:0/634801375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13a01029c0 msgr2=0x7f13a006a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13a01029c0 0x7f13a006a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:36.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 -- 172.21.15.98:0/634801375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a0108900 msgr2=0x7f13a01a1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a0108900 0x7f13a01a1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 -- 172.21.15.98:0/634801375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a01a1c20 con 0x7f13a0103c00 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a50c9640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13a01029c0 0x7f13a006a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a58ca640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a0108900 0x7f13a01a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a48c8640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a0103c00 0x7f13a019eee0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f139000c970 tx=0x7f139000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13967fc640 1 -- 172.21.15.98:0/634801375 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390007bb0 con 0x7f13a0103c00 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13967fc640 1 -- 172.21.15.98:0/634801375 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1390007d50 con 0x7f13a0103c00 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a01a1f10 con 0x7f13a0103c00 2024-04-25T16:29:36.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13967fc640 1 -- 172.21.15.98:0/634801375 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1390004e40 con 0x7f13a0103c00 2024-04-25T16:29:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.917+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a01a2380 con 0x7f13a0103c00 2024-04-25T16:29:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.918+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a0107710 con 0x7f13a0103c00 2024-04-25T16:29:36.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.920+0000 7f13967fc640 1 -- 172.21.15.98:0/634801375 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1390005000 con 0x7f13a0103c00 2024-04-25T16:29:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.920+0000 7f13967fc640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f138c0afaf0 0x7f138c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.921+0000 7f13a50c9640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f138c0afaf0 0x7f138c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.921+0000 7f13967fc640 1 -- 172.21.15.98:0/634801375 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f13900de990 con 0x7f13a0103c00 2024-04-25T16:29:36.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.921+0000 7f13a50c9640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f138c0afaf0 0x7f138c0b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f139c005f70 tx=0x7f139c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:36.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:36.924+0000 7f13967fc640 1 -- 172.21.15.98:0/634801375 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1390014030 con 0x7f13a0103c00 2024-04-25T16:29:37.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:36 smithi098 ceph-mon[29162]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:37.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:36 smithi173 ceph-mon[32556]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:37.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.147+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13a002dda0 con 0x7f13a0103c00 2024-04-25T16:29:37.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.148+0000 7f13967fc640 1 -- 172.21.15.98:0/634801375 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f13900a7a10 con 0x7f13a0103c00 2024-04-25T16:29:37.148 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:37.149 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T16:29:37.149 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-04-25T16:29:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f138c0afaf0 msgr2=0x7f138c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f138c0afaf0 0x7f138c0b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f139c005f70 tx=0x7f139c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:29:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a0103c00 msgr2=0x7f13a019eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a0103c00 0x7f13a019eee0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f139000c970 tx=0x7f139000ce30 comp rx=0 tx=0).stop 2024-04-25T16:29:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 shutdown_connections 2024-04-25T16:29:37.150 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f138c0afaf0 0x7f138c0b2000 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a0108900 0x7f13a01a1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f13a0103c00 0x7f13a019eee0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 --2- 172.21.15.98:0/634801375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f13a01029c0 0x7f13a006a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 >> 172.21.15.98:0/634801375 conn(0x7f13a00fdbd0 msgr2=0x7f13a01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 shutdown_connections 2024-04-25T16:29:37.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:37.150+0000 7f13a732d640 1 -- 172.21.15.98:0/634801375 wait complete. 2024-04-25T16:29:37.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:36 smithi203 ceph-mon[33115]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:boot 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: fsmap 2 up:standby 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:37.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:37.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: Deploying daemon mds.j on smithi098 2024-04-25T16:29:37.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:37 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/634801375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:boot 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: fsmap 2 up:standby 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: Deploying daemon mds.j on smithi098 2024-04-25T16:29:38.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:37 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/634801375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:38.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:boot 2024-04-25T16:29:38.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: fsmap 2 up:standby 2024-04-25T16:29:38.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:29:38.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:38.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:38.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:38.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:38.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:38.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:38.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: Deploying daemon mds.j on smithi098 2024-04-25T16:29:38.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:37 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/634801375' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:38.979 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:38 smithi173 ceph-mon[32556]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:39.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:38 smithi098 ceph-mon[29162]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:39.003 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:38 smithi098 systemd[1]: Starting Ceph mds.j for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:29:39.162 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:39.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:38 smithi203 ceph-mon[33115]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/3202309776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 msgr2=0x7ff09c106b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/3202309776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 0x7ff09c106b20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff090009900 tx=0x7ff09002a410 comp rx=0 tx=0).stop 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/3202309776 shutdown_connections 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/3202309776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff09c109780 0x7ff09c10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/3202309776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c102360 0x7ff09c1090f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/3202309776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 0x7ff09c106b20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/3202309776 >> 172.21.15.98:0/3202309776 conn(0x7ff09c0fd370 msgr2=0x7ff09c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/3202309776 shutdown_connections 2024-04-25T16:29:39.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/3202309776 wait complete. 2024-04-25T16:29:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 Processor -- start 2024-04-25T16:29:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.278+0000 7ff0a2b9e640 1 -- start start 2024-04-25T16:29:39.280 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a2b9e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 0x7ff09c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a2b9e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff09c102360 0x7ff09c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a2b9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c109780 0x7ff09c1a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a2b9e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c10f490 con 0x7ff09c101e60 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a2b9e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c10f310 con 0x7ff09c102360 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a2b9e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff09c10f610 con 0x7ff09c109780 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 0x7ff09c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 0x7ff09c19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38678/0 (socket says 172.21.15.98:38678) 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 -- 172.21.15.98:0/1709329807 learned_addr learned my addr 172.21.15.98:0/1709329807 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff09bfff640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff09c102360 0x7ff09c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a113b640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c109780 0x7ff09c1a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 -- 172.21.15.98:0/1709329807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c109780 msgr2=0x7ff09c1a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c109780 0x7ff09c1a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 -- 172.21.15.98:0/1709329807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff09c102360 msgr2=0x7ff09c19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff09c102360 0x7ff09c19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 -- 172.21.15.98:0/1709329807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff09c1a1c40 con 0x7ff09c101e60 2024-04-25T16:29:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a093a640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 0x7ff09c19a150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff090009230 tx=0x7ff0900040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff099ffb640 1 -- 172.21.15.98:0/1709329807 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09003f030 con 0x7ff09c101e60 2024-04-25T16:29:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff09c1a1ed0 con 0x7ff09c101e60 2024-04-25T16:29:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff099ffb640 1 -- 172.21.15.98:0/1709329807 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff0900044c0 con 0x7ff09c101e60 2024-04-25T16:29:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.279+0000 7ff099ffb640 1 -- 172.21.15.98:0/1709329807 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090035070 con 0x7ff09c101e60 2024-04-25T16:29:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.280+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff09c1a22e0 con 0x7ff09c101e60 2024-04-25T16:29:39.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.281+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff09c0fee90 con 0x7ff09c101e60 2024-04-25T16:29:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.282+0000 7ff099ffb640 1 -- 172.21.15.98:0/1709329807 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff090042030 con 0x7ff09c101e60 2024-04-25T16:29:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.283+0000 7ff099ffb640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff08c0afaf0 0x7ff08c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.283+0000 7ff09bfff640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff08c0afaf0 0x7ff08c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.283+0000 7ff099ffb640 1 -- 172.21.15.98:0/1709329807 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7ff0900feca0 con 0x7ff09c101e60 2024-04-25T16:29:39.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.283+0000 7ff09bfff640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff08c0afaf0 0x7ff08c0b2000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff09c19b1a0 tx=0x7ff08800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:39.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.287+0000 7ff099ffb640 1 -- 172.21.15.98:0/1709329807 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0900c7d00 con 0x7ff09c101e60 2024-04-25T16:29:39.362 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:39 smithi098 podman[69401]: 2024-04-25 16:29:38.98164278 +0000 UTC m=+0.017807579 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:29:39.362 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:39 smithi098 podman[69401]: 2024-04-25 16:29:39.118805521 +0000 UTC m=+0.154970303 container create 62399e7e22529553536415a18e45342142a13d55463bdb1783ab104af0dc64d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True) 2024-04-25T16:29:39.363 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:39 smithi098 podman[69401]: 2024-04-25 16:29:39.363367737 +0000 UTC m=+0.399532521 container init 62399e7e22529553536415a18e45342142a13d55463bdb1783ab104af0dc64d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T16:29:39.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.512+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff09c02dd70 con 0x7ff09c101e60 2024-04-25T16:29:39.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.513+0000 7ff099ffb640 1 -- 172.21.15.98:0/1709329807 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7ff0900c7600 con 0x7ff09c101e60 2024-04-25T16:29:39.513 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:39.513 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T16:29:39.513 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 3 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff08c0afaf0 msgr2=0x7ff08c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff08c0afaf0 0x7ff08c0b2000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff09c19b1a0 tx=0x7ff08800b390 comp rx=0 tx=0).stop 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 msgr2=0x7ff09c19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 0x7ff09c19a150 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff090009230 tx=0x7ff0900040c0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 shutdown_connections 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff08c0afaf0 0x7ff08c0b2000 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff09c109780 0x7ff09c1a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff09c102360 0x7ff09c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 --2- 172.21.15.98:0/1709329807 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff09c101e60 0x7ff09c19a150 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 >> 172.21.15.98:0/1709329807 conn(0x7ff09c0fd370 msgr2=0x7ff09c106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 shutdown_connections 2024-04-25T16:29:39.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:39.515+0000 7ff0a2b9e640 1 -- 172.21.15.98:0/1709329807 wait complete. 2024-04-25T16:29:39.708 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:39 smithi098 podman[69401]: 2024-04-25 16:29:39.366283029 +0000 UTC m=+0.402447813 container start 62399e7e22529553536415a18e45342142a13d55463bdb1783ab104af0dc64d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-25T16:29:39.708 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-mds[69459]: -- 172.21.15.98:0/3254484147 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5642bd7bda40 con 0x5642bd805600 2024-04-25T16:29:39.708 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j[69436]: starting mds.j at 2024-04-25T16:29:39.708 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:39 smithi098 bash[69401]: 62399e7e22529553536415a18e45342142a13d55463bdb1783ab104af0dc64d5 2024-04-25T16:29:39.708 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:29:39 smithi098 systemd[1]: Started Ceph mds.j for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:40.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1709329807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:40.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:40.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:40.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:39 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:40.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:39 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1709329807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:40.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:39 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:39 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:39 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:39 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:40.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:39 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:40.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:39 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:40.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:39 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1709329807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:40.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:39 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:39 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:39 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:40.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:39 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:40.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:39 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:40.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:39 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:41.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:40 smithi098 ceph-mon[29162]: Deploying daemon mds.d on smithi098 2024-04-25T16:29:41.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:40 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6834/441838145,v1:172.21.15.98:6835/441838145] up:boot 2024-04-25T16:29:41.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:40 smithi098 ceph-mon[29162]: fsmap 3 up:standby 2024-04-25T16:29:41.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:29:41.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:40 smithi098 ceph-mon[29162]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:41.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:40 smithi173 ceph-mon[32556]: Deploying daemon mds.d on smithi098 2024-04-25T16:29:41.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:40 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6834/441838145,v1:172.21.15.98:6835/441838145] up:boot 2024-04-25T16:29:41.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:40 smithi173 ceph-mon[32556]: fsmap 3 up:standby 2024-04-25T16:29:41.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:29:41.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:40 smithi173 ceph-mon[32556]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:41.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:40 smithi203 ceph-mon[33115]: Deploying daemon mds.d on smithi098 2024-04-25T16:29:41.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:40 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6834/441838145,v1:172.21.15.98:6835/441838145] up:boot 2024-04-25T16:29:41.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:40 smithi203 ceph-mon[33115]: fsmap 3 up:standby 2024-04-25T16:29:41.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T16:29:41.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:40 smithi203 ceph-mon[33115]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:41.530 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.775+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/815968042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8101730 msgr2=0x7ff8b8101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.775+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/815968042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8101730 0x7ff8b8101ba0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009900 tx=0x7ff8ac02a410 comp rx=0 tx=0).stop 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.775+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/815968042 shutdown_connections 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.775+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/815968042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8102230 0x7ff8b810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.775+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/815968042 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8101730 0x7ff8b8101ba0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.775+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/815968042 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105080 0x7ff8b8105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.776+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/815968042 >> 172.21.15.98:0/815968042 conn(0x7ff8b80fd330 msgr2=0x7ff8b80ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.776+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/815968042 shutdown_connections 2024-04-25T16:29:41.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.776+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/815968042 wait complete. 2024-04-25T16:29:41.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.776+0000 7ff8bd3f3640 1 Processor -- start 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.776+0000 7ff8bd3f3640 1 -- start start 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8bd3f3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8101730 0x7ff8b819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8bd3f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8102230 0x7ff8b819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8bd3f3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105080 0x7ff8b81a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8bd3f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b810f4a0 con 0x7ff8b8102230 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8bd3f3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b810f320 con 0x7ff8b8101730 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8bd3f3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b810f620 con 0x7ff8b8105080 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8102230 0x7ff8b819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8102230 0x7ff8b819a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38704/0 (socket says 172.21.15.98:38704) 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 -- 172.21.15.98:0/3742266152 learned_addr learned my addr 172.21.15.98:0/3742266152 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b77fe640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105080 0x7ff8b81a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:41.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b6ffd640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8101730 0x7ff8b819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 -- 172.21.15.98:0/3742266152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105080 msgr2=0x7ff8b81a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105080 0x7ff8b81a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 -- 172.21.15.98:0/3742266152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8101730 msgr2=0x7ff8b819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8101730 0x7ff8b819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 -- 172.21.15.98:0/3742266152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b81a1c70 con 0x7ff8b8102230 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b6ffd640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8101730 0x7ff8b819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff8b67fc640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8102230 0x7ff8b819a670 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009230 tx=0x7ff8ac0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.777+0000 7ff89ffff640 1 -- 172.21.15.98:0/3742266152 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac03f030 con 0x7ff8b8102230 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.778+0000 7ff89ffff640 1 -- 172.21.15.98:0/3742266152 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff8ac003c90 con 0x7ff8b8102230 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.778+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b81a1f00 con 0x7ff8b8102230 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.778+0000 7ff89ffff640 1 -- 172.21.15.98:0/3742266152 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac035070 con 0x7ff8b8102230 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.778+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b81a2310 con 0x7ff8b8102230 2024-04-25T16:29:41.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.779+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b8108a10 con 0x7ff8b8102230 2024-04-25T16:29:41.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.780+0000 7ff89ffff640 1 -- 172.21.15.98:0/3742266152 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff8ac042030 con 0x7ff8b8102230 2024-04-25T16:29:41.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.781+0000 7ff89ffff640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff8a00afaf0 0x7ff8a00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:41.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.781+0000 7ff8b6ffd640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff8a00afaf0 0x7ff8a00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:41.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.781+0000 7ff89ffff640 1 -- 172.21.15.98:0/3742266152 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7ff8ac0fed50 con 0x7ff8b8102230 2024-04-25T16:29:41.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.781+0000 7ff8b6ffd640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff8a00afaf0 0x7ff8a00b2000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80058c0 tx=0x7ff8a8005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:41.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:41.787+0000 7ff89ffff640 1 -- 172.21.15.98:0/3742266152 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8ac102030 con 0x7ff8b8102230 2024-04-25T16:29:41.925 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:41 smithi098 systemd[1]: Starting Ceph mds.d for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:29:42.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.011+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8b802dd70 con 0x7ff8b8102230 2024-04-25T16:29:42.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.012+0000 7ff89ffff640 1 -- 172.21.15.98:0/3742266152 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7ff8ac0c7dd0 con 0x7ff8b8102230 2024-04-25T16:29:42.012 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:42.012 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T16:29:42.012 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 4 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff8a00afaf0 msgr2=0x7ff8a00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff8a00afaf0 0x7ff8a00b2000 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff8a80058c0 tx=0x7ff8a8005850 comp rx=0 tx=0).stop 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8102230 msgr2=0x7ff8b819a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8102230 0x7ff8b819a670 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009230 tx=0x7ff8ac0042f0 comp rx=0 tx=0).stop 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 shutdown_connections 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff8a00afaf0 0x7ff8a00b2000 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff8b8105080 0x7ff8b81a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff8b8102230 0x7ff8b819a670 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 --2- 172.21.15.98:0/3742266152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8b8101730 0x7ff8b819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 >> 172.21.15.98:0/3742266152 conn(0x7ff8b80fd330 msgr2=0x7ff8b8105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 shutdown_connections 2024-04-25T16:29:42.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:42.014+0000 7ff8bd3f3640 1 -- 172.21.15.98:0/3742266152 wait complete. 2024-04-25T16:29:42.198 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:41 smithi098 podman[69786]: 2024-04-25 16:29:41.925861726 +0000 UTC m=+0.113437575 container create c6012484385a1a80826e7290b96209abe085c6ecda5bc0bfd1d25634dc8f5c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS) 2024-04-25T16:29:42.198 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:41 smithi098 podman[69786]: 2024-04-25 16:29:41.831231048 +0000 UTC m=+0.018806895 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:29:42.198 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:42 smithi098 podman[69786]: 2024-04-25 16:29:42.162589245 +0000 UTC m=+0.350165093 container init c6012484385a1a80826e7290b96209abe085c6ecda5bc0bfd1d25634dc8f5c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T16:29:42.198 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:42 smithi098 podman[69786]: 2024-04-25 16:29:42.165106494 +0000 UTC m=+0.352682349 container start c6012484385a1a80826e7290b96209abe085c6ecda5bc0bfd1d25634dc8f5c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True) 2024-04-25T16:29:42.502 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mds[69807]: -- 172.21.15.98:0/4225548677 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55646a099a40 con 0x55646a0e1600 2024-04-25T16:29:42.503 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d[69803]: starting mds.d at 2024-04-25T16:29:42.503 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:42 smithi098 bash[69786]: c6012484385a1a80826e7290b96209abe085c6ecda5bc0bfd1d25634dc8f5c96 2024-04-25T16:29:42.503 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:29:42 smithi098 systemd[1]: Started Ceph mds.d for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:43.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:42 smithi173 ceph-mon[32556]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:43.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:42 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3742266152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:43.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:43.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:43.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:42 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mon[29162]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3742266152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:43.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:42 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:43.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:42 smithi203 ceph-mon[33115]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:43.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:42 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3742266152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:43.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:43.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:43.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:43.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:42 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:44.025 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:44.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2299979643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c0760a0 msgr2=0x7f7e7c0764b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2299979643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c0760a0 0x7f7e7c0764b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7e78009900 tx=0x7f7e7802a410 comp rx=0 tx=0).stop 2024-04-25T16:29:44.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2299979643 shutdown_connections 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2299979643 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c110260 0x7f7e7c112680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2299979643 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e7c0769f0 0x7f7e7c10fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2299979643 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c0760a0 0x7f7e7c0764b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2299979643 >> 172.21.15.98:0/2299979643 conn(0x7f7e7c0fed10 msgr2=0x7f7e7c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2299979643 shutdown_connections 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2299979643 wait complete. 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 Processor -- start 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.149+0000 7f7e83f7d640 1 -- start start 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e83f7d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c0760a0 0x7f7e7c1a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e83f7d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e7c0769f0 0x7f7e7c1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e83f7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c110260 0x7f7e7c1a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e83f7d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c115af0 con 0x7f7e7c110260 2024-04-25T16:29:44.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e83f7d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c115970 con 0x7f7e7c0769f0 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e83f7d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7c115c70 con 0x7f7e7c0760a0 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c110260 0x7f7e7c1a7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c110260 0x7f7e7c1a7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38732/0 (socket says 172.21.15.98:38732) 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 -- 172.21.15.98:0/2413274771 learned_addr learned my addr 172.21.15.98:0/2413274771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e81d19640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c0760a0 0x7f7e7c1a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e81518640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e7c0769f0 0x7f7e7c1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 -- 172.21.15.98:0/2413274771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c0760a0 msgr2=0x7f7e7c1a0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c0760a0 0x7f7e7c1a0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 -- 172.21.15.98:0/2413274771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e7c0769f0 msgr2=0x7f7e7c1a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e7c0769f0 0x7f7e7c1a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 -- 172.21.15.98:0/2413274771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7c1a8280 con 0x7f7e7c110260 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e81d19640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c0760a0 0x7f7e7c1a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:29:44.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e81518640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e7c0769f0 0x7f7e7c1a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:44.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e8251a640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c110260 0x7f7e7c1a7c10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400eb40 tx=0x7f7e7400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:44.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e72ffd640 1 -- 172.21.15.98:0/2413274771 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74004b20 con 0x7f7e7c110260 2024-04-25T16:29:44.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e72ffd640 1 -- 172.21.15.98:0/2413274771 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7e74004cc0 con 0x7f7e7c110260 2024-04-25T16:29:44.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7c1a8570 con 0x7f7e7c110260 2024-04-25T16:29:44.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e72ffd640 1 -- 172.21.15.98:0/2413274771 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e74010690 con 0x7f7e7c110260 2024-04-25T16:29:44.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.150+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7c1a8ac0 con 0x7f7e7c110260 2024-04-25T16:29:44.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.151+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e7c105ef0 con 0x7f7e7c110260 2024-04-25T16:29:44.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.153+0000 7f7e72ffd640 1 -- 172.21.15.98:0/2413274771 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f7e74005540 con 0x7f7e7c110260 2024-04-25T16:29:44.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.153+0000 7f7e72ffd640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7e680afaf0 0x7f7e680b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:44.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.153+0000 7f7e81d19640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7e680afaf0 0x7f7e680b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:44.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.153+0000 7f7e72ffd640 1 -- 172.21.15.98:0/2413274771 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f7e740df090 con 0x7f7e7c110260 2024-04-25T16:29:44.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.154+0000 7f7e81d19640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7e680afaf0 0x7f7e680b2000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e78006010 tx=0x7f7e78000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:44.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.157+0000 7f7e72ffd640 1 -- 172.21.15.98:0/2413274771 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7e740a81a0 con 0x7f7e7c110260 2024-04-25T16:29:44.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:43 smithi098 ceph-mon[29162]: Deploying daemon mds.e on smithi173 2024-04-25T16:29:44.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:43 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6836/1001942637,v1:172.21.15.98:6837/1001942637] up:boot 2024-04-25T16:29:44.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:43 smithi098 ceph-mon[29162]: fsmap 4 up:standby 2024-04-25T16:29:44.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:43 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:29:44.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:43 smithi173 ceph-mon[32556]: Deploying daemon mds.e on smithi173 2024-04-25T16:29:44.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:43 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6836/1001942637,v1:172.21.15.98:6837/1001942637] up:boot 2024-04-25T16:29:44.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:43 smithi173 ceph-mon[32556]: fsmap 4 up:standby 2024-04-25T16:29:44.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:43 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:29:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.376+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e7c02db50 con 0x7f7e7c110260 2024-04-25T16:29:44.376 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.376+0000 7f7e72ffd640 1 -- 172.21.15.98:0/2413274771 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f7e740a7aa0 con 0x7f7e7c110260 2024-04-25T16:29:44.376 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:44.377 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T16:29:44.377 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 5 2024-04-25T16:29:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7e680afaf0 msgr2=0x7f7e680b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7e680afaf0 0x7f7e680b2000 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7e78006010 tx=0x7f7e78000bc0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c110260 msgr2=0x7f7e7c1a7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c110260 0x7f7e7c1a7c10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7e7400eb40 tx=0x7f7e7400c2d0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 shutdown_connections 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7e680afaf0 0x7f7e680b2000 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7e7c110260 0x7f7e7c1a7c10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7e7c0769f0 0x7f7e7c1a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 --2- 172.21.15.98:0/2413274771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e7c0760a0 0x7f7e7c1a0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 >> 172.21.15.98:0/2413274771 conn(0x7f7e7c0fed10 msgr2=0x7f7e7c1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 shutdown_connections 2024-04-25T16:29:44.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:44.378+0000 7f7e83f7d640 1 -- 172.21.15.98:0/2413274771 wait complete. 2024-04-25T16:29:44.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:43 smithi203 ceph-mon[33115]: Deploying daemon mds.e on smithi173 2024-04-25T16:29:44.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:43 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6836/1001942637,v1:172.21.15.98:6837/1001942637] up:boot 2024-04-25T16:29:44.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:43 smithi203 ceph-mon[33115]: fsmap 4 up:standby 2024-04-25T16:29:44.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:43 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T16:29:44.696 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:44 smithi173 systemd[1]: Starting Ceph mds.e for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:29:44.977 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:44 smithi173 podman[62161]: 2024-04-25 16:29:44.697215744 +0000 UTC m=+0.114993103 container create ac8af86570cca7515d05eefd46407d8c939dbf17d27f5c8144812f2a153c8350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:29:44.978 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:44 smithi173 podman[62161]: 2024-04-25 16:29:44.600730437 +0000 UTC m=+0.018507809 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:29:44.978 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:44 smithi173 podman[62161]: 2024-04-25 16:29:44.941611852 +0000 UTC m=+0.359389211 container init ac8af86570cca7515d05eefd46407d8c939dbf17d27f5c8144812f2a153c8350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T16:29:44.978 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:44 smithi173 podman[62161]: 2024-04-25 16:29:44.944281071 +0000 UTC m=+0.362058437 container start ac8af86570cca7515d05eefd46407d8c939dbf17d27f5c8144812f2a153c8350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T16:29:44.978 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:44 smithi173 ceph-mds[62180]: -- 172.21.15.173:0/74197925 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5594e4e35a40 con 0x5594e4e7d600 2024-04-25T16:29:45.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:44 smithi098 ceph-mon[29162]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:45.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:44 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2413274771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:45.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:44 smithi173 ceph-mon[32556]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:45.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:44 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2413274771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:45.265 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:44 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e[62176]: starting mds.e at 2024-04-25T16:29:45.265 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:45 smithi173 bash[62161]: ac8af86570cca7515d05eefd46407d8c939dbf17d27f5c8144812f2a153c8350 2024-04-25T16:29:45.265 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:29:45 smithi173 systemd[1]: Started Ceph mds.e for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:45.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:44 smithi203 ceph-mon[33115]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:45.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:44 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2413274771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:46.390 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:46.415 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:46.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:46.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:46.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: Deploying daemon mds.f on smithi203 2024-04-25T16:29:46.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:46.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] up:boot 2024-04-25T16:29:46.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: fsmap 5 up:standby 2024-04-25T16:29:46.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:29:46.450 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.450 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.450 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.450 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:46.450 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:46.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:46.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: Deploying daemon mds.f on smithi203 2024-04-25T16:29:46.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:46.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] up:boot 2024-04-25T16:29:46.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: fsmap 5 up:standby 2024-04-25T16:29:46.451 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:29:46.514 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 -- 172.21.15.98:0/2901045523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564104d10 msgr2=0x7fd564105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:46.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 --2- 172.21.15.98:0/2901045523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564104d10 0x7fd564105120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd54c009900 tx=0x7fd54c02a410 comp rx=0 tx=0).stop 2024-04-25T16:29:46.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 -- 172.21.15.98:0/2901045523 shutdown_connections 2024-04-25T16:29:46.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 --2- 172.21.15.98:0/2901045523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd564109a10 0x7fd56410be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 --2- 172.21.15.98:0/2901045523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564104d10 0x7fd564105120 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 --2- 172.21.15.98:0/2901045523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd564103ad0 0x7fd5641047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.515 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 -- 172.21.15.98:0/2901045523 >> 172.21.15.98:0/2901045523 conn(0x7fd5640fecc0 msgr2=0x7fd564101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 -- 172.21.15.98:0/2901045523 shutdown_connections 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.511+0000 7fd56979d640 1 -- 172.21.15.98:0/2901045523 wait complete. 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd56979d640 1 Processor -- start 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd56979d640 1 -- start start 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd56979d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564103ad0 0x7fd564076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd56979d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd564104d10 0x7fd564074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd56979d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd564109a10 0x7fd564075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd56979d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd56410f480 con 0x7fd564103ad0 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd56979d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd56410f300 con 0x7fd564104d10 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd56979d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd56410f600 con 0x7fd564109a10 2024-04-25T16:29:46.516 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd562ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564103ad0 0x7fd564076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd562ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564103ad0 0x7fd564076850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43020/0 (socket says 172.21.15.98:43020) 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.512+0000 7fd562ffd640 1 -- 172.21.15.98:0/357085650 learned_addr learned my addr 172.21.15.98:0/357085650 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd5637fe640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd564109a10 0x7fd564075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd55bfff640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd564104d10 0x7fd564074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd562ffd640 1 -- 172.21.15.98:0/357085650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd564109a10 msgr2=0x7fd564075430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd562ffd640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd564109a10 0x7fd564075430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd562ffd640 1 -- 172.21.15.98:0/357085650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd564104d10 msgr2=0x7fd564074ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd562ffd640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd564104d10 0x7fd564074ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd562ffd640 1 -- 172.21.15.98:0/357085650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd564075b60 con 0x7fd564103ad0 2024-04-25T16:29:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd55bfff640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd564104d10 0x7fd564074ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:29:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd562ffd640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564103ad0 0x7fd564076850 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd55400c970 tx=0x7fd55400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd560ff9640 1 -- 172.21.15.98:0/357085650 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd554004220 con 0x7fd564103ad0 2024-04-25T16:29:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd560ff9640 1 -- 172.21.15.98:0/357085650 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd554007e40 con 0x7fd564103ad0 2024-04-25T16:29:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd560ff9640 1 -- 172.21.15.98:0/357085650 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55400f450 con 0x7fd564103ad0 2024-04-25T16:29:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5641a41e0 con 0x7fd564103ad0 2024-04-25T16:29:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.513+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5641a46d0 con 0x7fd564103ad0 2024-04-25T16:29:46.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.514+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd564108820 con 0x7fd564103ad0 2024-04-25T16:29:46.519 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.519 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.519 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:46.519 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:46.519 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:46.520 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:46.520 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: Deploying daemon mds.f on smithi203 2024-04-25T16:29:46.520 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:46.520 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] up:boot 2024-04-25T16:29:46.520 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: fsmap 5 up:standby 2024-04-25T16:29:46.520 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:29:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.515+0000 7fd560ff9640 1 -- 172.21.15.98:0/357085650 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd5540079b0 con 0x7fd564103ad0 2024-04-25T16:29:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.516+0000 7fd560ff9640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5500afb70 0x7fd5500b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.516+0000 7fd55bfff640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5500afb70 0x7fd5500b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.516+0000 7fd560ff9640 1 -- 172.21.15.98:0/357085650 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fd5540de860 con 0x7fd564103ad0 2024-04-25T16:29:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.516+0000 7fd55bfff640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5500afb70 0x7fd5500b2080 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd54c005f70 tx=0x7fd54c005720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:46.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.520+0000 7fd560ff9640 1 -- 172.21.15.98:0/357085650 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd554014030 con 0x7fd564103ad0 2024-04-25T16:29:46.733 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.733+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd56402db50 con 0x7fd564103ad0 2024-04-25T16:29:46.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.734+0000 7fd560ff9640 1 -- 172.21.15.98:0/357085650 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4696 (secure 0 0 0) 0x7fd5540a7960 con 0x7fd564103ad0 2024-04-25T16:29:46.734 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:46.735 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T16:29:46.735 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 6 2024-04-25T16:29:46.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5500afb70 msgr2=0x7fd5500b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:46.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5500afb70 0x7fd5500b2080 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd54c005f70 tx=0x7fd54c005720 comp rx=0 tx=0).stop 2024-04-25T16:29:46.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564103ad0 msgr2=0x7fd564076850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:46.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564103ad0 0x7fd564076850 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd55400c970 tx=0x7fd55400ce30 comp rx=0 tx=0).stop 2024-04-25T16:29:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 shutdown_connections 2024-04-25T16:29:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd5500afb70 0x7fd5500b2080 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd564109a10 0x7fd564075430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd564104d10 0x7fd564074ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 --2- 172.21.15.98:0/357085650 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd564103ad0 0x7fd564076850 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 >> 172.21.15.98:0/357085650 conn(0x7fd5640fecc0 msgr2=0x7fd5641059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 shutdown_connections 2024-04-25T16:29:46.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:46.736+0000 7fd56979d640 1 -- 172.21.15.98:0/357085650 wait complete. 2024-04-25T16:29:47.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:47 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/357085650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 systemd[1]: Starting Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:29:47.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:47 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/357085650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:47.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:47 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/357085650' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:47.722 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 podman[63851]: 2024-04-25 16:29:47.344343209 +0000 UTC m=+0.017839972 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:29:47.722 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 podman[63851]: 2024-04-25 16:29:47.461933007 +0000 UTC m=+0.135429796 container create 28934d9ccdf742c818c522ec2bf178a15900131b25d519031c348b78d83b6f8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-25T16:29:47.723 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 podman[63851]: 2024-04-25 16:29:47.723063699 +0000 UTC m=+0.396560442 container init 28934d9ccdf742c818c522ec2bf178a15900131b25d519031c348b78d83b6f8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T16:29:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 podman[63851]: 2024-04-25 16:29:47.725737842 +0000 UTC m=+0.399234591 container start 28934d9ccdf742c818c522ec2bf178a15900131b25d519031c348b78d83b6f8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:29:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 ceph-mds[63870]: -- 172.21.15.203:0/2529247974 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f94807ba40 con 0x55f9480c3a80 2024-04-25T16:29:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: starting mds.f at 2024-04-25T16:29:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 bash[63851]: 28934d9ccdf742c818c522ec2bf178a15900131b25d519031c348b78d83b6f8b 2024-04-25T16:29:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:29:47 smithi203 systemd[1]: Started Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:48.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 -- 172.21.15.98:0/704177428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170108920 msgr2=0x7f917010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 --2- 172.21.15.98:0/704177428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170108920 0x7f917010ad40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9168009ca0 tx=0x7f916802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 -- 172.21.15.98:0/704177428 shutdown_connections 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 --2- 172.21.15.98:0/704177428 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170108920 0x7f917010ad40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 --2- 172.21.15.98:0/704177428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9170103c20 0x7f9170104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 --2- 172.21.15.98:0/704177428 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f91701029e0 0x7f91701036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 -- 172.21.15.98:0/704177428 >> 172.21.15.98:0/704177428 conn(0x7f91700fdbd0 msgr2=0x7f9170100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 -- 172.21.15.98:0/704177428 shutdown_connections 2024-04-25T16:29:48.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 -- 172.21.15.98:0/704177428 wait complete. 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.870+0000 7f91768d8640 1 Processor -- start 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f91768d8640 1 -- start start 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f91768d8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91701029e0 0x7f9170198f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f91768d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170103c20 0x7f91701994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f91768d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9170108920 0x7f91701a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f91768d8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f917010e3f0 con 0x7f9170103c20 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f91768d8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f917010e270 con 0x7f91701029e0 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f91768d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f917010e570 con 0x7f9170108920 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170103c20 0x7f91701994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170103c20 0x7f91701994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43032/0 (socket says 172.21.15.98:43032) 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916f7fe640 1 -- 172.21.15.98:0/1928509523 learned_addr learned my addr 172.21.15.98:0/1928509523 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916ffff640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91701029e0 0x7f9170198f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f9174e75640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9170108920 0x7f91701a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916f7fe640 1 -- 172.21.15.98:0/1928509523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9170108920 msgr2=0x7f91701a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:48.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916f7fe640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9170108920 0x7f91701a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916f7fe640 1 -- 172.21.15.98:0/1928509523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91701029e0 msgr2=0x7f9170198f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916f7fe640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91701029e0 0x7f9170198f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.871+0000 7f916f7fe640 1 -- 172.21.15.98:0/1928509523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91701a0ad0 con 0x7f9170103c20 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.872+0000 7f916f7fe640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170103c20 0x7f91701994d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f916000d950 tx=0x7f916000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.872+0000 7f916d7fa640 1 -- 172.21.15.98:0/1928509523 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160017030 con 0x7f9170103c20 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.872+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91701a0dc0 con 0x7f9170103c20 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.872+0000 7f916d7fa640 1 -- 172.21.15.98:0/1928509523 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9160004020 con 0x7f9170103c20 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.872+0000 7f916d7fa640 1 -- 172.21.15.98:0/1928509523 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91600055d0 con 0x7f9170103c20 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.872+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91701a1230 con 0x7f9170103c20 2024-04-25T16:29:48.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.873+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9170107730 con 0x7f9170103c20 2024-04-25T16:29:48.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.878+0000 7f916d7fa640 1 -- 172.21.15.98:0/1928509523 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9160009a30 con 0x7f9170103c20 2024-04-25T16:29:48.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.879+0000 7f916d7fa640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91580afaf0 0x7f91580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.879+0000 7f916d7fa640 1 -- 172.21.15.98:0/1928509523 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f91600de6a0 con 0x7f9170103c20 2024-04-25T16:29:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.879+0000 7f916ffff640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91580afaf0 0x7f91580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:48.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.879+0000 7f916ffff640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91580afaf0 0x7f91580b2000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9164005fd0 tx=0x7f916400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:48.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:48.882+0000 7f916d7fa640 1 -- 172.21.15.98:0/1928509523 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91600a7720 con 0x7f9170103c20 2024-04-25T16:29:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.096+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f917002db50 con 0x7f9170103c20 2024-04-25T16:29:49.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.097+0000 7f916d7fa640 1 -- 172.21.15.98:0/1928509523 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f91600a7020 con 0x7f9170103c20 2024-04-25T16:29:49.097 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:49.097 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T16:29:49.097 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 7 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91580afaf0 msgr2=0x7f91580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91580afaf0 0x7f91580b2000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9164005fd0 tx=0x7f916400b410 comp rx=0 tx=0).stop 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170103c20 msgr2=0x7f91701994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170103c20 0x7f91701994d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f916000d950 tx=0x7f916000de10 comp rx=0 tx=0).stop 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 shutdown_connections 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91580afaf0 0x7f91580b2000 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9170108920 0x7f91701a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9170103c20 0x7f91701994d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 --2- 172.21.15.98:0/1928509523 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91701029e0 0x7f9170198f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 >> 172.21.15.98:0/1928509523 conn(0x7f91700fdbd0 msgr2=0x7f9170104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:49.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 shutdown_connections 2024-04-25T16:29:49.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:49.099+0000 7f91768d8640 1 -- 172.21.15.98:0/1928509523 wait complete. 2024-04-25T16:29:49.109 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.109 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.109 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.109 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:49.109 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:49.109 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:49.110 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: Deploying daemon mds.c on smithi203 2024-04-25T16:29:49.110 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:49.110 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:boot 2024-04-25T16:29:49.110 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: fsmap 6 up:standby 2024-04-25T16:29:49.110 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:48 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:29:49.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:49.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:49.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:49.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: Deploying daemon mds.c on smithi203 2024-04-25T16:29:49.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:49.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:boot 2024-04-25T16:29:49.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: fsmap 6 up:standby 2024-04-25T16:29:49.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:48 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: Deploying daemon mds.c on smithi203 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:boot 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: fsmap 6 up:standby 2024-04-25T16:29:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:48 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:29:50.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:49 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1928509523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:50.189 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 systemd[1]: Starting Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:29:50.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:49 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1928509523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:50.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:49 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1928509523' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:50.553 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 podman[64164]: 2024-04-25 16:29:50.19805032 +0000 UTC m=+0.018406855 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:29:50.553 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 podman[64164]: 2024-04-25 16:29:50.309547011 +0000 UTC m=+0.129903543 container create 684bc2cb2319ff9f23ef8c4c53c3ac3f474ca3a37e8eba2a8b2387a10e99c7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T16:29:50.553 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 podman[64164]: 2024-04-25 16:29:50.553946867 +0000 UTC m=+0.374303399 container init 684bc2cb2319ff9f23ef8c4c53c3ac3f474ca3a37e8eba2a8b2387a10e99c7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T16:29:50.902 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 podman[64164]: 2024-04-25 16:29:50.556686573 +0000 UTC m=+0.377043105 container start 684bc2cb2319ff9f23ef8c4c53c3ac3f474ca3a37e8eba2a8b2387a10e99c7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T16:29:50.902 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-mds[64183]: -- 172.21.15.203:0/1188881453 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561575b09a40 con 0x561575b51a80 2024-04-25T16:29:50.902 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: starting mds.c at 2024-04-25T16:29:50.902 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 bash[64164]: 684bc2cb2319ff9f23ef8c4c53c3ac3f474ca3a37e8eba2a8b2387a10e99c7e9 2024-04-25T16:29:50.902 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:29:50 smithi203 systemd[1]: Started Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:51.112 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:51.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:50 smithi173 ceph-mon[32556]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:51.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:51.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:51.167 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:51.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-mon[33115]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:51.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:51.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:51.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 -- 172.21.15.98:0/708038364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb81041f0 msgr2=0x7fafb8106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 --2- 172.21.15.98:0/708038364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb81041f0 0x7fafb8106610 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fafac009900 tx=0x7fafac02a410 comp rx=0 tx=0).stop 2024-04-25T16:29:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 -- 172.21.15.98:0/708038364 shutdown_connections 2024-04-25T16:29:51.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 --2- 172.21.15.98:0/708038364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafb8109750 0x7fafb810bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 --2- 172.21.15.98:0/708038364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafb8106be0 0x7fafb81090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 --2- 172.21.15.98:0/708038364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb81041f0 0x7fafb8106610 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 -- 172.21.15.98:0/708038364 >> 172.21.15.98:0/708038364 conn(0x7fafb80fdbd0 msgr2=0x7fafb8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 -- 172.21.15.98:0/708038364 shutdown_connections 2024-04-25T16:29:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 -- 172.21.15.98:0/708038364 wait complete. 2024-04-25T16:29:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 Processor -- start 2024-04-25T16:29:51.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.234+0000 7fafbe714640 1 -- start start 2024-04-25T16:29:51.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafbe714640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafb81041f0 0x7fafb8103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:51.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafbe714640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb8106be0 0x7fafb8102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:51.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafbe714640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafb8109750 0x7fafb8102640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:51.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafbe714640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810f460 con 0x7fafb8106be0 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafbe714640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810f2e0 con 0x7fafb8109750 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafbe714640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fafb810f5e0 con 0x7fafb81041f0 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb8106be0 0x7fafb8102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafbccb1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafb8109750 0x7fafb8102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb8106be0 0x7fafb8102100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43052/0 (socket says 172.21.15.98:43052) 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb77fe640 1 -- 172.21.15.98:0/1337951647 learned_addr learned my addr 172.21.15.98:0/1337951647 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb7fff640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafb81041f0 0x7fafb8103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb77fe640 1 -- 172.21.15.98:0/1337951647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafb81041f0 msgr2=0x7fafb8103a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb77fe640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafb81041f0 0x7fafb8103a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb77fe640 1 -- 172.21.15.98:0/1337951647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafb8109750 msgr2=0x7fafb8102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb77fe640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafb8109750 0x7fafb8102640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.235+0000 7fafb77fe640 1 -- 172.21.15.98:0/1337951647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb8102ce0 con 0x7fafb8106be0 2024-04-25T16:29:51.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.236+0000 7fafb77fe640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb8106be0 0x7fafb8102100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fafa800c920 tx=0x7fafa800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:51.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.236+0000 7fafb57fa640 1 -- 172.21.15.98:0/1337951647 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa8007c70 con 0x7fafb8106be0 2024-04-25T16:29:51.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.236+0000 7fafb57fa640 1 -- 172.21.15.98:0/1337951647 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fafa8007e10 con 0x7fafb8106be0 2024-04-25T16:29:51.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.236+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb81a41c0 con 0x7fafb8106be0 2024-04-25T16:29:51.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.236+0000 7fafb57fa640 1 -- 172.21.15.98:0/1337951647 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa80186f0 con 0x7fafb8106be0 2024-04-25T16:29:51.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.236+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb81a45d0 con 0x7fafb8106be0 2024-04-25T16:29:51.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.237+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb80ff6f0 con 0x7fafb8106be0 2024-04-25T16:29:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.239+0000 7fafb57fa640 1 -- 172.21.15.98:0/1337951647 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fafa800f450 con 0x7fafb8106be0 2024-04-25T16:29:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.239+0000 7fafb57fa640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafa00afaf0 0x7fafa00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.239+0000 7fafb7fff640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafa00afaf0 0x7fafa00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.239+0000 7fafb57fa640 1 -- 172.21.15.98:0/1337951647 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fafa80debd0 con 0x7fafb8106be0 2024-04-25T16:29:51.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.240+0000 7fafb7fff640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafa00afaf0 0x7fafa00b2000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fafac006010 tx=0x7fafac000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:51.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.243+0000 7fafb57fa640 1 -- 172.21.15.98:0/1337951647 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafa8014030 con 0x7fafb8106be0 2024-04-25T16:29:51.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:50 smithi098 ceph-mon[29162]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:51.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:51.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:51.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:51.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:51.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.459+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafb802db50 con 0x7fafb8106be0 2024-04-25T16:29:51.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.460+0000 7fafb57fa640 1 -- 172.21.15.98:0/1337951647 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7fafa80a7ce0 con 0x7fafb8106be0 2024-04-25T16:29:51.460 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:51.460 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T16:29:51.460 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-04-25T16:29:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafa00afaf0 msgr2=0x7fafa00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafa00afaf0 0x7fafa00b2000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fafac006010 tx=0x7fafac000bc0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb8106be0 msgr2=0x7fafb8102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb8106be0 0x7fafb8102100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fafa800c920 tx=0x7fafa800cde0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 shutdown_connections 2024-04-25T16:29:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafa00afaf0 0x7fafa00b2000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafb8109750 0x7fafb8102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafb8106be0 0x7fafb8102100 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 --2- 172.21.15.98:0/1337951647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafb81041f0 0x7fafb8103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 >> 172.21.15.98:0/1337951647 conn(0x7fafb80fdbd0 msgr2=0x7fafb8106230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 shutdown_connections 2024-04-25T16:29:51.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:51.462+0000 7fafbe714640 1 -- 172.21.15.98:0/1337951647 wait complete. 2024-04-25T16:29:52.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:51 smithi173 ceph-mon[32556]: Deploying daemon mds.h on smithi173 2024-04-25T16:29:52.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:51 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] up:boot 2024-04-25T16:29:52.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:51 smithi173 ceph-mon[32556]: fsmap 7 up:standby 2024-04-25T16:29:52.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:51 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:29:52.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:51 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1337951647' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:52.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:51 smithi203 ceph-mon[33115]: Deploying daemon mds.h on smithi173 2024-04-25T16:29:52.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:51 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] up:boot 2024-04-25T16:29:52.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:51 smithi203 ceph-mon[33115]: fsmap 7 up:standby 2024-04-25T16:29:52.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:51 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:29:52.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:51 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1337951647' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:52.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:51 smithi098 ceph-mon[29162]: Deploying daemon mds.h on smithi173 2024-04-25T16:29:52.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:51 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] up:boot 2024-04-25T16:29:52.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:51 smithi098 ceph-mon[29162]: fsmap 7 up:standby 2024-04-25T16:29:52.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:51 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:29:52.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:51 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1337951647' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:52.943 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:52 smithi173 ceph-mon[32556]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:52 smithi203 ceph-mon[33115]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:52 smithi098 ceph-mon[29162]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:53.265 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:29:53 smithi173 podman[62482]: 2024-04-25 16:29:53.194495237 +0000 UTC m=+0.147692599 container create e35367aa004fd27ad58265fb85aea9df2269a3c33d89c8b7d4745b017273b04d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T16:29:53.474 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.596+0000 7f5de5261640 1 -- 172.21.15.98:0/983108533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de0103c00 msgr2=0x7f5de0104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.596+0000 7f5de5261640 1 --2- 172.21.15.98:0/983108533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de0103c00 0x7f5de0104010 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5dd4009900 tx=0x7f5dd402a410 comp rx=0 tx=0).stop 2024-04-25T16:29:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.596+0000 7f5de5261640 1 -- 172.21.15.98:0/983108533 shutdown_connections 2024-04-25T16:29:53.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.596+0000 7f5de5261640 1 --2- 172.21.15.98:0/983108533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de0108900 0x7f5de010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.596+0000 7f5de5261640 1 --2- 172.21.15.98:0/983108533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de0103c00 0x7f5de0104010 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.596+0000 7f5de5261640 1 --2- 172.21.15.98:0/983108533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de01029c0 0x7f5de01036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.596+0000 7f5de5261640 1 -- 172.21.15.98:0/983108533 >> 172.21.15.98:0/983108533 conn(0x7f5de00fdbd0 msgr2=0x7f5de0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.596+0000 7f5de5261640 1 -- 172.21.15.98:0/983108533 shutdown_connections 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.597+0000 7f5de5261640 1 -- 172.21.15.98:0/983108533 wait complete. 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.597+0000 7f5de5261640 1 Processor -- start 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.597+0000 7f5de5261640 1 -- start start 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.597+0000 7f5de5261640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de01029c0 0x7f5de006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.597+0000 7f5de5261640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de0103c00 0x7f5de019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.597+0000 7f5de5261640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de0108900 0x7f5de01a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.597+0000 7f5de5261640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5de010e390 con 0x7f5de01029c0 2024-04-25T16:29:53.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.597+0000 7f5de5261640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5de010e210 con 0x7f5de0108900 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5de5261640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5de010e510 con 0x7f5de0103c00 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de01029c0 0x7f5de006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de01029c0 0x7f5de006a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:43076/0 (socket says 172.21.15.98:43076) 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 -- 172.21.15.98:0/588621481 learned_addr learned my addr 172.21.15.98:0/588621481 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5dde7fc640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de0103c00 0x7f5de019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 -- 172.21.15.98:0/588621481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de0103c00 msgr2=0x7f5de019eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de0103c00 0x7f5de019eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 -- 172.21.15.98:0/588621481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de0108900 msgr2=0x7f5de01a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de0108900 0x7f5de01a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 -- 172.21.15.98:0/588621481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de01a1c40 con 0x7f5de01029c0 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5ddeffd640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de01029c0 0x7f5de006a920 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000eb50 tx=0x7f5dd000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5dc7fff640 1 -- 172.21.15.98:0/588621481 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0003b70 con 0x7f5de01029c0 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5dc7fff640 1 -- 172.21.15.98:0/588621481 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5dd000ee50 con 0x7f5de01029c0 2024-04-25T16:29:53.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de01a1f30 con 0x7f5de01029c0 2024-04-25T16:29:53.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5dc7fff640 1 -- 172.21.15.98:0/588621481 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0010720 con 0x7f5de01029c0 2024-04-25T16:29:53.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.598+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de01a23a0 con 0x7f5de01029c0 2024-04-25T16:29:53.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.599+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5de0107710 con 0x7f5de01029c0 2024-04-25T16:29:53.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.601+0000 7f5dc7fff640 1 -- 172.21.15.98:0/588621481 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5dd0004a40 con 0x7f5de01029c0 2024-04-25T16:29:53.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.601+0000 7f5dc7fff640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5dc80afaf0 0x7f5dc80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:53.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.602+0000 7f5dde7fc640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5dc80afaf0 0x7f5dc80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:53.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.602+0000 7f5dc7fff640 1 -- 172.21.15.98:0/588621481 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f5dd00defd0 con 0x7f5de01029c0 2024-04-25T16:29:53.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.602+0000 7f5dde7fc640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5dc80afaf0 0x7f5dc80b2000 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5de006a2c0 tx=0x7f5dd4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:53.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.605+0000 7f5dc7fff640 1 -- 172.21.15.98:0/588621481 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dd00a8050 con 0x7f5de01029c0 2024-04-25T16:29:53.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:29:53 smithi173 podman[62482]: 2024-04-25 16:29:53.439035085 +0000 UTC m=+0.392232458 container init e35367aa004fd27ad58265fb85aea9df2269a3c33d89c8b7d4745b017273b04d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-25T16:29:53.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:29:53 smithi173 podman[62482]: 2024-04-25 16:29:53.441659771 +0000 UTC m=+0.394857142 container start e35367aa004fd27ad58265fb85aea9df2269a3c33d89c8b7d4745b017273b04d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T16:29:53.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:29:53 smithi173 ceph-mds[62501]: -- 172.21.15.173:0/3320568249 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56101da5da40 con 0x56101daa5a80 2024-04-25T16:29:53.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:29:53 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: starting mds.h at 2024-04-25T16:29:53.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:29:53 smithi173 bash[62482]: e35367aa004fd27ad58265fb85aea9df2269a3c33d89c8b7d4745b017273b04d 2024-04-25T16:29:53.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:29:53 smithi173 systemd[1]: Started Ceph mds.h for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:53.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.835+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5de002db50 con 0x7f5de01029c0 2024-04-25T16:29:53.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.835+0000 7f5dc7fff640 1 -- 172.21.15.98:0/588621481 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6349 (secure 0 0 0) 0x7f5dd00a7950 con 0x7f5de01029c0 2024-04-25T16:29:53.835 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:53.835 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T16:29:53.836 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 8 2024-04-25T16:29:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5dc80afaf0 msgr2=0x7f5dc80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:53.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5dc80afaf0 0x7f5dc80b2000 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5de006a2c0 tx=0x7f5dd4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de01029c0 msgr2=0x7f5de006a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de01029c0 0x7f5de006a920 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5dd000eb50 tx=0x7f5dd000c6a0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 shutdown_connections 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5dc80afaf0 0x7f5dc80b2000 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5de0108900 0x7f5de01a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5de0103c00 0x7f5de019eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 --2- 172.21.15.98:0/588621481 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5de01029c0 0x7f5de006a920 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 >> 172.21.15.98:0/588621481 conn(0x7f5de00fdbd0 msgr2=0x7f5de01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 shutdown_connections 2024-04-25T16:29:53.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:53.837+0000 7f5de5261640 1 -- 172.21.15.98:0/588621481 wait complete. 2024-04-25T16:29:54.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: Deploying daemon mds.i on smithi203 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/588621481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] up:boot 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: fsmap 8 up:standby 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:29:54.939 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:54 smithi203 ceph-mon[33115]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:55.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: Deploying daemon mds.i on smithi203 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/588621481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] up:boot 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: fsmap 8 up:standby 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:29:55.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:54 smithi098 ceph-mon[29162]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:55.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: Deploying daemon mds.i on smithi203 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/588621481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] up:boot 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: fsmap 8 up:standby 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:29:55.015 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:54 smithi173 ceph-mon[32556]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:55.849 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:55.938 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:55 smithi203 systemd[1]: Starting Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:29:55.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 -- 172.21.15.98:0/3258793753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 msgr2=0x7f3040104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3258793753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 0x7f3040104010 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3034009900 tx=0x7f303402a410 comp rx=0 tx=0).stop 2024-04-25T16:29:55.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 -- 172.21.15.98:0/3258793753 shutdown_connections 2024-04-25T16:29:55.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3258793753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3040108900 0x7f304010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3258793753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 0x7f3040104010 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3258793753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30401029c0 0x7f30401036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 -- 172.21.15.98:0/3258793753 >> 172.21.15.98:0/3258793753 conn(0x7f30400fdbd0 msgr2=0x7f3040100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 -- 172.21.15.98:0/3258793753 shutdown_connections 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.972+0000 7f3046f1f640 1 -- 172.21.15.98:0/3258793753 wait complete. 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3046f1f640 1 Processor -- start 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3046f1f640 1 -- start start 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3046f1f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30401029c0 0x7f3040199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3046f1f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 0x7f3040199560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3046f1f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3040108900 0x7f30401a04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3046f1f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f304010e3e0 con 0x7f3040103c00 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3046f1f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f304010e260 con 0x7f30401029c0 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3046f1f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f304010e560 con 0x7f3040108900 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f303ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 0x7f3040199560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f303ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 0x7f3040199560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36820/0 (socket says 172.21.15.98:36820) 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f303ffff640 1 -- 172.21.15.98:0/3083594465 learned_addr learned my addr 172.21.15.98:0/3083594465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f3044cbb640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30401029c0 0x7f3040199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f30454bc640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3040108900 0x7f30401a04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f303ffff640 1 -- 172.21.15.98:0/3083594465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3040108900 msgr2=0x7f30401a04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f303ffff640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3040108900 0x7f30401a04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f303ffff640 1 -- 172.21.15.98:0/3083594465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30401029c0 msgr2=0x7f3040199020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f303ffff640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30401029c0 0x7f3040199020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f303ffff640 1 -- 172.21.15.98:0/3083594465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30401a0b60 con 0x7f3040103c00 2024-04-25T16:29:55.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.973+0000 7f30454bc640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3040108900 0x7f30401a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.974+0000 7f303ffff640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 0x7f3040199560 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3034005760 tx=0x7f3034003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.974+0000 7f303dffb640 1 -- 172.21.15.98:0/3083594465 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034035070 con 0x7f3040103c00 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.974+0000 7f303dffb640 1 -- 172.21.15.98:0/3083594465 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3034031990 con 0x7f3040103c00 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.974+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3040069e90 con 0x7f3040103c00 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.974+0000 7f303dffb640 1 -- 172.21.15.98:0/3083594465 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034038990 con 0x7f3040103c00 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.974+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304006a380 con 0x7f3040103c00 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.975+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f304006aec0 con 0x7f3040103c00 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.977+0000 7f303dffb640 1 -- 172.21.15.98:0/3083594465 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3034031500 con 0x7f3040103c00 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.977+0000 7f303dffb640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30280afaf0 0x7f30280b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.977+0000 7f3044cbb640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30280afaf0 0x7f30280b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:55.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.977+0000 7f303dffb640 1 -- 172.21.15.98:0/3083594465 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f3034100a40 con 0x7f3040103c00 2024-04-25T16:29:55.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.978+0000 7f3044cbb640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30280afaf0 0x7f30280b2000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f30401035c0 tx=0x7f3030005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:55.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:55.981+0000 7f303dffb640 1 -- 172.21.15.98:0/3083594465 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f30340c9ac0 con 0x7f3040103c00 2024-04-25T16:29:56.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.196+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3040107180 con 0x7f3040103c00 2024-04-25T16:29:56.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.197+0000 7f303dffb640 1 -- 172.21.15.98:0/3083594465 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f30340c93c0 con 0x7f3040103c00 2024-04-25T16:29:56.197 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:56.197 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T16:29:56.197 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 9 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30280afaf0 msgr2=0x7f30280b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30280afaf0 0x7f30280b2000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f30401035c0 tx=0x7f3030005f90 comp rx=0 tx=0).stop 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 msgr2=0x7f3040199560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 0x7f3040199560 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f3034005760 tx=0x7f3034003b90 comp rx=0 tx=0).stop 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 shutdown_connections 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30280afaf0 0x7f30280b2000 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3040108900 0x7f30401a04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3040103c00 0x7f3040199560 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 --2- 172.21.15.98:0/3083594465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f30401029c0 0x7f3040199020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:56.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 >> 172.21.15.98:0/3083594465 conn(0x7f30400fdbd0 msgr2=0x7f30401048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 shutdown_connections 2024-04-25T16:29:56.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:56.199+0000 7f3046f1f640 1 -- 172.21.15.98:0/3083594465 wait complete. 2024-04-25T16:29:56.315 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:56 smithi203 podman[64484]: 2024-04-25 16:29:55.951495363 +0000 UTC m=+0.018858215 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:29:56.315 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:56 smithi203 podman[64484]: 2024-04-25 16:29:56.080405147 +0000 UTC m=+0.147768030 container create 33242d8aa2635e47e909082b48a72890de0b43f7489d00a0e0472a8d82eeabd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:29:56.316 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:56 smithi203 podman[64484]: 2024-04-25 16:29:56.316364304 +0000 UTC m=+0.383727132 container init 33242d8aa2635e47e909082b48a72890de0b43f7489d00a0e0472a8d82eeabd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T16:29:56.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:56 smithi203 podman[64484]: 2024-04-25 16:29:56.31909369 +0000 UTC m=+0.386456527 container start 33242d8aa2635e47e909082b48a72890de0b43f7489d00a0e0472a8d82eeabd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T16:29:56.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mds[64503]: -- 172.21.15.203:0/2147204813 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564f9f0e7a40 con 0x564f9f12fa80 2024-04-25T16:29:56.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: starting mds.i at 2024-04-25T16:29:56.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:56 smithi203 bash[64484]: 33242d8aa2635e47e909082b48a72890de0b43f7489d00a0e0472a8d82eeabd7 2024-04-25T16:29:56.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:29:56 smithi203 systemd[1]: Started Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:29:57.232 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:56 smithi098 ceph-mon[29162]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:57.232 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:56 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3083594465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:57.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:57.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:57.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:56 smithi173 ceph-mon[32556]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:56 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3083594465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:57.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mon[33115]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3083594465' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:29:57.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:29:57.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:29:57.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:29:58.211 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:29:58.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:57 smithi098 ceph-mon[29162]: Deploying daemon mds.g on smithi098 2024-04-25T16:29:58.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:57 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] up:boot 2024-04-25T16:29:58.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:57 smithi098 ceph-mon[29162]: fsmap 9 up:standby 2024-04-25T16:29:58.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:29:58.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:57 smithi173 ceph-mon[32556]: Deploying daemon mds.g on smithi098 2024-04-25T16:29:58.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:57 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] up:boot 2024-04-25T16:29:58.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:57 smithi173 ceph-mon[32556]: fsmap 9 up:standby 2024-04-25T16:29:58.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:29:58.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 -- 172.21.15.98:0/2915297494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf01045c0 msgr2=0x7efdf01069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 --2- 172.21.15.98:0/2915297494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf01045c0 0x7efdf01069e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efdec009900 tx=0x7efdec02a410 comp rx=0 tx=0).stop 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 -- 172.21.15.98:0/2915297494 shutdown_connections 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 --2- 172.21.15.98:0/2915297494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf0109880 0x7efdf010bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 --2- 172.21.15.98:0/2915297494 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdf0106f20 0x7efdf0109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 --2- 172.21.15.98:0/2915297494 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf01045c0 0x7efdf01069e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 -- 172.21.15.98:0/2915297494 >> 172.21.15.98:0/2915297494 conn(0x7efdf00fdba0 msgr2=0x7efdf0100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 -- 172.21.15.98:0/2915297494 shutdown_connections 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.331+0000 7efdf870d640 1 -- 172.21.15.98:0/2915297494 wait complete. 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf870d640 1 Processor -- start 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf870d640 1 -- start start 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf870d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdf01045c0 0x7efdf019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf870d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0106f20 0x7efdf019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf870d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf0109880 0x7efdf01a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf870d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efdf010f520 con 0x7efdf0106f20 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf870d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efdf010f3a0 con 0x7efdf0109880 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf870d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efdf010f6a0 con 0x7efdf01045c0 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf5ca8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0106f20 0x7efdf019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf5ca8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0106f20 0x7efdf019a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36836/0 (socket says 172.21.15.98:36836) 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf5ca8640 1 -- 172.21.15.98:0/4110633987 learned_addr learned my addr 172.21.15.98:0/4110633987 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf64a9640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdf01045c0 0x7efdf019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf6caa640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf0109880 0x7efdf01a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf5ca8640 1 -- 172.21.15.98:0/4110633987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdf01045c0 msgr2=0x7efdf019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf5ca8640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdf01045c0 0x7efdf019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf5ca8640 1 -- 172.21.15.98:0/4110633987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf0109880 msgr2=0x7efdf01a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf5ca8640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf0109880 0x7efdf01a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.332+0000 7efdf5ca8640 1 -- 172.21.15.98:0/4110633987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf01a1cd0 con 0x7efdf0106f20 2024-04-25T16:29:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.333+0000 7efdf5ca8640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0106f20 0x7efdf019a6d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efde000d770 tx=0x7efde000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.333+0000 7efde77fe640 1 -- 172.21.15.98:0/4110633987 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0017070 con 0x7efdf0106f20 2024-04-25T16:29:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.333+0000 7efde77fe640 1 -- 172.21.15.98:0/4110633987 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efde00054b0 con 0x7efdf0106f20 2024-04-25T16:29:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.333+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf01a1fc0 con 0x7efdf0106f20 2024-04-25T16:29:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.333+0000 7efde77fe640 1 -- 172.21.15.98:0/4110633987 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde0004c60 con 0x7efdf0106f20 2024-04-25T16:29:58.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.333+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf01a24a0 con 0x7efdf0106f20 2024-04-25T16:29:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.334+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdf00ff6c0 con 0x7efdf0106f20 2024-04-25T16:29:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.335+0000 7efde77fe640 1 -- 172.21.15.98:0/4110633987 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7efde0004e00 con 0x7efdf0106f20 2024-04-25T16:29:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.336+0000 7efde77fe640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efddc0afaf0 0x7efddc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:29:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.336+0000 7efdf64a9640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efddc0afaf0 0x7efddc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:29:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.336+0000 7efde77fe640 1 -- 172.21.15.98:0/4110633987 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7efde00de170 con 0x7efdf0106f20 2024-04-25T16:29:58.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.337+0000 7efdf64a9640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efddc0afaf0 0x7efddc0b2000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efdec005ab0 tx=0x7efdec000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:29:58.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.341+0000 7efde77fe640 1 -- 172.21.15.98:0/4110633987 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efde00a6ba0 con 0x7efdf0106f20 2024-04-25T16:29:58.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:57 smithi203 ceph-mon[33115]: Deploying daemon mds.g on smithi098 2024-04-25T16:29:58.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:57 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] up:boot 2024-04-25T16:29:58.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:57 smithi203 ceph-mon[33115]: fsmap 9 up:standby 2024-04-25T16:29:58.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:29:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.581+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efdf002db50 con 0x7efdf0106f20 2024-04-25T16:29:58.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.582+0000 7efde77fe640 1 -- 172.21.15.98:0/4110633987 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7efde00a7e20 con 0x7efdf0106f20 2024-04-25T16:29:58.582 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:29:58.582 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T16:29:58.582 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 10 2024-04-25T16:29:58.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efddc0afaf0 msgr2=0x7efddc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efddc0afaf0 0x7efddc0b2000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efdec005ab0 tx=0x7efdec000bc0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0106f20 msgr2=0x7efdf019a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:29:58.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0106f20 0x7efdf019a6d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efde000d770 tx=0x7efde000dc30 comp rx=0 tx=0).stop 2024-04-25T16:29:58.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 shutdown_connections 2024-04-25T16:29:58.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efddc0afaf0 0x7efddc0b2000 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf0109880 0x7efdf01a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efdf0106f20 0x7efdf019a6d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 --2- 172.21.15.98:0/4110633987 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efdf01045c0 0x7efdf019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:29:58.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 >> 172.21.15.98:0/4110633987 conn(0x7efdf00fdba0 msgr2=0x7efdf01065f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:29:58.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 shutdown_connections 2024-04-25T16:29:58.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:29:58.584+0000 7efdf870d640 1 -- 172.21.15.98:0/4110633987 wait complete. 2024-04-25T16:29:58.909 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:58 smithi098 systemd[1]: Starting Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:29:59.172 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:58 smithi098 ceph-mon[29162]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:59.173 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:29:58 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4110633987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:59.173 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:58 smithi098 podman[70385]: 2024-04-25 16:29:58.908992956 +0000 UTC m=+0.113158276 container create ca2f757698d5afa7f459498c081e0f081289481535cc76e24e372b0c2ea36f1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:29:59.173 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:58 smithi098 podman[70385]: 2024-04-25 16:29:58.813816128 +0000 UTC m=+0.017981418 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:29:59.173 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:59 smithi098 podman[70385]: 2024-04-25 16:29:59.137332604 +0000 UTC m=+0.341497880 container init ca2f757698d5afa7f459498c081e0f081289481535cc76e24e372b0c2ea36f1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:29:59.173 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:59 smithi098 podman[70385]: 2024-04-25 16:29:59.140272841 +0000 UTC m=+0.344438121 container start ca2f757698d5afa7f459498c081e0f081289481535cc76e24e372b0c2ea36f1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:29:59.173 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:59 smithi098 ceph-mds[70405]: -- 172.21.15.98:0/4209352826 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55909c6a1a40 con 0x55909c6e9a80 2024-04-25T16:29:59.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:58 smithi173 ceph-mon[32556]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:59.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:29:58 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4110633987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:59.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:58 smithi203 ceph-mon[33115]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:29:59.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:29:58 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4110633987' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:29:59.502 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:59 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: starting mds.g at 2024-04-25T16:29:59.503 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:59 smithi098 bash[70385]: ca2f757698d5afa7f459498c081e0f081289481535cc76e24e372b0c2ea36f1f 2024-04-25T16:29:59.503 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:29:59 smithi098 systemd[1]: Started Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: Deploying daemon mds.a on smithi098 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:boot 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: fsmap 10 up:standby 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:30:00.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:00 smithi098 ceph-mon[29162]: overall HEALTH_OK 2024-04-25T16:30:00.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:00.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: Deploying daemon mds.a on smithi098 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:boot 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: fsmap 10 up:standby 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:30:00.689 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:00 smithi203 ceph-mon[33115]: overall HEALTH_OK 2024-04-25T16:30:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 -- 172.21.15.98:0/1455475420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0103c10 msgr2=0x7fb4d0104020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 --2- 172.21.15.98:0/1455475420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0103c10 0x7fb4d0104020 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc009900 tx=0x7fb4cc02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 -- 172.21.15.98:0/1455475420 shutdown_connections 2024-04-25T16:30:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 --2- 172.21.15.98:0/1455475420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4d0108910 0x7fb4d010ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 --2- 172.21.15.98:0/1455475420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0103c10 0x7fb4d0104020 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 --2- 172.21.15.98:0/1455475420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4d01029d0 0x7fb4d01036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 -- 172.21.15.98:0/1455475420 >> 172.21.15.98:0/1455475420 conn(0x7fb4d00fdba0 msgr2=0x7fb4d0100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:00.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 -- 172.21.15.98:0/1455475420 shutdown_connections 2024-04-25T16:30:00.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.712+0000 7fb4d7372640 1 -- 172.21.15.98:0/1455475420 wait complete. 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d7372640 1 Processor -- start 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d7372640 1 -- start start 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d7372640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4d01029d0 0x7fb4d0199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d7372640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4d0103c10 0x7fb4d0199550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d7372640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0108910 0x7fb4d01a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d7372640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d010e3b0 con 0x7fb4d0108910 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d7372640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d010e230 con 0x7fb4d0103c10 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d7372640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d010e530 con 0x7fb4d01029d0 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d590f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0108910 0x7fb4d01a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:00.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d590f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0108910 0x7fb4d01a04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36860/0 (socket says 172.21.15.98:36860) 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d590f640 1 -- 172.21.15.98:0/2405177577 learned_addr learned my addr 172.21.15.98:0/2405177577 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d510e640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4d01029d0 0x7fb4d0199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d490d640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4d0103c10 0x7fb4d0199550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d590f640 1 -- 172.21.15.98:0/2405177577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4d01029d0 msgr2=0x7fb4d0199010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d590f640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4d01029d0 0x7fb4d0199010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d590f640 1 -- 172.21.15.98:0/2405177577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4d0103c10 msgr2=0x7fb4d0199550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d590f640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4d0103c10 0x7fb4d0199550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.713+0000 7fb4d590f640 1 -- 172.21.15.98:0/2405177577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d01a0b50 con 0x7fb4d0108910 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.714+0000 7fb4d590f640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0108910 0x7fb4d01a04e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800eaa0 tx=0x7fb4c800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.714+0000 7fb4c67fc640 1 -- 172.21.15.98:0/2405177577 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c800cdb0 con 0x7fb4d0108910 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.714+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d01a0e40 con 0x7fb4d0108910 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.714+0000 7fb4c67fc640 1 -- 172.21.15.98:0/2405177577 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb4c8005020 con 0x7fb4d0108910 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.714+0000 7fb4c67fc640 1 -- 172.21.15.98:0/2405177577 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c80116d0 con 0x7fb4d0108910 2024-04-25T16:30:00.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.714+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d01a1300 con 0x7fb4d0108910 2024-04-25T16:30:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.715+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4d0107720 con 0x7fb4d0108910 2024-04-25T16:30:00.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.716+0000 7fb4c67fc640 1 -- 172.21.15.98:0/2405177577 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fb4c8004a40 con 0x7fb4d0108910 2024-04-25T16:30:00.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.717+0000 7fb4c67fc640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb4bc0afaf0 0x7fb4bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:00.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.717+0000 7fb4d510e640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb4bc0afaf0 0x7fb4bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:00.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.717+0000 7fb4c67fc640 1 -- 172.21.15.98:0/2405177577 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fb4c80decb0 con 0x7fb4d0108910 2024-04-25T16:30:00.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.718+0000 7fb4d510e640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb4bc0afaf0 0x7fb4bc0b2000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0005fd0 tx=0x7fb4c000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:00.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.725+0000 7fb4c67fc640 1 -- 172.21.15.98:0/2405177577 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb4c8016030 con 0x7fb4d0108910 2024-04-25T16:30:00.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: Deploying daemon mds.a on smithi098 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:boot 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: fsmap 10 up:standby 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:30:00.765 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:00 smithi173 ceph-mon[32556]: overall HEALTH_OK 2024-04-25T16:30:00.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.952+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4d002db50 con 0x7fb4d0108910 2024-04-25T16:30:00.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.953+0000 7fb4c67fc640 1 -- 172.21.15.98:0/2405177577 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8830 (secure 0 0 0) 0x7fb4c80a7d30 con 0x7fb4d0108910 2024-04-25T16:30:00.953 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:00.953 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T16:30:00.953 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 11 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb4bc0afaf0 msgr2=0x7fb4bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb4bc0afaf0 0x7fb4bc0b2000 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0005fd0 tx=0x7fb4c000b410 comp rx=0 tx=0).stop 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0108910 msgr2=0x7fb4d01a04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0108910 0x7fb4d01a04e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb4c800eaa0 tx=0x7fb4c800ef60 comp rx=0 tx=0).stop 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 shutdown_connections 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb4bc0afaf0 0x7fb4bc0b2000 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb4d0108910 0x7fb4d01a04e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4d0103c10 0x7fb4d0199550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 --2- 172.21.15.98:0/2405177577 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb4d01029d0 0x7fb4d0199010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:00.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 >> 172.21.15.98:0/2405177577 conn(0x7fb4d00fdba0 msgr2=0x7fb4d01048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 shutdown_connections 2024-04-25T16:30:00.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:00.955+0000 7fb4d7372640 1 -- 172.21.15.98:0/2405177577 wait complete. 2024-04-25T16:30:01.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:01 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2405177577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:01.503 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 systemd[1]: Starting Ceph mds.a for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:30:01.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:01 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2405177577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:01.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:01 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2405177577' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:01.837 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 podman[70747]: 2024-04-25 16:30:01.485833212 +0000 UTC m=+0.018227262 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:30:01.837 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 podman[70747]: 2024-04-25 16:30:01.59148405 +0000 UTC m=+0.123878078 container create c85411a4bd2ace02410ba3c35ee07e2422310b6afc9151d669f1eda9fe47f301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T16:30:01.837 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 podman[70747]: 2024-04-25 16:30:01.802540946 +0000 UTC m=+0.334934975 container init c85411a4bd2ace02410ba3c35ee07e2422310b6afc9151d669f1eda9fe47f301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T16:30:01.838 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 podman[70747]: 2024-04-25 16:30:01.805046103 +0000 UTC m=+0.337440133 container start c85411a4bd2ace02410ba3c35ee07e2422310b6afc9151d669f1eda9fe47f301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:30:01.838 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 ceph-mds[70766]: -- 172.21.15.98:0/3028681201 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560810f69a40 con 0x560810fb1180 2024-04-25T16:30:02.252 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a[70762]: starting mds.a at 2024-04-25T16:30:02.253 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 bash[70747]: c85411a4bd2ace02410ba3c35ee07e2422310b6afc9151d669f1eda9fe47f301 2024-04-25T16:30:02.253 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:30:01 smithi098 systemd[1]: Started Ceph mds.a for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:30:02.967 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:03.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 -- 172.21.15.98:0/3096465765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78108900 msgr2=0x7fef7810ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 --2- 172.21.15.98:0/3096465765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78108900 0x7fef7810ad20 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fef70009ca0 tx=0x7fef7002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 -- 172.21.15.98:0/3096465765 shutdown_connections 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 --2- 172.21.15.98:0/3096465765 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78108900 0x7fef7810ad20 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 --2- 172.21.15.98:0/3096465765 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78103c00 0x7fef78104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 --2- 172.21.15.98:0/3096465765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef781029c0 0x7fef781036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 -- 172.21.15.98:0/3096465765 >> 172.21.15.98:0/3096465765 conn(0x7fef780fdbd0 msgr2=0x7fef78100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 -- 172.21.15.98:0/3096465765 shutdown_connections 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 -- 172.21.15.98:0/3096465765 wait complete. 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 Processor -- start 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.086+0000 7fef7da51640 1 -- start start 2024-04-25T16:30:03.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7da51640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef781029c0 0x7fef7806a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7da51640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78103c00 0x7fef7819ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7da51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78108900 0x7fef781a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7da51640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef7810e3f0 con 0x7fef78103c00 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7da51640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef7810e270 con 0x7fef781029c0 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7da51640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef7810e570 con 0x7fef78108900 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78103c00 0x7fef7819ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78103c00 0x7fef7819ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36912/0 (socket says 172.21.15.98:36912) 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 -- 172.21.15.98:0/1539867103 learned_addr learned my addr 172.21.15.98:0/1539867103 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef76ffd640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef781029c0 0x7fef7806a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef777fe640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78108900 0x7fef781a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 -- 172.21.15.98:0/1539867103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78108900 msgr2=0x7fef781a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78108900 0x7fef781a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 -- 172.21.15.98:0/1539867103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef781029c0 msgr2=0x7fef7806a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef781029c0 0x7fef7806a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 -- 172.21.15.98:0/1539867103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef781a1c10 con 0x7fef78103c00 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef6ffff640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78103c00 0x7fef7819ee80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fef6000d770 tx=0x7fef6000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7ca4f640 1 -- 172.21.15.98:0/1539867103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60017070 con 0x7fef78103c00 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7ca4f640 1 -- 172.21.15.98:0/1539867103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef600054b0 con 0x7fef78103c00 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.087+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef781a1f00 con 0x7fef78103c00 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.088+0000 7fef7ca4f640 1 -- 172.21.15.98:0/1539867103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef60004c60 con 0x7fef78103c00 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.088+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef781a2370 con 0x7fef78103c00 2024-04-25T16:30:03.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.088+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef78107710 con 0x7fef78103c00 2024-04-25T16:30:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.091+0000 7fef7ca4f640 1 -- 172.21.15.98:0/1539867103 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fef60004e00 con 0x7fef78103c00 2024-04-25T16:30:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.091+0000 7fef7ca4f640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef640afaf0 0x7fef640b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.092+0000 7fef76ffd640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef640afaf0 0x7fef640b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.092+0000 7fef7ca4f640 1 -- 172.21.15.98:0/1539867103 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fef600de630 con 0x7fef78103c00 2024-04-25T16:30:03.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.092+0000 7fef76ffd640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef640afaf0 0x7fef640b2000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fef68005fd0 tx=0x7fef6800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:03.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.095+0000 7fef7ca4f640 1 -- 172.21.15.98:0/1539867103 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef600a7610 con 0x7fef78103c00 2024-04-25T16:30:03.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:03.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:30:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:30:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: Deploying daemon mds.b on smithi173 2024-04-25T16:30:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6840/373690634,v1:172.21.15.98:6841/373690634] up:boot 2024-04-25T16:30:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: fsmap 11 up:standby 2024-04-25T16:30:03.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:30:03.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: Deploying daemon mds.b on smithi173 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6840/373690634,v1:172.21.15.98:6841/373690634] up:boot 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: fsmap 11 up:standby 2024-04-25T16:30:03.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:30:03.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:03.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: Deploying daemon mds.b on smithi173 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6840/373690634,v1:172.21.15.98:6841/373690634] up:boot 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: fsmap 11 up:standby 2024-04-25T16:30:03.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:30:03.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.312+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef7802db50 con 0x7fef78103c00 2024-04-25T16:30:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.313+0000 7fef7ca4f640 1 -- 172.21.15.98:0/1539867103 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7fef600a6f10 con 0x7fef78103c00 2024-04-25T16:30:03.313 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:03.313 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-25T16:30:03.313 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 12 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef640afaf0 msgr2=0x7fef640b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef640afaf0 0x7fef640b2000 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fef68005fd0 tx=0x7fef6800b410 comp rx=0 tx=0).stop 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78103c00 msgr2=0x7fef7819ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78103c00 0x7fef7819ee80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fef6000d770 tx=0x7fef6000dc30 comp rx=0 tx=0).stop 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 shutdown_connections 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef640afaf0 0x7fef640b2000 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef78108900 0x7fef781a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef78103c00 0x7fef7819ee80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 --2- 172.21.15.98:0/1539867103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef781029c0 0x7fef7806a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:03.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 >> 172.21.15.98:0/1539867103 conn(0x7fef780fdbd0 msgr2=0x7fef781048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 shutdown_connections 2024-04-25T16:30:03.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:03.315+0000 7fef7da51640 1 -- 172.21.15.98:0/1539867103 wait complete. 2024-04-25T16:30:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:03 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1539867103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:03 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1539867103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:04.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:03 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1539867103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:04.265 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 systemd[1]: Starting Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:30:04.643 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 podman[62789]: 2024-04-25 16:30:04.365810502 +0000 UTC m=+0.111236477 container create 2d7b6bdfc8eb840845a042b20a4ab46713f429c27ea7ab15b9257b72e3a0423c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T16:30:04.643 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 podman[62789]: 2024-04-25 16:30:04.273145487 +0000 UTC m=+0.018571429 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:30:04.643 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 podman[62789]: 2024-04-25 16:30:04.576995214 +0000 UTC m=+0.322421143 container init 2d7b6bdfc8eb840845a042b20a4ab46713f429c27ea7ab15b9257b72e3a0423c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:30:04.643 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 podman[62789]: 2024-04-25 16:30:04.579755917 +0000 UTC m=+0.325181847 container start 2d7b6bdfc8eb840845a042b20a4ab46713f429c27ea7ab15b9257b72e3a0423c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T16:30:04.643 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-mds[62808]: -- 172.21.15.173:0/584279845 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5632511dda40 con 0x563251225180 2024-04-25T16:30:04.643 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: starting mds.b at 2024-04-25T16:30:04.644 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 bash[62789]: 2d7b6bdfc8eb840845a042b20a4ab46713f429c27ea7ab15b9257b72e3a0423c 2024-04-25T16:30:04.942 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:04 smithi173 systemd[1]: Started Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:30:05.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:04 smithi098 ceph-mon[29162]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:05.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:05.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-mon[32556]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:05.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:05.328 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:05.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:04 smithi203 ceph-mon[33115]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:05.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:05.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 -- 172.21.15.98:0/2195300597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108940 msgr2=0x7ff64410ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/2195300597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108940 0x7ff64410ad60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009ca0 tx=0x7ff63c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 -- 172.21.15.98:0/2195300597 shutdown_connections 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/2195300597 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644108940 0x7ff64410ad60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/2195300597 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff644103c40 0x7ff644104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/2195300597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff644102a00 0x7ff644103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 -- 172.21.15.98:0/2195300597 >> 172.21.15.98:0/2195300597 conn(0x7ff6440fdbf0 msgr2=0x7ff6441000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 -- 172.21.15.98:0/2195300597 shutdown_connections 2024-04-25T16:30:05.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.447+0000 7ff64a7a8640 1 -- 172.21.15.98:0/2195300597 wait complete. 2024-04-25T16:30:05.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.448+0000 7ff64a7a8640 1 Processor -- start 2024-04-25T16:30:05.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.448+0000 7ff64a7a8640 1 -- start start 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.448+0000 7ff64a7a8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff644102a00 0x7ff6441992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.448+0000 7ff64a7a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644103c40 0x7ff644199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.448+0000 7ff64a7a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff644108940 0x7ff6441a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.448+0000 7ff64a7a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff64410e430 con 0x7ff644103c40 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff64a7a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff64410e2b0 con 0x7ff644108940 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff64a7a8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff64410e5b0 con 0x7ff644102a00 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644103c40 0x7ff644199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644103c40 0x7ff644199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36928/0 (socket says 172.21.15.98:36928) 2024-04-25T16:30:05.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 -- 172.21.15.98:0/3482996306 learned_addr learned my addr 172.21.15.98:0/3482996306 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff643fff640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff644102a00 0x7ff6441992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff648d45640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff644108940 0x7ff6441a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 -- 172.21.15.98:0/3482996306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff644102a00 msgr2=0x7ff6441992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff644102a00 0x7ff6441992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 -- 172.21.15.98:0/3482996306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff644108940 msgr2=0x7ff6441a07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff644108940 0x7ff6441a07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 -- 172.21.15.98:0/3482996306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6441a0e30 con 0x7ff644103c40 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff648d45640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff644108940 0x7ff6441a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6437fe640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644103c40 0x7ff644199830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff63800e9f0 tx=0x7ff63800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6417fa640 1 -- 172.21.15.98:0/3482996306 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63800cd80 con 0x7ff644103c40 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6441a1120 con 0x7ff644103c40 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6417fa640 1 -- 172.21.15.98:0/3482996306 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6380054b0 con 0x7ff644103c40 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff6417fa640 1 -- 172.21.15.98:0/3482996306 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638010430 con 0x7ff644103c40 2024-04-25T16:30:05.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.449+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6441a1670 con 0x7ff644103c40 2024-04-25T16:30:05.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.450+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff644107750 con 0x7ff644103c40 2024-04-25T16:30:05.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.452+0000 7ff6417fa640 1 -- 172.21.15.98:0/3482996306 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff6380046c0 con 0x7ff644103c40 2024-04-25T16:30:05.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.453+0000 7ff6417fa640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff62c0afaf0 0x7ff62c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.453+0000 7ff643fff640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff62c0afaf0 0x7ff62c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.453+0000 7ff6417fa640 1 -- 172.21.15.98:0/3482996306 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7ff6380ded00 con 0x7ff644103c40 2024-04-25T16:30:05.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.454+0000 7ff643fff640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff62c0afaf0 0x7ff62c0b2000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff634005fd0 tx=0x7ff63400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:05.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.459+0000 7ff6417fa640 1 -- 172.21.15.98:0/3482996306 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff638053ff0 con 0x7ff644103c40 2024-04-25T16:30:05.683 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.683+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff644069b70 con 0x7ff644103c40 2024-04-25T16:30:05.684 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.684+0000 7ff6417fa640 1 -- 172.21.15.98:0/3482996306 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10482 (secure 0 0 0) 0x7ff6380a7d80 con 0x7ff644103c40 2024-04-25T16:30:05.684 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:05.684 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T16:30:05.684 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 13 2024-04-25T16:30:05.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff62c0afaf0 msgr2=0x7ff62c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:05.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff62c0afaf0 0x7ff62c0b2000 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff634005fd0 tx=0x7ff63400b410 comp rx=0 tx=0).stop 2024-04-25T16:30:05.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644103c40 msgr2=0x7ff644199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:05.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644103c40 0x7ff644199830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff63800e9f0 tx=0x7ff63800edf0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 shutdown_connections 2024-04-25T16:30:05.686 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff62c0afaf0 0x7ff62c0b2000 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff644108940 0x7ff6441a07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff644103c40 0x7ff644199830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 --2- 172.21.15.98:0/3482996306 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff644102a00 0x7ff6441992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 >> 172.21.15.98:0/3482996306 conn(0x7ff6440fdbf0 msgr2=0x7ff644104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:05.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 shutdown_connections 2024-04-25T16:30:05.687 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.686+0000 7ff64a7a8640 1 -- 172.21.15.98:0/3482996306 wait complete. 2024-04-25T16:30:05.698 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:05.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 -- 172.21.15.98:0/539949688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8103c50 msgr2=0x7f55a8104060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 --2- 172.21.15.98:0/539949688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8103c50 0x7f55a8104060 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009900 tx=0x7f55a402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 -- 172.21.15.98:0/539949688 shutdown_connections 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 --2- 172.21.15.98:0/539949688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a8108950 0x7f55a810ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 --2- 172.21.15.98:0/539949688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8103c50 0x7f55a8104060 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 --2- 172.21.15.98:0/539949688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a8102a10 0x7f55a8103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 -- 172.21.15.98:0/539949688 >> 172.21.15.98:0/539949688 conn(0x7f55a80fdc00 msgr2=0x7f55a81000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 -- 172.21.15.98:0/539949688 shutdown_connections 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.812+0000 7f55b0415640 1 -- 172.21.15.98:0/539949688 wait complete. 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55b0415640 1 Processor -- start 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55b0415640 1 -- start start 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55b0415640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a8102a10 0x7f55a80ffa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:05.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55b0415640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a8103c50 0x7f55a80fe130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55b0415640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8108950 0x7f55a80fe670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55b0415640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810e3f0 con 0x7f55a8108950 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55b0415640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810e270 con 0x7f55a8103c50 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55b0415640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810e570 con 0x7f55a8102a10 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55ae9b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8108950 0x7f55a80fe670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55ae9b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8108950 0x7f55a80fe670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48910/0 (socket says 172.21.15.98:48910) 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55ae9b2640 1 -- 172.21.15.98:0/889690553 learned_addr learned my addr 172.21.15.98:0/889690553 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55ae1b1640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a8102a10 0x7f55a80ffa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.813+0000 7f55ad9b0640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a8103c50 0x7f55a80fe130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55ae9b2640 1 -- 172.21.15.98:0/889690553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a8102a10 msgr2=0x7f55a80ffa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55ae9b2640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a8102a10 0x7f55a80ffa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55ae9b2640 1 -- 172.21.15.98:0/889690553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a8103c50 msgr2=0x7f55a80fe130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55ae9b2640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a8103c50 0x7f55a80fe130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55ae9b2640 1 -- 172.21.15.98:0/889690553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a80feda0 con 0x7f55a8108950 2024-04-25T16:30:05.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55ae1b1640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a8102a10 0x7f55a80ffa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55ae9b2640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8108950 0x7f55a80fe670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f55a000eb10 tx=0x7f55a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f559f7fe640 1 -- 172.21.15.98:0/889690553 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0004b20 con 0x7f55a8108950 2024-04-25T16:30:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a81a42c0 con 0x7f55a8108950 2024-04-25T16:30:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f559f7fe640 1 -- 172.21.15.98:0/889690553 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55a0004cc0 con 0x7f55a8108950 2024-04-25T16:30:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f559f7fe640 1 -- 172.21.15.98:0/889690553 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0010690 con 0x7f55a8108950 2024-04-25T16:30:05.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.814+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a81a4780 con 0x7f55a8108950 2024-04-25T16:30:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.815+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a81a4a40 con 0x7f55a8108950 2024-04-25T16:30:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.817+0000 7f559f7fe640 1 -- 172.21.15.98:0/889690553 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f55a0005000 con 0x7f55a8108950 2024-04-25T16:30:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.818+0000 7f559f7fe640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55940afaf0 0x7f55940b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.818+0000 7f55ae1b1640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55940afaf0 0x7f55940b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.818+0000 7f559f7fe640 1 -- 172.21.15.98:0/889690553 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f55a00deef0 con 0x7f55a8108950 2024-04-25T16:30:05.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.818+0000 7f55ae1b1640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55940afaf0 0x7f55940b2000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f55a8103610 tx=0x7f559800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:05.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:05.822+0000 7f559f7fe640 1 -- 172.21.15.98:0/889690553 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f55a0053ff0 con 0x7f55a8108950 2024-04-25T16:30:06.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.007+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55a81a4c10 con 0x7f55a8108950 2024-04-25T16:30:06.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.008+0000 7f559f7fe640 1 -- 172.21.15.98:0/889690553 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) ==== 74+0+18467 (secure 0 0 0) 0x7f55a00a7f70 con 0x7f55a8108950 2024-04-25T16:30:06.008 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:06.008 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":73,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:29:21.989091+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":67,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55940afaf0 msgr2=0x7f55940b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55940afaf0 0x7f55940b2000 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f55a8103610 tx=0x7f559800b390 comp rx=0 tx=0).stop 2024-04-25T16:30:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8108950 msgr2=0x7f55a80fe670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8108950 0x7f55a80fe670 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f55a000eb10 tx=0x7f55a000c2d0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 shutdown_connections 2024-04-25T16:30:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55940afaf0 0x7f55940b2000 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f55a8108950 0x7f55a80fe670 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55a8103c50 0x7f55a80fe130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 --2- 172.21.15.98:0/889690553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f55a8102a10 0x7f55a80ffa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 >> 172.21.15.98:0/889690553 conn(0x7f55a80fdc00 msgr2=0x7f55a8109f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 shutdown_connections 2024-04-25T16:30:06.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.010+0000 7f55b0415640 1 -- 172.21.15.98:0/889690553 wait complete. 2024-04-25T16:30:06.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:05 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] up:boot 2024-04-25T16:30:06.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:05 smithi173 ceph-mon[32556]: fsmap 12 up:standby 2024-04-25T16:30:06.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:05 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:30:06.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:05 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3482996306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:06.021 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:30:06.021 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:30:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 -- 172.21.15.98:0/3752445840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70103c20 msgr2=0x7f9b70104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 --2- 172.21.15.98:0/3752445840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70103c20 0x7f9b70104030 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009900 tx=0x7f9b5c02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 -- 172.21.15.98:0/3752445840 shutdown_connections 2024-04-25T16:30:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 --2- 172.21.15.98:0/3752445840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b70108920 0x7f9b7010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 --2- 172.21.15.98:0/3752445840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70103c20 0x7f9b70104030 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 --2- 172.21.15.98:0/3752445840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b701029e0 0x7f9b701036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 -- 172.21.15.98:0/3752445840 >> 172.21.15.98:0/3752445840 conn(0x7f9b700fdbd0 msgr2=0x7f9b70100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:06.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 -- 172.21.15.98:0/3752445840 shutdown_connections 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 -- 172.21.15.98:0/3752445840 wait complete. 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 Processor -- start 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 -- start start 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b701029e0 0x7f9b70199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b70103c20 0x7f9b70199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.133+0000 7f9b7718f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70108920 0x7f9b701a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7718f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7010e390 con 0x7f9b70108920 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7718f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7010e210 con 0x7f9b70103c20 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7718f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7010e510 con 0x7f9b701029e0 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70108920 0x7f9b701a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70108920 0x7f9b701a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48932/0 (socket says 172.21.15.98:48932) 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 -- 172.21.15.98:0/4000605166 learned_addr learned my addr 172.21.15.98:0/4000605166 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:06.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b74f2b640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b701029e0 0x7f9b70199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b6ffff640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b70103c20 0x7f9b70199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 -- 172.21.15.98:0/4000605166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b701029e0 msgr2=0x7f9b70199340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b701029e0 0x7f9b70199340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 -- 172.21.15.98:0/4000605166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b70103c20 msgr2=0x7f9b70199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b70103c20 0x7f9b70199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 -- 172.21.15.98:0/4000605166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b701a0e80 con 0x7f9b70108920 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7572c640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70108920 0x7f9b701a0810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800eaa0 tx=0x7f9b6800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b6dffb640 1 -- 172.21.15.98:0/4000605166 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6800cdb0 con 0x7f9b70108920 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b6dffb640 1 -- 172.21.15.98:0/4000605166 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b68005020 con 0x7f9b70108920 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b701a1170 con 0x7f9b70108920 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b6dffb640 1 -- 172.21.15.98:0/4000605166 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b680116d0 con 0x7f9b70108920 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.134+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b701a16c0 con 0x7f9b70108920 2024-04-25T16:30:06.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.135+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b70107730 con 0x7f9b70108920 2024-04-25T16:30:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.137+0000 7f9b6dffb640 1 -- 172.21.15.98:0/4000605166 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9b680051d0 con 0x7f9b70108920 2024-04-25T16:30:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.137+0000 7f9b6dffb640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9b600afb70 0x7f9b600b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.137+0000 7f9b74f2b640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9b600afb70 0x7f9b600b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.137+0000 7f9b6dffb640 1 -- 172.21.15.98:0/4000605166 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7f9b680deb10 con 0x7f9b70108920 2024-04-25T16:30:06.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.138+0000 7f9b74f2b640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9b600afb70 0x7f9b600b2080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9b64005fd0 tx=0x7f9b6400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:06.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.142+0000 7f9b6dffb640 1 -- 172.21.15.98:0/4000605166 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9b68016030 con 0x7f9b70108920 2024-04-25T16:30:06.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:05 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] up:boot 2024-04-25T16:30:06.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:05 smithi203 ceph-mon[33115]: fsmap 12 up:standby 2024-04-25T16:30:06.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:05 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:30:06.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:05 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3482996306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:06.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:05 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] up:boot 2024-04-25T16:30:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:05 smithi098 ceph-mon[29162]: fsmap 12 up:standby 2024-04-25T16:30:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:05 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:30:06.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:05 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3482996306' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:06.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.340+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9b7002dd50 con 0x7f9b70108920 2024-04-25T16:30:06.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.341+0000 7f9b6dffb640 1 -- 172.21.15.98:0/4000605166 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) ==== 93+0+10 (secure 0 0 0) 0x7f9b680a7540 con 0x7f9b70108920 2024-04-25T16:30:06.341 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9b600afb70 msgr2=0x7f9b600b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9b600afb70 0x7f9b600b2080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9b64005fd0 tx=0x7f9b6400b410 comp rx=0 tx=0).stop 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70108920 msgr2=0x7f9b701a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70108920 0x7f9b701a0810 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9b6800eaa0 tx=0x7f9b6800ef60 comp rx=0 tx=0).stop 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 shutdown_connections 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9b600afb70 0x7f9b600b2080 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9b70108920 0x7f9b701a0810 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b70103c20 0x7f9b70199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 --2- 172.21.15.98:0/4000605166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b701029e0 0x7f9b70199340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 >> 172.21.15.98:0/4000605166 conn(0x7f9b700fdbd0 msgr2=0x7f9b70104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:06.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 shutdown_connections 2024-04-25T16:30:06.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.343+0000 7f9b7718f640 1 -- 172.21.15.98:0/4000605166 wait complete. 2024-04-25T16:30:06.354 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T16:30:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 -- 172.21.15.98:0/2111075749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc102250 msgr2=0x7fe8dc10bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 --2- 172.21.15.98:0/2111075749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc102250 0x7fe8dc10bcb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4009ca0 tx=0x7fe8d402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 -- 172.21.15.98:0/2111075749 shutdown_connections 2024-04-25T16:30:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 --2- 172.21.15.98:0/2111075749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc102250 0x7fe8dc10bcb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 --2- 172.21.15.98:0/2111075749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8dc101750 0x7fe8dc101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 --2- 172.21.15.98:0/2111075749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8dc1050a0 0x7fe8dc1054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 -- 172.21.15.98:0/2111075749 >> 172.21.15.98:0/2111075749 conn(0x7fe8dc0fd370 msgr2=0x7fe8dc0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 -- 172.21.15.98:0/2111075749 shutdown_connections 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.472+0000 7fe8e356c640 1 -- 172.21.15.98:0/2111075749 wait complete. 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e356c640 1 Processor -- start 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e356c640 1 -- start start 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e356c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8dc101750 0x7fe8dc19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e356c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8dc102250 0x7fe8dc19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e356c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc1050a0 0x7fe8dc1a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e356c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc10f530 con 0x7fe8dc1050a0 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e356c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc10f3b0 con 0x7fe8dc102250 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e356c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe8dc10f6b0 con 0x7fe8dc101750 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1b09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc1050a0 0x7fe8dc1a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e0b07640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8dc102250 0x7fe8dc19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1b09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc1050a0 0x7fe8dc1a1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48946/0 (socket says 172.21.15.98:48946) 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e0b07640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8dc102250 0x7fe8dc19a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:32866/0 (socket says 172.21.15.98:32866) 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1b09640 1 -- 172.21.15.98:0/4177312840 learned_addr learned my addr 172.21.15.98:0/4177312840 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1308640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8dc101750 0x7fe8dc19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1b09640 1 -- 172.21.15.98:0/4177312840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8dc101750 msgr2=0x7fe8dc19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1b09640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8dc101750 0x7fe8dc19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1b09640 1 -- 172.21.15.98:0/4177312840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8dc102250 msgr2=0x7fe8dc19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1b09640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8dc102250 0x7fe8dc19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e1b09640 1 -- 172.21.15.98:0/4177312840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8dc1a1bd0 con 0x7fe8dc1050a0 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.473+0000 7fe8e0b07640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8dc102250 0x7fe8dc19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.474+0000 7fe8e1b09640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc1050a0 0x7fe8dc1a1560 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4031040 tx=0x7fe8d4003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.474+0000 7fe8d27fc640 1 -- 172.21.15.98:0/4177312840 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4003d60 con 0x7fe8dc1050a0 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.474+0000 7fe8d27fc640 1 -- 172.21.15.98:0/4177312840 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe8d40044d0 con 0x7fe8dc1050a0 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.474+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8dc1a1e60 con 0x7fe8dc1050a0 2024-04-25T16:30:06.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.474+0000 7fe8d27fc640 1 -- 172.21.15.98:0/4177312840 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4030c50 con 0x7fe8dc1050a0 2024-04-25T16:30:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.474+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8dc1a22c0 con 0x7fe8dc1050a0 2024-04-25T16:30:06.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.475+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8dc06a1c0 con 0x7fe8dc1050a0 2024-04-25T16:30:06.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.476+0000 7fe8d27fc640 1 -- 172.21.15.98:0/4177312840 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fe8d4004040 con 0x7fe8dc1050a0 2024-04-25T16:30:06.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.477+0000 7fe8d27fc640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe8c80afaf0 0x7fe8c80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.477+0000 7fe8e1308640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe8c80afaf0 0x7fe8c80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.478+0000 7fe8d27fc640 1 -- 172.21.15.98:0/4177312840 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fe8d40ffd20 con 0x7fe8dc1050a0 2024-04-25T16:30:06.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.478+0000 7fe8e1308640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe8c80afaf0 0x7fe8c80b2000 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc06b320 tx=0x7fe8d8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:06.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.481+0000 7fe8d27fc640 1 -- 172.21.15.98:0/4177312840 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8dc06a1c0 con 0x7fe8dc1050a0 2024-04-25T16:30:06.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.702+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe8dc02dc10 con 0x7fe8dc1050a0 2024-04-25T16:30:06.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.702+0000 7fe8d27fc640 1 -- 172.21.15.98:0/4177312840 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) ==== 84+0+65 (secure 0 0 0) 0x7fe8d40c8d00 con 0x7fe8dc1050a0 2024-04-25T16:30:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:06.702 INFO:teuthology.orchestra.run.smithi098.stdout:[ 2024-04-25T16:30:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:30:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "poolnum": 1, 2024-04-25T16:30:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: "poolname": ".mgr" 2024-04-25T16:30:06.702 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:30:06.703 INFO:teuthology.orchestra.run.smithi098.stdout:] 2024-04-25T16:30:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe8c80afaf0 msgr2=0x7fe8c80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe8c80afaf0 0x7fe8c80b2000 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc06b320 tx=0x7fe8d8005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc1050a0 msgr2=0x7fe8dc1a1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc1050a0 0x7fe8dc1a1560 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4031040 tx=0x7fe8d4003970 comp rx=0 tx=0).stop 2024-04-25T16:30:06.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 shutdown_connections 2024-04-25T16:30:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe8c80afaf0 0x7fe8c80b2000 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe8dc1050a0 0x7fe8dc1a1560 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8dc102250 0x7fe8dc19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 --2- 172.21.15.98:0/4177312840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8dc101750 0x7fe8dc19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 >> 172.21.15.98:0/4177312840 conn(0x7fe8dc0fd370 msgr2=0x7fe8dc107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 shutdown_connections 2024-04-25T16:30:06.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.704+0000 7fe8e356c640 1 -- 172.21.15.98:0/4177312840 wait complete. 2024-04-25T16:30:06.715 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T16:30:06.715 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T16:30:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 -- 172.21.15.98:0/4093483229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c1050b0 msgr2=0x7fee5c1054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 --2- 172.21.15.98:0/4093483229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c1050b0 0x7fee5c1054c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fee4c009900 tx=0x7fee4c02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 -- 172.21.15.98:0/4093483229 shutdown_connections 2024-04-25T16:30:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 --2- 172.21.15.98:0/4093483229 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee5c102260 0x7fee5c10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 --2- 172.21.15.98:0/4093483229 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5c101760 0x7fee5c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 --2- 172.21.15.98:0/4093483229 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c1050b0 0x7fee5c1054c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 -- 172.21.15.98:0/4093483229 >> 172.21.15.98:0/4093483229 conn(0x7fee5c0fd380 msgr2=0x7fee5c0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 -- 172.21.15.98:0/4093483229 shutdown_connections 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.826+0000 7fee61cbd640 1 -- 172.21.15.98:0/4093483229 wait complete. 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee61cbd640 1 Processor -- start 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee61cbd640 1 -- start start 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee61cbd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c101760 0x7fee5c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee61cbd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5c102260 0x7fee5c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee61cbd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee5c1050b0 0x7fee5c1a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee61cbd640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c10f4b0 con 0x7fee5c101760 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee61cbd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c10f330 con 0x7fee5c102260 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee61cbd640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fee5c10f630 con 0x7fee5c1050b0 2024-04-25T16:30:06.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee5b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c101760 0x7fee5c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee5b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c101760 0x7fee5c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48954/0 (socket says 172.21.15.98:48954) 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee5b7fe640 1 -- 172.21.15.98:0/222624070 learned_addr learned my addr 172.21.15.98:0/222624070 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.827+0000 7fee5bfff640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee5c1050b0 0x7fee5c1a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee5b7fe640 1 -- 172.21.15.98:0/222624070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee5c1050b0 msgr2=0x7fee5c1a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee5b7fe640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee5c1050b0 0x7fee5c1a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee5affd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5c102260 0x7fee5c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee5b7fe640 1 -- 172.21.15.98:0/222624070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5c102260 msgr2=0x7fee5c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee5b7fe640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5c102260 0x7fee5c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee5b7fe640 1 -- 172.21.15.98:0/222624070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5c1a1ce0 con 0x7fee5c101760 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee5affd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5c102260 0x7fee5c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee5b7fe640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c101760 0x7fee5c19a210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fee4c005760 tx=0x7fee4c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee58ff9640 1 -- 172.21.15.98:0/222624070 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c03f030 con 0x7fee5c101760 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee58ff9640 1 -- 172.21.15.98:0/222624070 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fee4c003a80 con 0x7fee5c101760 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee5c1a1f70 con 0x7fee5c101760 2024-04-25T16:30:06.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee58ff9640 1 -- 172.21.15.98:0/222624070 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee4c035070 con 0x7fee5c101760 2024-04-25T16:30:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.828+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee5c1a2370 con 0x7fee5c101760 2024-04-25T16:30:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.829+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee5c108a40 con 0x7fee5c101760 2024-04-25T16:30:06.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.830+0000 7fee58ff9640 1 -- 172.21.15.98:0/222624070 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fee4c042030 con 0x7fee5c101760 2024-04-25T16:30:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.831+0000 7fee58ff9640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fee440afaf0 0x7fee440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.831+0000 7fee5affd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fee440afaf0 0x7fee440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.832+0000 7fee58ff9640 1 -- 172.21.15.98:0/222624070 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7127+0+0 (secure 0 0 0) 0x7fee4c0ff320 con 0x7fee5c101760 2024-04-25T16:30:06.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.832+0000 7fee5affd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fee440afaf0 0x7fee440b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fee5c19b2b0 tx=0x7fee50005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:06.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:06.835+0000 7fee58ff9640 1 -- 172.21.15.98:0/222624070 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee4c0c8300 con 0x7fee5c101760 2024-04-25T16:30:07.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/889690553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4000605166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.003 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:06 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4177312840' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T16:30:07.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.015+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fee5c02dbf0 con 0x7fee5c101760 2024-04-25T16:30:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/889690553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4000605166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:06 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4177312840' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T16:30:07.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:07.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/889690553' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:07.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4000605166' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:07.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:07.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:06 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4177312840' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T16:30:07.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.959+0000 7fee58ff9640 1 -- 172.21.15.98:0/222624070 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v74) ==== 138+0+0 (secure 0 0 0) 0x7fee4c0d0380 con 0x7fee5c101760 2024-04-25T16:30:07.959 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_metadata' created 2024-04-25T16:30:07.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.961+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fee440afaf0 msgr2=0x7fee440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:07.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.961+0000 7fee61cbd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fee440afaf0 0x7fee440b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fee5c19b2b0 tx=0x7fee50005e70 comp rx=0 tx=0).stop 2024-04-25T16:30:07.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c101760 msgr2=0x7fee5c19a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c101760 0x7fee5c19a210 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fee4c005760 tx=0x7fee4c004270 comp rx=0 tx=0).stop 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 shutdown_connections 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fee440afaf0 0x7fee440b2000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee5c1050b0 0x7fee5c1a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee5c102260 0x7fee5c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 --2- 172.21.15.98:0/222624070 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee5c101760 0x7fee5c19a210 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 >> 172.21.15.98:0/222624070 conn(0x7fee5c0fd380 msgr2=0x7fee5c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 shutdown_connections 2024-04-25T16:30:07.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:07.962+0000 7fee61cbd640 1 -- 172.21.15.98:0/222624070 wait complete. 2024-04-25T16:30:07.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T16:30:08.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 -- 172.21.15.98:0/3779744811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c108940 msgr2=0x7f436c10ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 --2- 172.21.15.98:0/3779744811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c108940 0x7f436c10ad60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4364009ca0 tx=0x7f436402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 -- 172.21.15.98:0/3779744811 shutdown_connections 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 --2- 172.21.15.98:0/3779744811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c108940 0x7f436c10ad60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 --2- 172.21.15.98:0/3779744811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f436c103c40 0x7f436c104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 --2- 172.21.15.98:0/3779744811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c102a00 0x7f436c103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 -- 172.21.15.98:0/3779744811 >> 172.21.15.98:0/3779744811 conn(0x7f436c0fdbf0 msgr2=0x7f436c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 -- 172.21.15.98:0/3779744811 shutdown_connections 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 -- 172.21.15.98:0/3779744811 wait complete. 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.092+0000 7f43712ee640 1 Processor -- start 2024-04-25T16:30:08.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f43712ee640 1 -- start start 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f43712ee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f436c102a00 0x7f436c198fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f43712ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c103c40 0x7f436c1994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f43712ee640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c108940 0x7f436c1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f43712ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f436c10e430 con 0x7f436c103c40 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f43712ee640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f436c10e2b0 con 0x7f436c108940 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f43712ee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f436c10e5b0 con 0x7f436c102a00 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c103c40 0x7f436c1994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436a7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c103c40 0x7f436c1994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48970/0 (socket says 172.21.15.98:48970) 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436a7fc640 1 -- 172.21.15.98:0/1473699563 learned_addr learned my addr 172.21.15.98:0/1473699563 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436affd640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f436c102a00 0x7f436c198fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:08.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436b7fe640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c108940 0x7f436c1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436a7fc640 1 -- 172.21.15.98:0/1473699563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f436c102a00 msgr2=0x7f436c198fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436a7fc640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f436c102a00 0x7f436c198fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436a7fc640 1 -- 172.21.15.98:0/1473699563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c108940 msgr2=0x7f436c1a0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436a7fc640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c108940 0x7f436c1a0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436a7fc640 1 -- 172.21.15.98:0/1473699563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f436c1a0ae0 con 0x7f436c103c40 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.093+0000 7f436b7fe640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c108940 0x7f436c1a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.094+0000 7f436a7fc640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c103c40 0x7f436c1994e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f436000d770 tx=0x7f436000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.094+0000 7f4353fff640 1 -- 172.21.15.98:0/1473699563 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360017070 con 0x7f436c103c40 2024-04-25T16:30:08.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.094+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f436c1a0dd0 con 0x7f436c103c40 2024-04-25T16:30:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.094+0000 7f4353fff640 1 -- 172.21.15.98:0/1473699563 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43600054b0 con 0x7f436c103c40 2024-04-25T16:30:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.094+0000 7f4353fff640 1 -- 172.21.15.98:0/1473699563 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360004c60 con 0x7f436c103c40 2024-04-25T16:30:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.094+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f436c1a1230 con 0x7f436c103c40 2024-04-25T16:30:08.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.095+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f436c107750 con 0x7f436c103c40 2024-04-25T16:30:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.096+0000 7f4353fff640 1 -- 172.21.15.98:0/1473699563 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4360004e00 con 0x7f436c103c40 2024-04-25T16:30:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.097+0000 7f4353fff640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43540afaf0 0x7f43540b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.097+0000 7f436affd640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43540afaf0 0x7f43540b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.097+0000 7f4353fff640 1 -- 172.21.15.98:0/1473699563 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7497+0+0 (secure 0 0 0) 0x7f43600de710 con 0x7f436c103c40 2024-04-25T16:30:08.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.098+0000 7f436affd640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43540afaf0 0x7f43540b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f436c103600 tx=0x7f435c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:08.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.101+0000 7f4353fff640 1 -- 172.21.15.98:0/1473699563 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43600a7620 con 0x7f436c103c40 2024-04-25T16:30:08.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:07 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/222624070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T16:30:08.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:08.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:08.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:08.205 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:08.206 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:08.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:07 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/222624070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T16:30:08.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:08.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:08.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:07 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/222624070' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T16:30:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:30:08.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:08.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.287+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f436c02db50 con 0x7f436c103c40 2024-04-25T16:30:08.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.966+0000 7f4353fff640 1 -- 172.21.15.98:0/1473699563 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v75) ==== 171+0+0 (secure 0 0 0) 0x7f43600a6f20 con 0x7f436c103c40 2024-04-25T16:30:08.967 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data' created 2024-04-25T16:30:08.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.970+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43540afaf0 msgr2=0x7f43540b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:08.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.970+0000 7f43712ee640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43540afaf0 0x7f43540b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f436c103600 tx=0x7f435c005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:08.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.970+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c103c40 msgr2=0x7f436c1994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:08.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.970+0000 7f43712ee640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c103c40 0x7f436c1994e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f436000d770 tx=0x7f436000dc30 comp rx=0 tx=0).stop 2024-04-25T16:30:08.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.970+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 shutdown_connections 2024-04-25T16:30:08.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.971+0000 7f43712ee640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43540afaf0 0x7f43540b2000 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.971+0000 7f43712ee640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c108940 0x7f436c1a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.971+0000 7f43712ee640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f436c103c40 0x7f436c1994e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.971+0000 7f43712ee640 1 --2- 172.21.15.98:0/1473699563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f436c102a00 0x7f436c198fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.971+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 >> 172.21.15.98:0/1473699563 conn(0x7f436c0fdbf0 msgr2=0x7f436c104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.971+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 shutdown_connections 2024-04-25T16:30:08.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:08.971+0000 7f43712ee640 1 -- 172.21.15.98:0/1473699563 wait complete. 2024-04-25T16:30:08.993 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T16:30:09.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 -- 172.21.15.98:0/811213380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f41045f0 msgr2=0x7f29f4106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:09.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 --2- 172.21.15.98:0/811213380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f41045f0 0x7f29f4106a10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009900 tx=0x7f29e402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:09.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 -- 172.21.15.98:0/811213380 shutdown_connections 2024-04-25T16:30:09.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 --2- 172.21.15.98:0/811213380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29f41098b0 0x7f29f410bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 --2- 172.21.15.98:0/811213380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29f4106f50 0x7f29f4109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 --2- 172.21.15.98:0/811213380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f41045f0 0x7f29f4106a10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 -- 172.21.15.98:0/811213380 >> 172.21.15.98:0/811213380 conn(0x7f29f40fdbf0 msgr2=0x7f29f41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 -- 172.21.15.98:0/811213380 shutdown_connections 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.111+0000 7f29f9734640 1 -- 172.21.15.98:0/811213380 wait complete. 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.112+0000 7f29f9734640 1 Processor -- start 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.112+0000 7f29f9734640 1 -- start start 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.112+0000 7f29f9734640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29f41045f0 0x7f29f419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.112+0000 7f29f9734640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f4106f50 0x7f29f419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.112+0000 7f29f9734640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29f41098b0 0x7f29f41a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.112+0000 7f29f9734640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f29f410f490 con 0x7f29f4106f50 2024-04-25T16:30:09.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.112+0000 7f29f9734640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f29f410f310 con 0x7f29f41098b0 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.112+0000 7f29f9734640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f29f410f610 con 0x7f29f41045f0 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f4106f50 0x7f29f419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f4106f50 0x7f29f419a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48980/0 (socket says 172.21.15.98:48980) 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 -- 172.21.15.98:0/2355802355 learned_addr learned my addr 172.21.15.98:0/2355802355 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f37fe640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29f41098b0 0x7f29f41a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f2ffd640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29f41045f0 0x7f29f419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 -- 172.21.15.98:0/2355802355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29f41045f0 msgr2=0x7f29f419a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29f41045f0 0x7f29f419a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 -- 172.21.15.98:0/2355802355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29f41098b0 msgr2=0x7f29f41a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29f41098b0 0x7f29f41a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 -- 172.21.15.98:0/2355802355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f41a1c90 con 0x7f29f4106f50 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f27fc640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f4106f50 0x7f29f419a690 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f29e800e9f0 tx=0x7f29e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:09.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29dbfff640 1 -- 172.21.15.98:0/2355802355 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e800cd80 con 0x7f29f4106f50 2024-04-25T16:30:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29dbfff640 1 -- 172.21.15.98:0/2355802355 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f29e80108c0 con 0x7f29f4106f50 2024-04-25T16:30:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f41a1f80 con 0x7f29f4106f50 2024-04-25T16:30:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29dbfff640 1 -- 172.21.15.98:0/2355802355 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e80187c0 con 0x7f29f4106f50 2024-04-25T16:30:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.113+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f41a23f0 con 0x7f29f4106f50 2024-04-25T16:30:09.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.114+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f40ff710 con 0x7f29f4106f50 2024-04-25T16:30:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.116+0000 7f29dbfff640 1 -- 172.21.15.98:0/2355802355 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f29e8010430 con 0x7f29f4106f50 2024-04-25T16:30:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.116+0000 7f29dbfff640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29dc0afaf0 0x7f29dc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.116+0000 7f29f2ffd640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29dc0afaf0 0x7f29dc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.116+0000 7f29dbfff640 1 -- 172.21.15.98:0/2355802355 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7879+0+0 (secure 0 0 0) 0x7f29e80df290 con 0x7f29f4106f50 2024-04-25T16:30:09.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.117+0000 7f29f2ffd640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29dc0afaf0 0x7f29dc0b2000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f29e4005ab0 tx=0x7f29e4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:09.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.120+0000 7f29dbfff640 1 -- 172.21.15.98:0/2355802355 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29e80a79b0 con 0x7f29f4106f50 2024-04-25T16:30:09.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:08 smithi098 ceph-mon[29162]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:08 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/222624070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T16:30:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:08 smithi098 ceph-mon[29162]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T16:30:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:08 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1473699563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T16:30:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:09.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:09.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:08 smithi173 ceph-mon[32556]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:08 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/222624070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T16:30:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:08 smithi173 ceph-mon[32556]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T16:30:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:08 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1473699563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T16:30:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:09.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:09.341+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f29f4063810 con 0x7f29f4106f50 2024-04-25T16:30:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:08 smithi203 ceph-mon[33115]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:08 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/222624070' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T16:30:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:08 smithi203 ceph-mon[33115]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T16:30:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:08 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1473699563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-25T16:30:09.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:09.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1473699563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2355802355' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T16:30:10.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:09 smithi098 ceph-mon[29162]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:30:10.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1473699563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T16:30:10.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T16:30:10.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:10.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:10.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2355802355' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T16:30:10.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:09 smithi173 ceph-mon[32556]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:30:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.335+0000 7f29dbfff640 1 -- 172.21.15.98:0/2355802355 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f29e80b0080 con 0x7f29f4106f50 2024-04-25T16:30:10.335 INFO:teuthology.orchestra.run.smithi098.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.337+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29dc0afaf0 msgr2=0x7f29dc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.337+0000 7f29f9734640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29dc0afaf0 0x7f29dc0b2000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f29e4005ab0 tx=0x7f29e4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.337+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f4106f50 msgr2=0x7f29f419a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.337+0000 7f29f9734640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f4106f50 0x7f29f419a690 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f29e800e9f0 tx=0x7f29e800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.337+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 shutdown_connections 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.337+0000 7f29f9734640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f29dc0afaf0 0x7f29dc0b2000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.337+0000 7f29f9734640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f29f41098b0 0x7f29f41a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.337+0000 7f29f9734640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f29f4106f50 0x7f29f419a690 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.338+0000 7f29f9734640 1 --2- 172.21.15.98:0/2355802355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f29f41045f0 0x7f29f419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.338+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 >> 172.21.15.98:0/2355802355 conn(0x7f29f40fdbf0 msgr2=0x7f29f4106650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:10.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.338+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 shutdown_connections 2024-04-25T16:30:10.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.338+0000 7f29f9734640 1 -- 172.21.15.98:0/2355802355 wait complete. 2024-04-25T16:30:10.352 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T16:30:10.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T16:30:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1473699563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-25T16:30:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T16:30:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:30:10.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:30:10.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:30:10.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2355802355' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T16:30:10.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:09 smithi203 ceph-mon[33115]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:30:10.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 -- 172.21.15.98:0/334975273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 msgr2=0x7ff39810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 --2- 172.21.15.98:0/334975273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 0x7ff39810ad40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff390009ca0 tx=0x7ff39002a8e0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 -- 172.21.15.98:0/334975273 shutdown_connections 2024-04-25T16:30:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 --2- 172.21.15.98:0/334975273 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 0x7ff39810ad40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 --2- 172.21.15.98:0/334975273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff398103c20 0x7ff398104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 --2- 172.21.15.98:0/334975273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3981029e0 0x7ff3981036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 -- 172.21.15.98:0/334975273 >> 172.21.15.98:0/334975273 conn(0x7ff3980fdbd0 msgr2=0x7ff398100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 -- 172.21.15.98:0/334975273 shutdown_connections 2024-04-25T16:30:10.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.466+0000 7ff39e440640 1 -- 172.21.15.98:0/334975273 wait complete. 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39e440640 1 Processor -- start 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39e440640 1 -- start start 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39e440640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3981029e0 0x7ff39806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39e440640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff398103c20 0x7ff39819f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39e440640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 0x7ff3981a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39e440640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff39810e490 con 0x7ff398108920 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39e440640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff39810e310 con 0x7ff398103c20 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39e440640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff39810e610 con 0x7ff3981029e0 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39c9dd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 0x7ff3981a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39c9dd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 0x7ff3981a1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48984/0 (socket says 172.21.15.98:48984) 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39c9dd640 1 -- 172.21.15.98:0/1082459974 learned_addr learned my addr 172.21.15.98:0/1082459974 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:10.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff397fff640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3981029e0 0x7ff39806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff3977fe640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff398103c20 0x7ff39819f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39c9dd640 1 -- 172.21.15.98:0/1082459974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3981029e0 msgr2=0x7ff39806a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39c9dd640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3981029e0 0x7ff39806a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39c9dd640 1 -- 172.21.15.98:0/1082459974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff398103c20 msgr2=0x7ff39819f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39c9dd640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff398103c20 0x7ff39819f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.467+0000 7ff39c9dd640 1 -- 172.21.15.98:0/1082459974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3981a2080 con 0x7ff398108920 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.468+0000 7ff39c9dd640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 0x7ff3981a1860 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff390006010 tx=0x7ff390003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.468+0000 7ff3957fa640 1 -- 172.21.15.98:0/1082459974 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff390004240 con 0x7ff398108920 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.468+0000 7ff3957fa640 1 -- 172.21.15.98:0/1082459974 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff3900305e0 con 0x7ff398108920 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.468+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3981a2310 con 0x7ff398108920 2024-04-25T16:30:10.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.468+0000 7ff3957fa640 1 -- 172.21.15.98:0/1082459974 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39003b660 con 0x7ff398108920 2024-04-25T16:30:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.468+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3981a28e0 con 0x7ff398108920 2024-04-25T16:30:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.469+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff398107730 con 0x7ff398108920 2024-04-25T16:30:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.470+0000 7ff3957fa640 1 -- 172.21.15.98:0/1082459974 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff390030ad0 con 0x7ff398108920 2024-04-25T16:30:10.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.471+0000 7ff3957fa640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff3800afaf0 0x7ff3800b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.471+0000 7ff397fff640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff3800afaf0 0x7ff3800b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.471+0000 7ff3957fa640 1 -- 172.21.15.98:0/1082459974 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7979+0+0 (secure 0 0 0) 0x7ff390100340 con 0x7ff398108920 2024-04-25T16:30:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.471+0000 7ff397fff640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff3800afaf0 0x7ff3800b2000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff3981035e0 tx=0x7ff38c009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:10.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.474+0000 7ff3957fa640 1 -- 172.21.15.98:0/1082459974 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3900c9070 con 0x7ff398108920 2024-04-25T16:30:10.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.658+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7ff39806c2a0 con 0x7ff398108920 2024-04-25T16:30:10.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.974+0000 7ff3957fa640 1 -- 172.21.15.98:0/1082459974 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v77) ==== 157+0+0 (secure 0 0 0) 0x7ff3900c8970 con 0x7ff398108920 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff3800afaf0 msgr2=0x7ff3800b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff3800afaf0 0x7ff3800b2000 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff3981035e0 tx=0x7ff38c009110 comp rx=0 tx=0).stop 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 msgr2=0x7ff3981a1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 0x7ff3981a1860 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff390006010 tx=0x7ff390003f10 comp rx=0 tx=0).stop 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 shutdown_connections 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff3800afaf0 0x7ff3800b2000 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff398108920 0x7ff3981a1860 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff398103c20 0x7ff39819f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 --2- 172.21.15.98:0/1082459974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3981029e0 0x7ff39806a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:10.977 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 >> 172.21.15.98:0/1082459974 conn(0x7ff3980fdbd0 msgr2=0x7ff398104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 shutdown_connections 2024-04-25T16:30:10.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:10.977+0000 7ff39e440640 1 -- 172.21.15.98:0/1082459974 wait complete. 2024-04-25T16:30:10.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T16:30:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 -- 172.21.15.98:0/2878479677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201080e0 msgr2=0x7f412010a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 --2- 172.21.15.98:0/2878479677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201080e0 0x7f412010a500 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4118009ca0 tx=0x7f411802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 -- 172.21.15.98:0/2878479677 shutdown_connections 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 --2- 172.21.15.98:0/2878479677 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201080e0 0x7f412010a500 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 --2- 172.21.15.98:0/2878479677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41201033e0 0x7f41201037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 --2- 172.21.15.98:0/2878479677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41201021a0 0x7f4120102ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 -- 172.21.15.98:0/2878479677 >> 172.21.15.98:0/2878479677 conn(0x7f41200fd370 msgr2=0x7f41200ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 -- 172.21.15.98:0/2878479677 shutdown_connections 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.104+0000 7f4126d72640 1 -- 172.21.15.98:0/2878479677 wait complete. 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4126d72640 1 Processor -- start 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4126d72640 1 -- start start 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4126d72640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41201021a0 0x7f41200fe320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4126d72640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201033e0 0x7f412019ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4126d72640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41201080e0 0x7f41201a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4126d72640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f412010dbc0 con 0x7f41201033e0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4126d72640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f412010da40 con 0x7f41201080e0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4126d72640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f412010dd40 con 0x7f41201021a0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f411ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201033e0 0x7f412019ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f411ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201033e0 0x7f412019ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48992/0 (socket says 172.21.15.98:48992) 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f411ffff640 1 -- 172.21.15.98:0/414390622 learned_addr learned my addr 172.21.15.98:0/414390622 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f4124b0e640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41201021a0 0x7f41200fe320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f412530f640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41201080e0 0x7f41201a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f411ffff640 1 -- 172.21.15.98:0/414390622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41201021a0 msgr2=0x7f41200fe320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.105+0000 7f411ffff640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41201021a0 0x7f41200fe320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f411ffff640 1 -- 172.21.15.98:0/414390622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41201080e0 msgr2=0x7f41201a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f411ffff640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41201080e0 0x7f41201a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f411ffff640 1 -- 172.21.15.98:0/414390622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41201a1be0 con 0x7f41201033e0 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f412530f640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41201080e0 0x7f41201a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f411ffff640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201033e0 0x7f412019ee10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f410c00d950 tx=0x7f410c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f411dffb640 1 -- 172.21.15.98:0/414390622 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c017070 con 0x7f41201033e0 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f411dffb640 1 -- 172.21.15.98:0/414390622 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f410c009c30 con 0x7f41201033e0 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41201a1ed0 con 0x7f41201033e0 2024-04-25T16:30:11.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f411dffb640 1 -- 172.21.15.98:0/414390622 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f410c00f620 con 0x7f41201033e0 2024-04-25T16:30:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.106+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41201a2340 con 0x7f41201033e0 2024-04-25T16:30:11.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.107+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41201077e0 con 0x7f41201033e0 2024-04-25T16:30:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.108+0000 7f411dffb640 1 -- 172.21.15.98:0/414390622 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f410c010460 con 0x7f41201033e0 2024-04-25T16:30:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.109+0000 7f411dffb640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41100afaf0 0x7f41100b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.109+0000 7f4124b0e640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41100afaf0 0x7f41100b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.109+0000 7f411dffb640 1 -- 172.21.15.98:0/414390622 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8277+0+0 (secure 0 0 0) 0x7f410c0de860 con 0x7f41201033e0 2024-04-25T16:30:11.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.109+0000 7f4124b0e640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41100afaf0 0x7f41100b2000 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4114005fd0 tx=0x7f411400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:11.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.113+0000 7f411dffb640 1 -- 172.21.15.98:0/414390622 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f410c0a7410 con 0x7f41201033e0 2024-04-25T16:30:11.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:10 smithi098 ceph-mon[29162]: pgmap v276: 97 pgs: 22 creating+peering, 71 unknown, 4 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:11.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:10 smithi098 ceph-mon[29162]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T16:30:11.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:10 smithi098 ceph-mon[29162]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:30:11.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:10 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2355802355' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T16:30:11.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:10 smithi098 ceph-mon[29162]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T16:30:11.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:10 smithi098 ceph-mon[29162]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:30:11.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:10 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1082459974' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T16:30:11.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:10 smithi173 ceph-mon[32556]: pgmap v276: 97 pgs: 22 creating+peering, 71 unknown, 4 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:11.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:10 smithi173 ceph-mon[32556]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T16:30:11.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:10 smithi173 ceph-mon[32556]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:30:11.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:10 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2355802355' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T16:30:11.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:10 smithi173 ceph-mon[32556]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T16:30:11.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:10 smithi173 ceph-mon[32556]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:30:11.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:10 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1082459974' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T16:30:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:11.292+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f412002db50 con 0x7f41201033e0 2024-04-25T16:30:11.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:10 smithi203 ceph-mon[33115]: pgmap v276: 97 pgs: 22 creating+peering, 71 unknown, 4 active+clean; 577 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:11.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:10 smithi203 ceph-mon[33115]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T16:30:11.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:10 smithi203 ceph-mon[33115]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:30:11.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:10 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2355802355' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T16:30:11.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:10 smithi203 ceph-mon[33115]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T16:30:11.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:10 smithi203 ceph-mon[33115]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:30:11.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:10 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1082459974' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-25T16:30:12.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:11 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1082459974' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T16:30:12.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:11 smithi098 ceph-mon[29162]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T16:30:12.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:11 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/414390622' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T16:30:12.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:11 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] up:active 2024-04-25T16:30:12.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:11 smithi098 ceph-mon[29162]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:12.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:11 smithi098 ceph-mon[29162]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:12.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:11 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1082459974' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T16:30:12.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:11 smithi173 ceph-mon[32556]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T16:30:12.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:11 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/414390622' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T16:30:12.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:11 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] up:active 2024-04-25T16:30:12.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:11 smithi173 ceph-mon[32556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:12.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:11 smithi173 ceph-mon[32556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:12.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:11 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1082459974' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-25T16:30:12.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:11 smithi203 ceph-mon[33115]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T16:30:12.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:11 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/414390622' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T16:30:12.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:11 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] up:active 2024-04-25T16:30:12.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:11 smithi203 ceph-mon[33115]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:12.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:11 smithi203 ceph-mon[33115]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:12.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.978+0000 7f411dffb640 1 -- 172.21.15.98:0/414390622 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v79) ==== 228+0+0 (secure 0 0 0) 0x7f410c009de0 con 0x7f41201033e0 2024-04-25T16:30:12.978 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'cephfs_data_ec' created 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.980+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41100afaf0 msgr2=0x7f41100b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.980+0000 7f4126d72640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41100afaf0 0x7f41100b2000 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4114005fd0 tx=0x7f411400b390 comp rx=0 tx=0).stop 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201033e0 msgr2=0x7f412019ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201033e0 0x7f412019ee10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f410c00d950 tx=0x7f410c00de10 comp rx=0 tx=0).stop 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 shutdown_connections 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41100afaf0 0x7f41100b2000 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41201080e0 0x7f41201a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f41201033e0 0x7f412019ee10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 --2- 172.21.15.98:0/414390622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41201021a0 0x7f41200fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 >> 172.21.15.98:0/414390622 conn(0x7f41200fd370 msgr2=0x7f41201006c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:12.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 shutdown_connections 2024-04-25T16:30:12.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:12.981+0000 7f4126d72640 1 -- 172.21.15.98:0/414390622 wait complete. 2024-04-25T16:30:13.000 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T16:30:13.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 -- 172.21.15.98:0/3876024383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 msgr2=0x7fba5c10bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 --2- 172.21.15.98:0/3876024383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 0x7fba5c10bca0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fba54009ca0 tx=0x7fba5402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 -- 172.21.15.98:0/3876024383 shutdown_connections 2024-04-25T16:30:13.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 --2- 172.21.15.98:0/3876024383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 0x7fba5c10bca0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 --2- 172.21.15.98:0/3876024383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c101740 0x7fba5c101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 --2- 172.21.15.98:0/3876024383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba5c105090 0x7fba5c1054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 -- 172.21.15.98:0/3876024383 >> 172.21.15.98:0/3876024383 conn(0x7fba5c0fd340 msgr2=0x7fba5c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 -- 172.21.15.98:0/3876024383 shutdown_connections 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 -- 172.21.15.98:0/3876024383 wait complete. 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 Processor -- start 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 -- start start 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c101740 0x7fba5c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.118+0000 7fba631b2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 0x7fba5c19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba631b2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba5c105090 0x7fba5c1a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba631b2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c10f520 con 0x7fba5c102240 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba631b2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c10f3a0 con 0x7fba5c105090 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba631b2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba5c10f6a0 con 0x7fba5c101740 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 0x7fba5c19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 0x7fba5c19a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49010/0 (socket says 172.21.15.98:49010) 2024-04-25T16:30:13.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 -- 172.21.15.98:0/1023306663 learned_addr learned my addr 172.21.15.98:0/1023306663 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba60f4e640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c101740 0x7fba5c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba6174f640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba5c105090 0x7fba5c1a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 -- 172.21.15.98:0/1023306663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c101740 msgr2=0x7fba5c19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c101740 0x7fba5c19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 -- 172.21.15.98:0/1023306663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba5c105090 msgr2=0x7fba5c1a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba5c105090 0x7fba5c1a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 -- 172.21.15.98:0/1023306663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba5c1a1bb0 con 0x7fba5c102240 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba6174f640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba5c105090 0x7fba5c1a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba5bfff640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 0x7fba5c19a660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fba5000d770 tx=0x7fba5000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba59ffb640 1 -- 172.21.15.98:0/1023306663 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50017070 con 0x7fba5c102240 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba59ffb640 1 -- 172.21.15.98:0/1023306663 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba500054b0 con 0x7fba5c102240 2024-04-25T16:30:13.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba5c1a1ea0 con 0x7fba5c102240 2024-04-25T16:30:13.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.119+0000 7fba59ffb640 1 -- 172.21.15.98:0/1023306663 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba50004c60 con 0x7fba5c102240 2024-04-25T16:30:13.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.120+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba5c1a2300 con 0x7fba5c102240 2024-04-25T16:30:13.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.120+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba5c10aa30 con 0x7fba5c102240 2024-04-25T16:30:13.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.122+0000 7fba59ffb640 1 -- 172.21.15.98:0/1023306663 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fba50004e00 con 0x7fba5c102240 2024-04-25T16:30:13.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.122+0000 7fba59ffb640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba440afaf0 0x7fba440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:13.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.122+0000 7fba60f4e640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba440afaf0 0x7fba440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:13.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.122+0000 7fba59ffb640 1 -- 172.21.15.98:0/1023306663 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8770+0+0 (secure 0 0 0) 0x7fba500de8e0 con 0x7fba5c102240 2024-04-25T16:30:13.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.123+0000 7fba60f4e640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba440afaf0 0x7fba440b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fba5c069bf0 tx=0x7fba4c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:13.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.127+0000 7fba59ffb640 1 -- 172.21.15.98:0/1023306663 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fba500a72a0 con 0x7fba5c102240 2024-04-25T16:30:13.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:12 smithi098 ceph-mon[29162]: pgmap v279: 97 pgs: 22 creating+peering, 54 unknown, 21 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-25T16:30:13.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:12 smithi098 ceph-mon[29162]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:30:13.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:12 smithi098 ceph-mon[29162]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T16:30:13.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:12 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:30:13.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:12 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/414390622' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T16:30:13.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:12 smithi173 ceph-mon[32556]: pgmap v279: 97 pgs: 22 creating+peering, 54 unknown, 21 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-25T16:30:13.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:12 smithi173 ceph-mon[32556]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:30:13.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:12 smithi173 ceph-mon[32556]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T16:30:13.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:12 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:30:13.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:12 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/414390622' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T16:30:13.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.305+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fba5c02db50 con 0x7fba5c102240 2024-04-25T16:30:13.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:12 smithi203 ceph-mon[33115]: pgmap v279: 97 pgs: 22 creating+peering, 54 unknown, 21 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 1 op/s 2024-04-25T16:30:13.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:12 smithi203 ceph-mon[33115]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:30:13.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:12 smithi203 ceph-mon[33115]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-25T16:30:13.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:12 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:30:13.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:12 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/414390622' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T16:30:13.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.979+0000 7fba59ffb640 1 -- 172.21.15.98:0/1023306663 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v80) ==== 169+0+0 (secure 0 0 0) 0x7fba500a6ba0 con 0x7fba5c102240 2024-04-25T16:30:13.980 INFO:teuthology.orchestra.run.smithi098.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T16:30:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.982+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba440afaf0 msgr2=0x7fba440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.982+0000 7fba631b2640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba440afaf0 0x7fba440b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fba5c069bf0 tx=0x7fba4c005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.982+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 msgr2=0x7fba5c19a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.982+0000 7fba631b2640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 0x7fba5c19a660 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fba5000d770 tx=0x7fba5000dc30 comp rx=0 tx=0).stop 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.983+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 shutdown_connections 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.983+0000 7fba631b2640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba440afaf0 0x7fba440b2000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.983+0000 7fba631b2640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba5c105090 0x7fba5c1a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.983+0000 7fba631b2640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba5c102240 0x7fba5c19a660 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.983+0000 7fba631b2640 1 --2- 172.21.15.98:0/1023306663 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c101740 0x7fba5c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.983+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 >> 172.21.15.98:0/1023306663 conn(0x7fba5c0fd340 msgr2=0x7fba5c107c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.983+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 shutdown_connections 2024-04-25T16:30:13.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:13.983+0000 7fba631b2640 1 -- 172.21.15.98:0/1023306663 wait complete. 2024-04-25T16:30:14.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.117+0000 7fd54a620640 1 -- 172.21.15.98:0/1053267368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 msgr2=0x7fd544106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.117+0000 7fd54a620640 1 --2- 172.21.15.98:0/1053267368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 0x7fd544106a10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd534009900 tx=0x7fd53402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.117+0000 7fd54a620640 1 -- 172.21.15.98:0/1053267368 shutdown_connections 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.117+0000 7fd54a620640 1 --2- 172.21.15.98:0/1053267368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5441098b0 0x7fd54410bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.117+0000 7fd54a620640 1 --2- 172.21.15.98:0/1053267368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd544106f50 0x7fd544109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.117+0000 7fd54a620640 1 --2- 172.21.15.98:0/1053267368 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 0x7fd544106a10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.117+0000 7fd54a620640 1 -- 172.21.15.98:0/1053267368 >> 172.21.15.98:0/1053267368 conn(0x7fd5440fdbf0 msgr2=0x7fd5441000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 -- 172.21.15.98:0/1053267368 shutdown_connections 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 -- 172.21.15.98:0/1053267368 wait complete. 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 Processor -- start 2024-04-25T16:30:14.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 -- start start 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 0x7fd54419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd544106f50 0x7fd54419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5441098b0 0x7fd5441a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd54410f4b0 con 0x7fd5441045f0 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd54410f330 con 0x7fd544106f50 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.118+0000 7fd54a620640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd54410f630 con 0x7fd5441098b0 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 0x7fd54419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 0x7fd54419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49016/0 (socket says 172.21.15.98:49016) 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 -- 172.21.15.98:0/2956398045 learned_addr learned my addr 172.21.15.98:0/2956398045 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:14.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd5437fe640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd544106f50 0x7fd54419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd548bbd640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5441098b0 0x7fd5441a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 -- 172.21.15.98:0/2956398045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5441098b0 msgr2=0x7fd5441a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5441098b0 0x7fd5441a1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 -- 172.21.15.98:0/2956398045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd544106f50 msgr2=0x7fd54419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd544106f50 0x7fd54419a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 -- 172.21.15.98:0/2956398045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5441a1ce0 con 0x7fd5441045f0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd543fff640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 0x7fd54419a1a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd534031460 tx=0x7fd534004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd5417fa640 1 -- 172.21.15.98:0/2956398045 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53403f030 con 0x7fd5441045f0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd5417fa640 1 -- 172.21.15.98:0/2956398045 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd534003a80 con 0x7fd5441045f0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5441a1f70 con 0x7fd5441045f0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd5417fa640 1 -- 172.21.15.98:0/2956398045 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534035070 con 0x7fd5441045f0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.119+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5441a2380 con 0x7fd5441045f0 2024-04-25T16:30:14.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.120+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5440ff710 con 0x7fd5441045f0 2024-04-25T16:30:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.122+0000 7fd5417fa640 1 -- 172.21.15.98:0/2956398045 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd534042030 con 0x7fd5441045f0 2024-04-25T16:30:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.122+0000 7fd5417fa640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd52c0afb70 0x7fd52c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.123+0000 7fd5437fe640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd52c0afb70 0x7fd52c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:14.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.123+0000 7fd5417fa640 1 -- 172.21.15.98:0/2956398045 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8770+0+0 (secure 0 0 0) 0x7fd5340ffbc0 con 0x7fd5441045f0 2024-04-25T16:30:14.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.123+0000 7fd5437fe640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd52c0afb70 0x7fd52c0b2080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd538005fd0 tx=0x7fd538005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:14.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.126+0000 7fd5417fa640 1 -- 172.21.15.98:0/2956398045 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd5340c8520 con 0x7fd5441045f0 2024-04-25T16:30:14.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:14 smithi173 ceph-mon[32556]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:30:14.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:14 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1023306663' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T16:30:14.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:14 smithi173 ceph-mon[32556]: pgmap v282: 129 pgs: 18 creating+peering, 31 unknown, 80 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-25T16:30:14.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:14 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1023306663' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T16:30:14.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:14 smithi173 ceph-mon[32556]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T16:30:14.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:14.346+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fd54402db50 con 0x7fd5441045f0 2024-04-25T16:30:14.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:14 smithi203 ceph-mon[33115]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:30:14.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:14 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1023306663' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T16:30:14.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:14 smithi203 ceph-mon[33115]: pgmap v282: 129 pgs: 18 creating+peering, 31 unknown, 80 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-25T16:30:14.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:14 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1023306663' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T16:30:14.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:14 smithi203 ceph-mon[33115]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T16:30:14.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:14 smithi098 ceph-mon[29162]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:30:14.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:14 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1023306663' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-25T16:30:14.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:14 smithi098 ceph-mon[29162]: pgmap v282: 129 pgs: 18 creating+peering, 31 unknown, 80 active+clean; 578 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s wr, 9 op/s 2024-04-25T16:30:14.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:14 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1023306663' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-25T16:30:14.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:14 smithi098 ceph-mon[29162]: osdmap e80: 12 total, 12 up, 12 in 2024-04-25T16:30:15.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:15 smithi173 ceph-mon[32556]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:30:15.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:15 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2956398045' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T16:30:15.265 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:30:14.953+0000 7f8d10d3c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T16:30:15.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.330+0000 7fd5417fa640 1 -- 172.21.15.98:0/2956398045 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7fd5340c7e20 con 0x7fd5441045f0 2024-04-25T16:30:15.331 INFO:teuthology.orchestra.run.smithi098.stderr:added data pool 4 to fsmap 2024-04-25T16:30:15.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd52c0afb70 msgr2=0x7fd52c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd52c0afb70 0x7fd52c0b2080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd538005fd0 tx=0x7fd538005850 comp rx=0 tx=0).stop 2024-04-25T16:30:15.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 msgr2=0x7fd54419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 0x7fd54419a1a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd534031460 tx=0x7fd534004270 comp rx=0 tx=0).stop 2024-04-25T16:30:15.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 shutdown_connections 2024-04-25T16:30:15.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd52c0afb70 0x7fd52c0b2080 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd5441098b0 0x7fd5441a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd544106f50 0x7fd54419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 --2- 172.21.15.98:0/2956398045 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd5441045f0 0x7fd54419a1a0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 >> 172.21.15.98:0/2956398045 conn(0x7fd5440fdbf0 msgr2=0x7fd544106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 shutdown_connections 2024-04-25T16:30:15.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.333+0000 7fd54a620640 1 -- 172.21.15.98:0/2956398045 wait complete. 2024-04-25T16:30:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:15 smithi098 ceph-mon[29162]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:30:15.347 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:15 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2956398045' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T16:30:15.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:15.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:15 smithi203 ceph-mon[33115]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-25T16:30:15.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:15 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2956398045' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 -- 172.21.15.98:0/291995258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 msgr2=0x7fdcc81036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/291995258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 0x7fdcc81036c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8009900 tx=0x7fdcb802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 -- 172.21.15.98:0/291995258 shutdown_connections 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/291995258 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcc8108900 0x7fdcc810ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/291995258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc8103c00 0x7fdcc8104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/291995258 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 0x7fdcc81036c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 -- 172.21.15.98:0/291995258 >> 172.21.15.98:0/291995258 conn(0x7fdcc80fdbd0 msgr2=0x7fdcc8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 -- 172.21.15.98:0/291995258 shutdown_connections 2024-04-25T16:30:15.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 -- 172.21.15.98:0/291995258 wait complete. 2024-04-25T16:30:15.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.463+0000 7fdccc8f9640 1 Processor -- start 2024-04-25T16:30:15.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdccc8f9640 1 -- start start 2024-04-25T16:30:15.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdccc8f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 0x7fdcc8199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdccc8f9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc8103c00 0x7fdcc8199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdccc8f9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcc8108900 0x7fdcc81a0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdccc8f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc810e350 con 0x7fdcc81029c0 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdccc8f9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc810e1d0 con 0x7fdcc8108900 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdccc8f9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdcc810e4d0 con 0x7fdcc8103c00 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 0x7fdcc8199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 0x7fdcc8199040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48360/0 (socket says 172.21.15.98:48360) 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 -- 172.21.15.98:0/1050766351 learned_addr learned my addr 172.21.15.98:0/1050766351 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcbffff640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc8103c00 0x7fdcc8199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc6d9d640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcc8108900 0x7fdcc81a0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 -- 172.21.15.98:0/1050766351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc8103c00 msgr2=0x7fdcc8199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc8103c00 0x7fdcc8199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 -- 172.21.15.98:0/1050766351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcc8108900 msgr2=0x7fdcc81a0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcc8108900 0x7fdcc81a0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 -- 172.21.15.98:0/1050766351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcc81a0b80 con 0x7fdcc81029c0 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcc659c640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 0x7fdcc8199040 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8005760 tx=0x7fdcb8004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.464+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/1050766351 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb803f030 con 0x7fdcc81029c0 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.465+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/1050766351 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdcb8003a80 con 0x7fdcc81029c0 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.465+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcc81a0e10 con 0x7fdcc81029c0 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.465+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/1050766351 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb8035070 con 0x7fdcc81029c0 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.465+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcc81a1210 con 0x7fdcc81029c0 2024-04-25T16:30:15.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.466+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcc8107710 con 0x7fdcc81029c0 2024-04-25T16:30:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.467+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/1050766351 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fdcb8042030 con 0x7fdcc81029c0 2024-04-25T16:30:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.468+0000 7fdcbf7fe640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdcb40afaf0 0x7fdcb40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.468+0000 7fdcbffff640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdcb40afaf0 0x7fdcb40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.468+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/1050766351 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fdcb80ff4d0 con 0x7fdcc81029c0 2024-04-25T16:30:15.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.468+0000 7fdcbffff640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdcb40afaf0 0x7fdcb40b2000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdcc819a0e0 tx=0x7fdcb000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.473+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/1050766351 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdcb80c7820 con 0x7fdcc81029c0 2024-04-25T16:30:15.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.687+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcc80637d0 con 0x7fdcc81029c0 2024-04-25T16:30:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.688+0000 7fdcbf7fe640 1 -- 172.21.15.98:0/1050766351 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7fdcb80cbb20 con 0x7fdcc81029c0 2024-04-25T16:30:15.689 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:15.689 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:30:14.347856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24539,"qdb_cluster":[24539]},"id":1}]} 2024-04-25T16:30:15.689 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2024-04-25T16:30:15.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdcb40afaf0 msgr2=0x7fdcb40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdcb40afaf0 0x7fdcb40b2000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdcc819a0e0 tx=0x7fdcb000b390 comp rx=0 tx=0).stop 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 msgr2=0x7fdcc8199040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 0x7fdcc8199040 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdcb8005760 tx=0x7fdcb8004270 comp rx=0 tx=0).stop 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 shutdown_connections 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdcb40afaf0 0x7fdcb40b2000 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdcc8108900 0x7fdcc81a0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdcc8103c00 0x7fdcc8199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 --2- 172.21.15.98:0/1050766351 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdcc81029c0 0x7fdcc8199040 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.690+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 >> 172.21.15.98:0/1050766351 conn(0x7fdcc80fdbd0 msgr2=0x7fdcc81048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.691+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 shutdown_connections 2024-04-25T16:30:15.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.691+0000 7fdccc8f9640 1 -- 172.21.15.98:0/1050766351 wait complete. 2024-04-25T16:30:15.703 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:15.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 -- 172.21.15.98:0/720958384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 msgr2=0x7f94c0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 --2- 172.21.15.98:0/720958384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 0x7f94c0106a10 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f94b4009900 tx=0x7f94b402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:15.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 -- 172.21.15.98:0/720958384 shutdown_connections 2024-04-25T16:30:15.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 --2- 172.21.15.98:0/720958384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c01098b0 0x7f94c010bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.813 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 --2- 172.21.15.98:0/720958384 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94c0106f50 0x7f94c0109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 --2- 172.21.15.98:0/720958384 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 0x7f94c0106a10 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 -- 172.21.15.98:0/720958384 >> 172.21.15.98:0/720958384 conn(0x7f94c00fdbf0 msgr2=0x7f94c01000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 -- 172.21.15.98:0/720958384 shutdown_connections 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 -- 172.21.15.98:0/720958384 wait complete. 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 Processor -- start 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.810+0000 7f94c66d1640 1 -- start start 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94c66d1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 0x7f94c0103a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94c66d1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94c0106f50 0x7f94c01020e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94c66d1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c01098b0 0x7f94c0102620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94c66d1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94c010f4b0 con 0x7f94c01045f0 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94c66d1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94c010f330 con 0x7f94c01098b0 2024-04-25T16:30:15.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94c66d1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94c010f630 con 0x7f94c0106f50 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 0x7f94c0103a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 0x7f94c0103a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48374/0 (socket says 172.21.15.98:48374) 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 -- 172.21.15.98:0/1545913242 learned_addr learned my addr 172.21.15.98:0/1545913242 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94c4c6e640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c01098b0 0x7f94c0102620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bf7fe640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94c0106f50 0x7f94c01020e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 -- 172.21.15.98:0/1545913242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94c0106f50 msgr2=0x7f94c01020e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94c0106f50 0x7f94c01020e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 -- 172.21.15.98:0/1545913242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c01098b0 msgr2=0x7f94c0102620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c01098b0 0x7f94c0102620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 -- 172.21.15.98:0/1545913242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c0102d50 con 0x7f94c01045f0 2024-04-25T16:30:15.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bffff640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 0x7f94c0103a40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f94b4031460 tx=0x7f94b4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.811+0000 7f94bd7fa640 1 -- 172.21.15.98:0/1545913242 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b403f030 con 0x7f94c01045f0 2024-04-25T16:30:15.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.812+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c01a4200 con 0x7f94c01045f0 2024-04-25T16:30:15.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.812+0000 7f94bd7fa640 1 -- 172.21.15.98:0/1545913242 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94b4003a80 con 0x7f94c01045f0 2024-04-25T16:30:15.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.812+0000 7f94bd7fa640 1 -- 172.21.15.98:0/1545913242 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b4035070 con 0x7f94c01045f0 2024-04-25T16:30:15.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.812+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c01a4660 con 0x7f94c01045f0 2024-04-25T16:30:15.816 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.812+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94c00ff710 con 0x7f94c01045f0 2024-04-25T16:30:15.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.814+0000 7f94bd7fa640 1 -- 172.21.15.98:0/1545913242 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f94b4042030 con 0x7f94c01045f0 2024-04-25T16:30:15.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.814+0000 7f94bd7fa640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afaf0 0x7f94a80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:15.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.815+0000 7f94bf7fe640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afaf0 0x7f94a80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:15.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.815+0000 7f94bd7fa640 1 -- 172.21.15.98:0/1545913242 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f94b40ffb00 con 0x7f94c01045f0 2024-04-25T16:30:15.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.815+0000 7f94bf7fe640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afaf0 0x7f94a80b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f94b0005fd0 tx=0x7f94b0005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:15.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.818+0000 7f94bd7fa640 1 -- 172.21.15.98:0/1545913242 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94b40c84a0 con 0x7f94c01045f0 2024-04-25T16:30:15.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.997+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94c002dd20 con 0x7f94c01045f0 2024-04-25T16:30:15.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:15.998+0000 7f94bd7fa640 1 -- 172.21.15.98:0/1545913242 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7f94b40c7da0 con 0x7f94c01045f0 2024-04-25T16:30:15.999 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:15.999 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:16.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afaf0 msgr2=0x7f94a80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afaf0 0x7f94a80b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f94b0005fd0 tx=0x7f94b0005850 comp rx=0 tx=0).stop 2024-04-25T16:30:16.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 msgr2=0x7f94c0103a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 0x7f94c0103a40 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f94b4031460 tx=0x7f94b4004270 comp rx=0 tx=0).stop 2024-04-25T16:30:16.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 shutdown_connections 2024-04-25T16:30:16.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afaf0 0x7f94a80b2000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94c01098b0 0x7f94c0102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94c0106f50 0x7f94c01020e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 --2- 172.21.15.98:0/1545913242 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94c01045f0 0x7f94c0103a40 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 >> 172.21.15.98:0/1545913242 conn(0x7f94c00fdbf0 msgr2=0x7f94c0106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:16.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 shutdown_connections 2024-04-25T16:30:16.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.000+0000 7f94c66d1640 1 -- 172.21.15.98:0/1545913242 wait complete. 2024-04-25T16:30:16.014 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:16.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 -- 172.21.15.98:0/1150049460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8102a00 msgr2=0x7f86b8103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 --2- 172.21.15.98:0/1150049460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8102a00 0x7f86b8103700 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f86b4009900 tx=0x7f86b402a3f0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 -- 172.21.15.98:0/1150049460 shutdown_connections 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 --2- 172.21.15.98:0/1150049460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b8108940 0x7f86b810ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 --2- 172.21.15.98:0/1150049460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b8103c40 0x7f86b8104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 --2- 172.21.15.98:0/1150049460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8102a00 0x7f86b8103700 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 -- 172.21.15.98:0/1150049460 >> 172.21.15.98:0/1150049460 conn(0x7f86b80fdbf0 msgr2=0x7f86b81000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 -- 172.21.15.98:0/1150049460 shutdown_connections 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.120+0000 7f86c00e7640 1 -- 172.21.15.98:0/1150049460 wait complete. 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86c00e7640 1 Processor -- start 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86c00e7640 1 -- start start 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86c00e7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b8102a00 0x7f86b80758a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86c00e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8103c40 0x7f86b819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86c00e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b8108940 0x7f86b81a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86c00e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86b810e390 con 0x7f86b8103c40 2024-04-25T16:30:16.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86c00e7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86b810e210 con 0x7f86b8102a00 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86c00e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86b810e510 con 0x7f86b8108940 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bd682640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8103c40 0x7f86b819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bd682640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8103c40 0x7f86b819edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48388/0 (socket says 172.21.15.98:48388) 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bd682640 1 -- 172.21.15.98:0/3166006642 learned_addr learned my addr 172.21.15.98:0/3166006642 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bde83640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b8102a00 0x7f86b80758a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86be684640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b8108940 0x7f86b81a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bd682640 1 -- 172.21.15.98:0/3166006642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b8108940 msgr2=0x7f86b81a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bd682640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b8108940 0x7f86b81a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bd682640 1 -- 172.21.15.98:0/3166006642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b8102a00 msgr2=0x7f86b80758a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bd682640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b8102a00 0x7f86b80758a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.121+0000 7f86bd682640 1 -- 172.21.15.98:0/3166006642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b81a1b30 con 0x7f86b8103c40 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.122+0000 7f86bd682640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8103c40 0x7f86b819edf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f86a800c920 tx=0x7f86a800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.122+0000 7f86aeffd640 1 -- 172.21.15.98:0/3166006642 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8007c70 con 0x7f86b8103c40 2024-04-25T16:30:16.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.122+0000 7f86aeffd640 1 -- 172.21.15.98:0/3166006642 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86a8007e10 con 0x7f86b8103c40 2024-04-25T16:30:16.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.122+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b81a1e20 con 0x7f86b8103c40 2024-04-25T16:30:16.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.122+0000 7f86aeffd640 1 -- 172.21.15.98:0/3166006642 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8018710 con 0x7f86b8103c40 2024-04-25T16:30:16.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.122+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b81a2280 con 0x7f86b8103c40 2024-04-25T16:30:16.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.123+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86b8107750 con 0x7f86b8103c40 2024-04-25T16:30:16.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.124+0000 7f86aeffd640 1 -- 172.21.15.98:0/3166006642 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f86a800f450 con 0x7f86b8103c40 2024-04-25T16:30:16.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.125+0000 7f86aeffd640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86a40afaf0 0x7f86a40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.125+0000 7f86bde83640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86a40afaf0 0x7f86a40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.125+0000 7f86aeffd640 1 -- 172.21.15.98:0/3166006642 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f86a80df260 con 0x7f86b8103c40 2024-04-25T16:30:16.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.125+0000 7f86bde83640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86a40afaf0 0x7f86a40b2000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f86b4005760 tx=0x7f86b4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:16.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.128+0000 7f86aeffd640 1 -- 172.21.15.98:0/3166006642 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f86a8014030 con 0x7f86b8103c40 2024-04-25T16:30:16.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:16 smithi173 ceph-mon[32556]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T16:30:16.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:16 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2956398045' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T16:30:16.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:16 smithi173 ceph-mon[32556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:16.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:16 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1050766351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:16.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:16 smithi173 ceph-mon[32556]: pgmap v285: 129 pgs: 8 creating+peering, 18 unknown, 103 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-25T16:30:16.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:16 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1545913242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:16.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.307+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86b802dd20 con 0x7f86b8103c40 2024-04-25T16:30:16.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.309+0000 7f86aeffd640 1 -- 172.21.15.98:0/3166006642 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7f86a80a7c90 con 0x7f86b8103c40 2024-04-25T16:30:16.310 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:16.310 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86a40afaf0 msgr2=0x7f86a40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86a40afaf0 0x7f86a40b2000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f86b4005760 tx=0x7f86b4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8103c40 msgr2=0x7f86b819edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8103c40 0x7f86b819edf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f86a800c920 tx=0x7f86a800cde0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 shutdown_connections 2024-04-25T16:30:16.312 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86a40afaf0 0x7f86a40b2000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86b8108940 0x7f86b81a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f86b8103c40 0x7f86b819edf0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 --2- 172.21.15.98:0/3166006642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b8102a00 0x7f86b80758a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 >> 172.21.15.98:0/3166006642 conn(0x7f86b80fdbf0 msgr2=0x7f86b8104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 shutdown_connections 2024-04-25T16:30:16.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.311+0000 7f86c00e7640 1 -- 172.21.15.98:0/3166006642 wait complete. 2024-04-25T16:30:16.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:16 smithi098 ceph-mon[29162]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T16:30:16.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:16 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2956398045' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T16:30:16.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:16 smithi098 ceph-mon[29162]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:16.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:16 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1050766351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:16.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:16 smithi098 ceph-mon[29162]: pgmap v285: 129 pgs: 8 creating+peering, 18 unknown, 103 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-25T16:30:16.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:16 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1545913242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:16.339 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:16.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:16 smithi203 ceph-mon[33115]: osdmap e81: 12 total, 12 up, 12 in 2024-04-25T16:30:16.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:16 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2956398045' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-25T16:30:16.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:16 smithi203 ceph-mon[33115]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:16.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:16 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1050766351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:16.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:16 smithi203 ceph-mon[33115]: pgmap v285: 129 pgs: 8 creating+peering, 18 unknown, 103 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 9 op/s 2024-04-25T16:30:16.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:16 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1545913242' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 -- 172.21.15.98:0/2465834560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594108940 msgr2=0x7f759410ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 --2- 172.21.15.98:0/2465834560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594108940 0x7f759410ad60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f758c009ca0 tx=0x7f758c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 -- 172.21.15.98:0/2465834560 shutdown_connections 2024-04-25T16:30:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 --2- 172.21.15.98:0/2465834560 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594108940 0x7f759410ad60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 --2- 172.21.15.98:0/2465834560 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7594103c40 0x7f7594104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 --2- 172.21.15.98:0/2465834560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7594102a00 0x7f7594103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 -- 172.21.15.98:0/2465834560 >> 172.21.15.98:0/2465834560 conn(0x7f75940fdbf0 msgr2=0x7f75941000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 -- 172.21.15.98:0/2465834560 shutdown_connections 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.445+0000 7f75996b6640 1 -- 172.21.15.98:0/2465834560 wait complete. 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75996b6640 1 Processor -- start 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75996b6640 1 -- start start 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75996b6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7594102a00 0x7f7594075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75996b6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594103c40 0x7f759419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75996b6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7594108940 0x7f75941a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75996b6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f759410e430 con 0x7f7594103c40 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75996b6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f759410e2b0 con 0x7f7594102a00 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75996b6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f759410e5b0 con 0x7f7594108940 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f758bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594103c40 0x7f759419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f758bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594103c40 0x7f759419ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48404/0 (socket says 172.21.15.98:48404) 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f758bfff640 1 -- 172.21.15.98:0/3969301656 learned_addr learned my addr 172.21.15.98:0/3969301656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.446+0000 7f75937fe640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7594108940 0x7f75941a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f7592ffd640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7594102a00 0x7f7594075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f758bfff640 1 -- 172.21.15.98:0/3969301656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7594108940 msgr2=0x7f75941a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f758bfff640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7594108940 0x7f75941a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f758bfff640 1 -- 172.21.15.98:0/3969301656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7594102a00 msgr2=0x7f7594075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f758bfff640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7594102a00 0x7f7594075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f758bfff640 1 -- 172.21.15.98:0/3969301656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75941a1bf0 con 0x7f7594103c40 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f758bfff640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594103c40 0x7f759419ee80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f757c00d770 tx=0x7f757c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f7590ff9640 1 -- 172.21.15.98:0/3969301656 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c017070 con 0x7f7594103c40 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f7590ff9640 1 -- 172.21.15.98:0/3969301656 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f757c0054b0 con 0x7f7594103c40 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75941a1ee0 con 0x7f7594103c40 2024-04-25T16:30:16.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f7590ff9640 1 -- 172.21.15.98:0/3969301656 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c004c60 con 0x7f7594103c40 2024-04-25T16:30:16.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.447+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75941a2340 con 0x7f7594103c40 2024-04-25T16:30:16.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.448+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7594107750 con 0x7f7594103c40 2024-04-25T16:30:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.449+0000 7f7590ff9640 1 -- 172.21.15.98:0/3969301656 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f757c004e00 con 0x7f7594103c40 2024-04-25T16:30:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.450+0000 7f7590ff9640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f75800afaf0 0x7f75800b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.450+0000 7f7592ffd640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f75800afaf0 0x7f75800b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.450+0000 7f7590ff9640 1 -- 172.21.15.98:0/3969301656 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f757c0decf0 con 0x7f7594103c40 2024-04-25T16:30:16.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.451+0000 7f7592ffd640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f75800afaf0 0x7f75800b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7584005d00 tx=0x7f7584005c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:16.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.454+0000 7f7590ff9640 1 -- 172.21.15.98:0/3969301656 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f757c0a7690 con 0x7f7594103c40 2024-04-25T16:30:16.670 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.670+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f759402db50 con 0x7f7594103c40 2024-04-25T16:30:16.671 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.670+0000 7f7590ff9640 1 -- 172.21.15.98:0/3969301656 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11800 (secure 0 0 0) 0x7f757c0a6f90 con 0x7f7594103c40 2024-04-25T16:30:16.671 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:16.672 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24443,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24445,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24473,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24484,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:30:14.347856+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24539},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24539,"qdb_cluster":[24539]},"id":1}]} 2024-04-25T16:30:16.672 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 17 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f75800afaf0 msgr2=0x7f75800b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f75800afaf0 0x7f75800b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7584005d00 tx=0x7f7584005c50 comp rx=0 tx=0).stop 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594103c40 msgr2=0x7f759419ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594103c40 0x7f759419ee80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f757c00d770 tx=0x7f757c00dc30 comp rx=0 tx=0).stop 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 shutdown_connections 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f75800afaf0 0x7f75800b2000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7594108940 0x7f75941a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7594103c40 0x7f759419ee80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 --2- 172.21.15.98:0/3969301656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7594102a00 0x7f7594075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.674 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 >> 172.21.15.98:0/3969301656 conn(0x7f75940fdbf0 msgr2=0x7f7594104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:16.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 shutdown_connections 2024-04-25T16:30:16.675 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.673+0000 7f75996b6640 1 -- 172.21.15.98:0/3969301656 wait complete. 2024-04-25T16:30:16.685 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:16.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.791+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1779989187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f981043b0 msgr2=0x7f6f98102db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.791+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1779989187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f981043b0 0x7f6f98102db0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009900 tx=0x7f6f8c02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:16.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1779989187 shutdown_connections 2024-04-25T16:30:16.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1779989187 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f98108a00 0x7f6f9810ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1779989187 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f981043b0 0x7f6f98102db0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1779989187 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f98103ed0 0x7f6f981027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1779989187 >> 172.21.15.98:0/1779989187 conn(0x7f6f980fd330 msgr2=0x7f6f980ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1779989187 shutdown_connections 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1779989187 wait complete. 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 Processor -- start 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.792+0000 7f6f9cd38640 1 -- start start 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9cd38640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f98103ed0 0x7f6f98198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9cd38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f98108a00 0x7f6f981991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9cd38640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f9819e100 0x7f6f981a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9cd38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9810e3c0 con 0x7f6f98108a00 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9cd38640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9810e240 con 0x7f6f9819e100 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9cd38640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6f9810e540 con 0x7f6f98103ed0 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f98108a00 0x7f6f981991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f98108a00 0x7f6f981991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48416/0 (socket says 172.21.15.98:48416) 2024-04-25T16:30:16.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 -- 172.21.15.98:0/1268641071 learned_addr learned my addr 172.21.15.98:0/1268641071 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9659c640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f98103ed0 0x7f6f98198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f96d9d640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f9819e100 0x7f6f981a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 -- 172.21.15.98:0/1268641071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f98103ed0 msgr2=0x7f6f98198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f98103ed0 0x7f6f98198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 -- 172.21.15.98:0/1268641071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f9819e100 msgr2=0x7f6f981a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f9819e100 0x7f6f981a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 -- 172.21.15.98:0/1268641071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f981a0b10 con 0x7f6f98108a00 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f96d9d640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f9819e100 0x7f6f981a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f95d9b640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f98108a00 0x7f6f981991c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009230 tx=0x7f6f8c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f877fe640 1 -- 172.21.15.98:0/1268641071 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c03f030 con 0x7f6f98108a00 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f981a0da0 con 0x7f6f98108a00 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f877fe640 1 -- 172.21.15.98:0/1268641071 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6f8c003c90 con 0x7f6f98108a00 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f877fe640 1 -- 172.21.15.98:0/1268641071 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c035070 con 0x7f6f98108a00 2024-04-25T16:30:16.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.793+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f981a1220 con 0x7f6f98108a00 2024-04-25T16:30:16.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.794+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f98107100 con 0x7f6f98108a00 2024-04-25T16:30:16.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.796+0000 7f6f877fe640 1 -- 172.21.15.98:0/1268641071 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6f8c042030 con 0x7f6f98108a00 2024-04-25T16:30:16.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.797+0000 7f6f877fe640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6f800afaf0 0x7f6f800b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:16.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.797+0000 7f6f9659c640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6f800afaf0 0x7f6f800b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:16.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.797+0000 7f6f877fe640 1 -- 172.21.15.98:0/1268641071 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f6f8c0ff940 con 0x7f6f98108a00 2024-04-25T16:30:16.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.797+0000 7f6f9659c640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6f800afaf0 0x7f6f800b2000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6f9819a070 tx=0x7f6f88009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:16.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.801+0000 7f6f877fe640 1 -- 172.21.15.98:0/1268641071 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f8c0c8280 con 0x7f6f98108a00 2024-04-25T16:30:16.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.980+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6f980637a0 con 0x7f6f98108a00 2024-04-25T16:30:16.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.981+0000 7f6f877fe640 1 -- 172.21.15.98:0/1268641071 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7f6f8c0c7b80 con 0x7f6f98108a00 2024-04-25T16:30:16.983 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:16.983 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6f800afaf0 msgr2=0x7f6f800b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6f800afaf0 0x7f6f800b2000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6f9819a070 tx=0x7f6f88009680 comp rx=0 tx=0).stop 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f98108a00 msgr2=0x7f6f981991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f98108a00 0x7f6f981991c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009230 tx=0x7f6f8c0042f0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 shutdown_connections 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6f800afaf0 0x7f6f800b2000 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f9819e100 0x7f6f981a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6f98108a00 0x7f6f981991c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 --2- 172.21.15.98:0/1268641071 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f98103ed0 0x7f6f98198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:16.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 >> 172.21.15.98:0/1268641071 conn(0x7f6f980fd330 msgr2=0x7f6f980ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 shutdown_connections 2024-04-25T16:30:16.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:16.984+0000 7f6f9cd38640 1 -- 172.21.15.98:0/1268641071 wait complete. 2024-04-25T16:30:16.997 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T16:30:09.342794+0000', 'modified': '2024-04-25T16:30:14.347856+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24539}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.173:6839/2050448663', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2050448663}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2050448663}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24539, 'qdb_cluster': [24539]} 2024-04-25T16:30:16.997 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T16:30:16.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T16:30:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 -- 172.21.15.98:0/4261975935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68103b00 msgr2=0x7fca68104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/4261975935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68103b00 0x7fca68104800 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fca64009900 tx=0x7fca6402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:17.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 -- 172.21.15.98:0/4261975935 shutdown_connections 2024-04-25T16:30:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/4261975935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca68109a40 0x7fca6810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/4261975935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca68104d40 0x7fca68105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/4261975935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68103b00 0x7fca68104800 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 -- 172.21.15.98:0/4261975935 >> 172.21.15.98:0/4261975935 conn(0x7fca680fed10 msgr2=0x7fca681011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 -- 172.21.15.98:0/4261975935 shutdown_connections 2024-04-25T16:30:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.104+0000 7fca6f0ee640 1 -- 172.21.15.98:0/4261975935 wait complete. 2024-04-25T16:30:17.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6f0ee640 1 Processor -- start 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6f0ee640 1 -- start start 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6f0ee640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca68103b00 0x7fca6819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6f0ee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca68104d40 0x7fca6819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6f0ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68109a40 0x7fca681a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6f0ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fca6810f500 con 0x7fca68109a40 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6f0ee640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fca6810f380 con 0x7fca68103b00 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6f0ee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fca6810f680 con 0x7fca68104d40 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6d68b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68109a40 0x7fca681a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6d68b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68109a40 0x7fca681a16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48438/0 (socket says 172.21.15.98:48438) 2024-04-25T16:30:17.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6d68b640 1 -- 172.21.15.98:0/3190922599 learned_addr learned my addr 172.21.15.98:0/3190922599 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca5ffff640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca68104d40 0x7fca6819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca68103b00 0x7fca6819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6d68b640 1 -- 172.21.15.98:0/3190922599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca68104d40 msgr2=0x7fca6819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6d68b640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca68104d40 0x7fca6819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6d68b640 1 -- 172.21.15.98:0/3190922599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca68103b00 msgr2=0x7fca6819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6d68b640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca68103b00 0x7fca6819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.105+0000 7fca6d68b640 1 -- 172.21.15.98:0/3190922599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca681a1c80 con 0x7fca68109a40 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.106+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca68103b00 0x7fca6819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.106+0000 7fca6d68b640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68109a40 0x7fca681a16c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fca6000cce0 tx=0x7fca6000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.106+0000 7fca5dffb640 1 -- 172.21.15.98:0/3190922599 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca6000f450 con 0x7fca68109a40 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.106+0000 7fca5dffb640 1 -- 172.21.15.98:0/3190922599 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca6000fa80 con 0x7fca68109a40 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.106+0000 7fca6f0ee640 1 -- 172.21.15.98:0/3190922599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca681a1f70 con 0x7fca68109a40 2024-04-25T16:30:17.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.106+0000 7fca5dffb640 1 -- 172.21.15.98:0/3190922599 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca60018ad0 con 0x7fca68109a40 2024-04-25T16:30:17.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.106+0000 7fca6f0ee640 1 -- 172.21.15.98:0/3190922599 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca681a23d0 con 0x7fca68109a40 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.108+0000 7fca5dffb640 1 -- 172.21.15.98:0/3190922599 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fca6000f5f0 con 0x7fca68109a40 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.108+0000 7fca75fa0740 1 Processor -- start 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca75fa0740 1 -- start start 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 0x562fffd50370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd508f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd577f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca75fa0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562fffcb5740 con 0x562fff98e590 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca75fa0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562fffc513a0 con 0x562fff9863b0 2024-04-25T16:30:17.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca75fa0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562fffcb1700 con 0x562fff9af270 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd577f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd577f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48444/0 (socket says 172.21.15.98:48444) 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 -- 172.21.15.98:0/4094703768 learned_addr learned my addr 172.21.15.98:0/4094703768 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5dffb640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fca500afb70 0x7fca500b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca6d68b640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fca500afb70 0x7fca500b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 0x562fffd50370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 -- 172.21.15.98:0/4094703768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 msgr2=0x562fffd50370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 0x562fffd50370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 -- 172.21.15.98:0/4094703768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 msgr2=0x562fffd508f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd508f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5dffb640 1 -- 172.21.15.98:0/3190922599 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fca600df620 con 0x7fca68109a40 2024-04-25T16:30:17.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.109+0000 7fca5ffff640 1 -- 172.21.15.98:0/4094703768 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fffc6d750 con 0x562fff98e590 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.110+0000 7fca5ffff640 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd577f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fca58005c30 tx=0x7fca5800a770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0e46fea07873e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.110+0000 7fca6d68b640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fca500afb70 0x7fca500b2080 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fca6000ccb0 tx=0x7fca60090040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.110+0000 7fca4effd640 1 -- 172.21.15.98:0/4094703768 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca58019030 con 0x562fff98e590 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.110+0000 7fca4effd640 1 -- 172.21.15.98:0/4094703768 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca580110d0 con 0x562fff98e590 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.110+0000 7fca4effd640 1 -- 172.21.15.98:0/4094703768 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca58004c60 con 0x562fff98e590 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.110+0000 7fca75fa0740 1 -- 172.21.15.98:0/4094703768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 msgr2=0x562fffd577f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.110+0000 7fca75fa0740 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd577f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fca58005c30 tx=0x7fca5800a770 comp rx=0 tx=0).stop 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 -- 172.21.15.98:0/4094703768 shutdown_connections 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd577f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd508f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 --2- 172.21.15.98:0/4094703768 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 0x562fffd50370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 -- 172.21.15.98:0/4094703768 >> 172.21.15.98:0/4094703768 conn(0x562fff9a33c0 msgr2=0x562fffd4bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 -- 172.21.15.98:0/4094703768 shutdown_connections 2024-04-25T16:30:17.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 -- 172.21.15.98:0/4094703768 wait complete. 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 -- start start 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 Processor -- start 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd5a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 0x562fffd616e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562fffd58240 con 0x562fff98e590 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562fffc3cfe0 con 0x562fff9863b0 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.111+0000 7fca75fa0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562fffd583c0 con 0x562fff9af270 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca75fa0740 10 client.0 _ll_register_callbacks cb 0x562fffc53150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48450/0 (socket says 172.21.15.98:48450) 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 -- 172.21.15.98:0/3963380188 learned_addr learned my addr 172.21.15.98:0/3963380188 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 -- 172.21.15.98:0/3963380188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 msgr2=0x562fffd616e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:17.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca5ffff640 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 0x562fffd616e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 0x562fffd616e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca5ffff640 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd5a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 -- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 msgr2=0x562fffd5a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd5a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 -- 172.21.15.98:0/3963380188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fff9c72d0 con 0x562fff98e590 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca5ffff640 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd5a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5a780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fca64006010 tx=0x7fca640045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943b91bad4cd34a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca47fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca47fff640 1 -- 172.21.15.98:0/3963380188 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca64035040 con 0x562fff98e590 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca47fff640 1 -- 172.21.15.98:0/3963380188 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fca64003c40 con 0x562fff98e590 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca75fa0740 1 -- 172.21.15.98:0/3963380188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca580110d0 con 0x562fff98e590 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca47fff640 1 -- 172.21.15.98:0/3963380188 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fca64003e40 con 0x562fff98e590 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca75fa0740 10 client.14766 fetch_fsmap learned FSMap version 17 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca75fa0740 1 -- 172.21.15.98:0/3963380188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x562fffa916b0 con 0x562fff98e590 2024-04-25T16:30:17.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.112+0000 7fca47fff640 1 -- 172.21.15.98:0/3963380188 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca580110d0 con 0x562fff98e590 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca47fff640 1 -- 172.21.15.98:0/3963380188 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fca6403d500 con 0x562fff98e590 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca75fa0740 10 client.14766 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca75fa0740 10 client.14766 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca75fa0740 20 client.14766 populate_metadata read hostname 'smithi098' 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca75fa0740 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562fff96de40 0x562fffd68020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca75fa0740 4 client.14766 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca75fa0740 1 -- 172.21.15.98:0/3963380188 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562fffcc3350 con 0x562fff96de40 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca5ffff640 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562fff96de40 0x562fffd68020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca47fff640 10 client.14766 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.113+0000 7fca5ffff640 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562fff96de40 0x562fffd68020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f031edd8fb28943e server_cookie=95875ebb7e9db99f in_seq=0 out_seq=0 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.114+0000 7fca47fff640 1 -- 172.21.15.98:0/3963380188 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562fffcc3350 con 0x562fff96de40 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.114+0000 7fca47fff640 10 client.14766 handle_command_reply: tid=0 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.114+0000 7fca75fa0740 1 client.14766 shutdown 2024-04-25T16:30:17.123 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.114+0000 7fca75fa0740 1 -- 172.21.15.98:0/3963380188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 msgr2=0x562fffd5a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.114+0000 7fca75fa0740 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5a780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fca64006010 tx=0x7fca640045a0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 1 -- 172.21.15.98:0/3963380188 shutdown_connections 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562fff96de40 0x562fffd68020 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9af270 0x562fffd616e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca47fff640 0 client.14766 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5a780 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 1 --2- 172.21.15.98:0/3963380188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd5a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 1 -- 172.21.15.98:0/3963380188 >> 172.21.15.98:0/3963380188 conn(0x562fff9a33c0 msgr2=0x562fffd4ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 1 -- 172.21.15.98:0/3963380188 shutdown_connections 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 1 -- 172.21.15.98:0/3963380188 wait complete. 2024-04-25T16:30:17.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.115+0000 7fca75fa0740 20 client.14766 trim_cache size 0 max 16384 2024-04-25T16:30:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca75fa0740 1 Processor -- start 2024-04-25T16:30:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca75fa0740 1 -- start start 2024-04-25T16:30:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9a33c0 0x562fffd59c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd61600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd46a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca75fa0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562fffd66360 con 0x562fff98e590 2024-04-25T16:30:17.140 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca75fa0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562fffd66a30 con 0x562fff9863b0 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca75fa0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562fffd67100 con 0x562fff9a33c0 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca5ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd46a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca5ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd46a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48464/0 (socket says 172.21.15.98:48464) 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.133+0000 7fca5ffff640 1 -- 172.21.15.98:0/1435239340 learned_addr learned my addr 172.21.15.98:0/1435239340 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca5ffff640 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9a33c0 0x562fffd59c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd61600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca5ffff640 1 -- 172.21.15.98:0/1435239340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9a33c0 msgr2=0x562fffd59c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca5ffff640 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9a33c0 0x562fffd59c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca5ffff640 1 -- 172.21.15.98:0/1435239340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 msgr2=0x562fffd61600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca5ffff640 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd61600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca5ffff640 1 -- 172.21.15.98:0/1435239340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562fffd55090 con 0x562fff98e590 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca5ffff640 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd46a20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fca58005c00 tx=0x7fca5800a770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96017bfe1ee68c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca4e7fc640 1 -- 172.21.15.98:0/1435239340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca58019040 con 0x562fff98e590 2024-04-25T16:30:17.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca4e7fc640 1 -- 172.21.15.98:0/1435239340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca580110d0 con 0x562fff98e590 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.134+0000 7fca4e7fc640 1 -- 172.21.15.98:0/1435239340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca58029ab0 con 0x562fff98e590 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- 172.21.15.98:0/1435239340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 msgr2=0x562fffd46a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd46a20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fca58005c00 tx=0x7fca5800a770 comp rx=0 tx=0).stop 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- 172.21.15.98:0/1435239340 shutdown_connections 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd46a20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9863b0 0x562fffd61600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 --2- 172.21.15.98:0/1435239340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9a33c0 0x562fffd59c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- 172.21.15.98:0/1435239340 >> 172.21.15.98:0/1435239340 conn(0x562fffd53f10 msgr2=0x562fffd594f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- 172.21.15.98:0/1435239340 shutdown_connections 2024-04-25T16:30:17.142 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- 172.21.15.98:0/1435239340 wait complete. 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- start start 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 Processor -- start 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9863b0 0x562fffd5d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9a33c0 0x562fffd609a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562fffd67ea0 con 0x562fff98e590 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562fffd677d0 con 0x562fff9a33c0 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.135+0000 7fca75fa0740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562fffd68570 con 0x562fff9863b0 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca75fa0740 10 client.0 _ll_register_callbacks cb 0x7fca64004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48470/0 (socket says 172.21.15.98:48470) 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 -- 172.21.15.98:0/894375826 learned_addr learned my addr 172.21.15.98:0/894375826 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:17.143 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca5ffff640 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9a33c0 0x562fffd609a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca5ffff640 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9863b0 0x562fffd5d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 -- 172.21.15.98:0/894375826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9863b0 msgr2=0x562fffd5d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9863b0 0x562fffd5d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 -- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9a33c0 msgr2=0x562fffd609a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9a33c0 0x562fffd609a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 -- 172.21.15.98:0/894375826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562fffd60f60 con 0x562fff98e590 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca5ffff640 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9a33c0 0x562fffd609a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca6ce8a640 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5da60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x562fffd4cb30 tx=0x7fca6403d680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9920dd6c1508539 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca47fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca47fff640 1 -- 172.21.15.98:0/894375826 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca64035030 con 0x562fff98e590 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca47fff640 1 -- 172.21.15.98:0/894375826 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fca64031700 con 0x562fff98e590 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca75fa0740 1 -- 172.21.15.98:0/894375826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca580110d0 con 0x562fff98e590 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca47fff640 1 -- 172.21.15.98:0/894375826 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7fca640041c0 con 0x562fff98e590 2024-04-25T16:30:17.144 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca75fa0740 10 client.14772 fetch_fsmap learned FSMap version 17 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.136+0000 7fca75fa0740 1 -- 172.21.15.98:0/894375826 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x562fffdd1520 con 0x562fff98e590 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca47fff640 1 -- 172.21.15.98:0/894375826 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fca580110d0 con 0x562fff98e590 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca47fff640 1 -- 172.21.15.98:0/894375826 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7fca6404c840 con 0x562fff98e590 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca75fa0740 10 client.14772 fetch_fsmap finished waiting for FSMap version 17 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca75fa0740 10 client.14772 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca75fa0740 20 client.14772 populate_metadata read hostname 'smithi098' 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca75fa0740 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562fff9af270 0x562fffdd2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca75fa0740 4 client.14772 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca75fa0740 1 -- 172.21.15.98:0/894375826 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "status"}) -- 0x562fffcbd1c0 con 0x562fff9af270 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.137+0000 7fca5ffff640 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562fff9af270 0x562fffdd2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca47fff640 10 client.14772 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca5ffff640 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562fff9af270 0x562fffdd2c00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30743abefba96d36 server_cookie=feecbc6454d9ba2d in_seq=0 out_seq=0 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca47fff640 1 -- 172.21.15.98:0/894375826 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x562fffcbd1c0 con 0x562fff9af270 2024-04-25T16:30:17.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca47fff640 10 client.14772 handle_command_reply: tid=0 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 client.14772 shutdown 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 -- 172.21.15.98:0/894375826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 msgr2=0x562fffd5da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5da60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x562fffd4cb30 tx=0x7fca6403d680 comp rx=0 tx=0).stop 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 -- 172.21.15.98:0/894375826 shutdown_connections 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562fff9af270 0x562fffdd2c00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562fff9a33c0 0x562fffd609a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca47fff640 0 client.14772 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562fff98e590 0x562fffd5da60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 --2- 172.21.15.98:0/894375826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562fff9863b0 0x562fffd5d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 -- 172.21.15.98:0/894375826 >> 172.21.15.98:0/894375826 conn(0x562fffd53f10 msgr2=0x562fffd4b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:17.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 -- 172.21.15.98:0/894375826 shutdown_connections 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 1 -- 172.21.15.98:0/894375826 wait complete. 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.138+0000 7fca75fa0740 20 client.14772 trim_cache size 0 max 16384 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24539, 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:30:17.147 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 6.8023697370000002, 2024-04-25T16:30:17.148 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 17, 2024-04-25T16:30:17.148 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 81, 2024-04-25T16:30:17.148 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:30:17.148 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 12.523599754999999 2024-04-25T16:30:17.148 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 -- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fca500afb70 msgr2=0x7fca500b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fca500afb70 0x7fca500b2080 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fca6000ccb0 tx=0x7fca60090040 comp rx=0 tx=0).stop 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 -- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68109a40 msgr2=0x7fca681a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68109a40 0x7fca681a16c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fca6000cce0 tx=0x7fca6000a620 comp rx=0 tx=0).stop 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 -- 172.21.15.98:0/3190922599 shutdown_connections 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fca500afb70 0x7fca500b2080 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fca68109a40 0x7fca681a16c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca68104d40 0x7fca6819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 --2- 172.21.15.98:0/3190922599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca68103b00 0x7fca6819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 -- 172.21.15.98:0/3190922599 >> 172.21.15.98:0/3190922599 conn(0x7fca680fed10 msgr2=0x7fca68105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 -- 172.21.15.98:0/3190922599 shutdown_connections 2024-04-25T16:30:17.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:17.139+0000 7fca6f0ee640 1 -- 172.21.15.98:0/3190922599 wait complete. 2024-04-25T16:30:17.151 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T16:30:17.151 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T16:30:17.152 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T16:30:17.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:17 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3166006642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:17 smithi098 ceph-mon[29162]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T16:30:17.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:17 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:30:17.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:17 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3969301656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:17 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1268641071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:17 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3166006642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:17 smithi173 ceph-mon[32556]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T16:30:17.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:17 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:30:17.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:17 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3969301656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:17 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1268641071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.386 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:17.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:17 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3166006642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:17 smithi203 ceph-mon[33115]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-25T16:30:17.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:17 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:30:17.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:17 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3969301656' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:17.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:17 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1268641071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:18.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:18 smithi098 ceph-mon[29162]: pgmap v286: 129 pgs: 8 creating+peering, 6 unknown, 115 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-04-25T16:30:18.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:18 smithi203 ceph-mon[33115]: pgmap v286: 129 pgs: 8 creating+peering, 6 unknown, 115 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-04-25T16:30:18.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:18 smithi173 ceph-mon[32556]: pgmap v286: 129 pgs: 8 creating+peering, 6 unknown, 115 active+clean; 579 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 6 op/s 2024-04-25T16:30:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.010+0000 7fafcea0d700 1 -- 172.21.15.98:0/1992925952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc8109720 msgr2=0x7fafc8111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.010+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1992925952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc8109720 0x7fafc8111080 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fafc000b0d0 tx=0x7fafc000b3e0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.010+0000 7fafcea0d700 1 -- 172.21.15.98:0/1992925952 shutdown_connections 2024-04-25T16:30:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.010+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1992925952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafc81044a0 0x7fafc81048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.010+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1992925952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc8109720 0x7fafc8111080 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.010+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1992925952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc8108dd0 0x7fafc81091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.011+0000 7fafcea0d700 1 -- 172.21.15.98:0/1992925952 >> 172.21.15.98:0/1992925952 conn(0x7fafc80ffdb0 msgr2=0x7fafc8102200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.011+0000 7fafcea0d700 1 -- 172.21.15.98:0/1992925952 shutdown_connections 2024-04-25T16:30:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.011+0000 7fafcea0d700 1 -- 172.21.15.98:0/1992925952 wait complete. 2024-04-25T16:30:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.011+0000 7fafcea0d700 1 Processor -- start 2024-04-25T16:30:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.011+0000 7fafcea0d700 1 -- start start 2024-04-25T16:30:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafcea0d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc81044a0 0x7fafc81a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafcea0d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafc8108dd0 0x7fafc81a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafcea0d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc8109720 0x7fafc819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafcea0d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fafc8114220 con 0x7fafc81044a0 2024-04-25T16:30:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafcea0d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81140a0 con 0x7fafc8109720 2024-04-25T16:30:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafcea0d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fafc81143a0 con 0x7fafc8108dd0 2024-04-25T16:30:19.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc81044a0 0x7fafc81a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc81044a0 0x7fafc81a0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48494/0 (socket says 172.21.15.98:48494) 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 -- 172.21.15.98:0/1203863619 learned_addr learned my addr 172.21.15.98:0/1203863619 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc77fe700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafc8108dd0 0x7fafc81a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafccfaa700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc8109720 0x7fafc819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 -- 172.21.15.98:0/1203863619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafc8108dd0 msgr2=0x7fafc81a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafc8108dd0 0x7fafc81a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 -- 172.21.15.98:0/1203863619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc8109720 msgr2=0x7fafc819aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc8109720 0x7fafc819aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 -- 172.21.15.98:0/1203863619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc819b060 con 0x7fafc81044a0 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafccfaa700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc8109720 0x7fafc819aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc7fff700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc81044a0 0x7fafc81a0800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fafb000d980 tx=0x7fafb000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc57fa700 1 -- 172.21.15.98:0/1203863619 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb0017070 con 0x7fafc81044a0 2024-04-25T16:30:19.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc57fa700 1 -- 172.21.15.98:0/1203863619 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fafb0004830 con 0x7fafc81044a0 2024-04-25T16:30:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafc57fa700 1 -- 172.21.15.98:0/1203863619 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb00053a0 con 0x7fafc81044a0 2024-04-25T16:30:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.012+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc819b350 con 0x7fafc81044a0 2024-04-25T16:30:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.013+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc819b840 con 0x7fafc81044a0 2024-04-25T16:30:19.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.014+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc8051190 con 0x7fafc81044a0 2024-04-25T16:30:19.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.015+0000 7fafc57fa700 1 -- 172.21.15.98:0/1203863619 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fafb00049e0 con 0x7fafc81044a0 2024-04-25T16:30:19.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.016+0000 7fafc57fa700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafb40afb80 0x7fafb40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:19.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.016+0000 7fafc77fe700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafb40afb80 0x7fafb40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:19.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.016+0000 7fafc57fa700 1 -- 172.21.15.98:0/1203863619 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fafb00df1d0 con 0x7fafc81044a0 2024-04-25T16:30:19.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.016+0000 7fafc77fe700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafb40afb80 0x7fafb40b2090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fafb8005fd0 tx=0x7fafb800af20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:19.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.020+0000 7fafc57fa700 1 -- 172.21.15.98:0/1203863619 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafb0020020 con 0x7fafc81044a0 2024-04-25T16:30:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.264+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fafc802e510 con 0x7fafc81044a0 2024-04-25T16:30:19.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.264+0000 7fafc57fa700 1 -- 172.21.15.98:0/1203863619 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fafb0005540 con 0x7fafc81044a0 2024-04-25T16:30:19.266 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T16:30:19.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.266+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafb40afb80 msgr2=0x7fafb40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:19.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.266+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafb40afb80 0x7fafb40b2090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fafb8005fd0 tx=0x7fafb800af20 comp rx=0 tx=0).stop 2024-04-25T16:30:19.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.266+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc81044a0 msgr2=0x7fafc81a0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:19.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.266+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc81044a0 0x7fafc81a0800 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fafb000d980 tx=0x7fafb000dd40 comp rx=0 tx=0).stop 2024-04-25T16:30:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.267+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 shutdown_connections 2024-04-25T16:30:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.267+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fafb40afb80 0x7fafb40b2090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.267+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fafc8109720 0x7fafc819aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.267+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fafc81044a0 0x7fafc81a0800 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.267+0000 7fafcea0d700 1 --2- 172.21.15.98:0/1203863619 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafc8108dd0 0x7fafc81a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.267+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 >> 172.21.15.98:0/1203863619 conn(0x7fafc80ffdb0 msgr2=0x7fafc81085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.267+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 shutdown_connections 2024-04-25T16:30:19.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19.267+0000 7fafcea0d700 1 -- 172.21.15.98:0/1203863619 wait complete. 2024-04-25T16:30:19.579 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:19 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1203863619' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:30:19.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:19 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1203863619' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:30:19.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:19 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1203863619' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:30:19.864 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T16:30:19.865 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T16:30:19.865 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-25T16:30:19.865 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T16:30:19.865 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T16:30:19.865 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T16:30:19.865 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T16:30:19.865 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T16:30:19.865 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T16:30:19.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:19.865 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:30:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42236sec preferred_lft 42236sec 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::f29f:9dff:ddbc:4615/64 scope link noprefixroute 2024-04-25T16:30:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:30:19.883 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T16:30:19.883 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:19.883 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:30:19.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-04-25T16:30:19.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-04-25T16:30:19.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-04-25T16:30:19.883 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T16:30:19.883 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:30:19.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:20.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:20.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:20.062 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T16:30:20.085 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-25T16:30:20.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:20.087 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-25T16:30:20.149 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-25T16:30:20.149 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T16:30:20.149 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T16:30:20.149 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T16:30:20.149 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T16:30:20.150 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:20.150 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:30:20.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T16:30:20.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T16:30:20.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T16:30:20.150 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:30:20.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:20.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:20.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:20.411 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-25T16:30:20.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:20.427 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-04-25T16:30:20.481 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:20.481 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:30:20.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T16:30:20.482 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T16:30:20.482 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:30:20.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:20 smithi098 ceph-mon[29162]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 442 B/s wr, 1 op/s 2024-04-25T16:30:20.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:20.571 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:20.577 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T16:30:20.577 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:20.577 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:30:20.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T16:30:20.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T16:30:20.577 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T16:30:20.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T16:30:20.578 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T16:30:20.578 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:30:20.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:20.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:20 smithi203 ceph-mon[33115]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 442 B/s wr, 1 op/s 2024-04-25T16:30:20.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:20.707 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:30:20.707 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:30:20.707 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-25T16:30:20.708 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T16:30:20.708 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:30:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:30:20.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:20 smithi173 ceph-mon[32556]: pgmap v287: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 442 B/s wr, 1 op/s 2024-04-25T16:30:20.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:20.782 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T16:30:20.782 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T16:30:20.782 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T16:30:20.796 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T16:30:20.796 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T16:30:20.796 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T16:30:20.851 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-25T16:30:20.916 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T16:30:20.972 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-25T16:30:20.972 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-25T16:30:20.972 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-25T16:30:20.972 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-25T16:30:20.973 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/260d120b743fdefc47097bdfa51ca2592ce504e600e0b8aa9a7681e4cc8f09a3/merged 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c3e265ddc44ba607adedfa4bb7c4a1d5a16bf204960d9b3e7f03b229a2fd58e8/merged 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b11a75d862ab4ca91fa24f775f6dfcfe5f4198c0807f0f4909bf047ffb1350b9/merged 2024-04-25T16:30:20.974 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/39738d8aa159ba108f0d78e1529bb35664af715ac389a39fa6d060bcf363cfa4/merged 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b8b9dc7ee394ce0927571fb358b669ca734319cbfaf2342876758534c9e46e2d/merged 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3c1452187a1df91e01df6afd698ac4a4f426c127b144f2bce196417fec79813a/merged 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c0fbdfa79403832c61b1e9461210414679dd82f9527d126e63606fdb9abe4097/merged 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/7107c4da4556b4a7b2a6e9113780c8e9f29261d548251acae86d609e859597e3/merged 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/12e8d5140c0933c0da3677f0f5235e29dcbea42d6a1e700d2d28d98996739286/merged 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/8f45f93f747246be0f168199353f8c0eb210046bac0c70867ae9f250862cd38b/merged 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T16:30:20.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:20.975 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-25T16:30:21.027 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T16:30:21.027 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T16:30:21.070 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-25T16:30:21.092 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T16:30:21.148 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-25T16:30:21.148 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-25T16:30:21.149 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-25T16:30:21.150 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/260d120b743fdefc47097bdfa51ca2592ce504e600e0b8aa9a7681e4cc8f09a3/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c3e265ddc44ba607adedfa4bb7c4a1d5a16bf204960d9b3e7f03b229a2fd58e8/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b11a75d862ab4ca91fa24f775f6dfcfe5f4198c0807f0f4909bf047ffb1350b9/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/39738d8aa159ba108f0d78e1529bb35664af715ac389a39fa6d060bcf363cfa4/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b8b9dc7ee394ce0927571fb358b669ca734319cbfaf2342876758534c9e46e2d/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3c1452187a1df91e01df6afd698ac4a4f426c127b144f2bce196417fec79813a/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c0fbdfa79403832c61b1e9461210414679dd82f9527d126e63606fdb9abe4097/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/7107c4da4556b4a7b2a6e9113780c8e9f29261d548251acae86d609e859597e3/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/12e8d5140c0933c0da3677f0f5235e29dcbea42d6a1e700d2d28d98996739286/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/8f45f93f747246be0f168199353f8c0eb210046bac0c70867ae9f250862cd38b/merged 2024-04-25T16:30:21.151 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-25T16:30:21.152 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T16:30:21.152 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T16:30:21.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:21.152 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-25T16:30:21.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:2024-04-25T16:30:21.177+0000 7f62c7fb8480 -1 init, newargv = 0x7f62ac054370 newargc=15 2024-04-25T16:30:21.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[72879]: starting ceph client 2024-04-25T16:30:21.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[72879]: starting fuse 2024-04-25T16:30:21.191 INFO:teuthology.orchestra.run.smithi098.stdout:115 2024-04-25T16:30:21.191 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-25T16:30:21.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-25T16:30:21.191 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-25T16:30:21.191 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-25T16:30:21.191 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-25T16:30:21.191 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-25T16:30:21.191 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T16:30:21.192 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T16:30:21.193 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-25T16:30:21.193 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-25T16:30:21.193 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-25T16:30:21.193 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T16:30:21.193 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-25T16:30:21.193 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.admin")) 2024-04-25T16:30:21.193 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-25T16:30:21.308 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.admin.72879.asok 2024-04-25T16:30:21.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:21.319 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72879.asok 2024-04-25T16:30:21.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:21.321 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72879.asok status 2024-04-25T16:30:21.461 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:30:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-25T16:30:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T16:30:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T16:30:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "admin", 2024-04-25T16:30:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-25T16:30:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T16:30:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "72879", 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/" 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 14784, 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 14784 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-25T16:30:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2858438889 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2858438889 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.14784 192.168.0.1:0/2858438889", 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/2858438889", 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 1, 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 17, 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 81, 2024-04-25T16:30:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-25T16:30:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-25T16:30:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-25T16:30:21.464 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:30:21.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:21.470 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-25T16:30:21.470 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T16:30:21.485 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-25T16:30:21.485 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T16:30:21.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:21.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T16:30:21.550 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T16:30:21.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T16:30:21.664 INFO:teuthology.orchestra.run.smithi098.stdout:# file: . 2024-04-25T16:30:21.664 INFO:teuthology.orchestra.run.smithi098.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T16:30:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:21.664 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:30:21.670 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:30:21.670 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:30:21.686 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:21.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 -- 172.21.15.98:0/3473053201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404109a40 msgr2=0x7f640410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:21.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 --2- 172.21.15.98:0/3473053201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404109a40 0x7f640410be60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009ca0 tx=0x7f63fc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:21.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 -- 172.21.15.98:0/3473053201 shutdown_connections 2024-04-25T16:30:21.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 --2- 172.21.15.98:0/3473053201 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404109a40 0x7f640410be60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:21.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 --2- 172.21.15.98:0/3473053201 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6404104d40 0x7f6404105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:21.834 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 --2- 172.21.15.98:0/3473053201 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6404103b00 0x7f6404104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:21.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 -- 172.21.15.98:0/3473053201 >> 172.21.15.98:0/3473053201 conn(0x7f64040fed10 msgr2=0x7f64041011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:21.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 -- 172.21.15.98:0/3473053201 shutdown_connections 2024-04-25T16:30:21.835 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.834+0000 7f640ad8f640 1 -- 172.21.15.98:0/3473053201 wait complete. 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f640ad8f640 1 Processor -- start 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f640ad8f640 1 -- start start 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f640ad8f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6404103b00 0x7f640419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f640ad8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404104d40 0x7f640419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f640ad8f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6404109a40 0x7f64041a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f640ad8f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f640410f510 con 0x7f6404104d40 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f640ad8f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f640410f390 con 0x7f6404103b00 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f640ad8f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f640410f690 con 0x7f6404109a40 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6403fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404104d40 0x7f640419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:21.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6403fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404104d40 0x7f640419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48512/0 (socket says 172.21.15.98:48512) 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6403fff640 1 -- 172.21.15.98:0/1830539425 learned_addr learned my addr 172.21.15.98:0/1830539425 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6408b2b640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6404103b00 0x7f640419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6403fff640 1 -- 172.21.15.98:0/1830539425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6404109a40 msgr2=0x7f64041a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6403fff640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6404109a40 0x7f64041a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6403fff640 1 -- 172.21.15.98:0/1830539425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6404103b00 msgr2=0x7f640419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6403fff640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6404103b00 0x7f640419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.835+0000 7f6403fff640 1 -- 172.21.15.98:0/1830539425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64041a1b70 con 0x7f6404104d40 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.836+0000 7f6403fff640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404104d40 0x7f640419a620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f63f800c920 tx=0x7f63f800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.836+0000 7f6401ffb640 1 -- 172.21.15.98:0/1830539425 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8007c70 con 0x7f6404104d40 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.836+0000 7f6401ffb640 1 -- 172.21.15.98:0/1830539425 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63f8007e10 con 0x7f6404104d40 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.836+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64041a1e60 con 0x7f6404104d40 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.836+0000 7f6401ffb640 1 -- 172.21.15.98:0/1830539425 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f8018710 con 0x7f6404104d40 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.836+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64041a22d0 con 0x7f6404104d40 2024-04-25T16:30:21.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.837+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6404108850 con 0x7f6404104d40 2024-04-25T16:30:21.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.838+0000 7f6401ffb640 1 -- 172.21.15.98:0/1830539425 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f63f800f450 con 0x7f6404104d40 2024-04-25T16:30:21.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.839+0000 7f6401ffb640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63ec0afaf0 0x7f63ec0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:21.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.839+0000 7f6408b2b640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63ec0afaf0 0x7f63ec0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:21.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.839+0000 7f6401ffb640 1 -- 172.21.15.98:0/1830539425 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f63f80df040 con 0x7f6404104d40 2024-04-25T16:30:21.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.839+0000 7f6408b2b640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63ec0afaf0 0x7f63ec0b2000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63f4005fd0 tx=0x7f63f4005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:21.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:21.844+0000 7f6401ffb640 1 -- 172.21.15.98:0/1830539425 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63f8014030 con 0x7f6404104d40 2024-04-25T16:30:22.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.028+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6404063840 con 0x7f6404104d40 2024-04-25T16:30:22.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.030+0000 7f6401ffb640 1 -- 172.21.15.98:0/1830539425 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7f63f80a79e0 con 0x7f6404104d40 2024-04-25T16:30:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:22.030 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63ec0afaf0 msgr2=0x7f63ec0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63ec0afaf0 0x7f63ec0b2000 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63f4005fd0 tx=0x7f63f4005c70 comp rx=0 tx=0).stop 2024-04-25T16:30:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404104d40 msgr2=0x7f640419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404104d40 0x7f640419a620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f63f800c920 tx=0x7f63f800cde0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 shutdown_connections 2024-04-25T16:30:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63ec0afaf0 0x7f63ec0b2000 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6404109a40 0x7f64041a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6404104d40 0x7f640419a620 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 --2- 172.21.15.98:0/1830539425 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6404103b00 0x7f640419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 >> 172.21.15.98:0/1830539425 conn(0x7f64040fed10 msgr2=0x7f6404105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.032+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 shutdown_connections 2024-04-25T16:30:22.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.033+0000 7f640ad8f640 1 -- 172.21.15.98:0/1830539425 wait complete. 2024-04-25T16:30:22.044 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:30:22.044 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:30:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 -- 172.21.15.98:0/2578234313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778108920 msgr2=0x7f877810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 --2- 172.21.15.98:0/2578234313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778108920 0x7f877810ad40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8770009ca0 tx=0x7f877002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 -- 172.21.15.98:0/2578234313 shutdown_connections 2024-04-25T16:30:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 --2- 172.21.15.98:0/2578234313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778108920 0x7f877810ad40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 --2- 172.21.15.98:0/2578234313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8778103c20 0x7f8778104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 --2- 172.21.15.98:0/2578234313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f87781029e0 0x7f87781036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 -- 172.21.15.98:0/2578234313 >> 172.21.15.98:0/2578234313 conn(0x7f87780fdbd0 msgr2=0x7f8778100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 -- 172.21.15.98:0/2578234313 shutdown_connections 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 -- 172.21.15.98:0/2578234313 wait complete. 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.155+0000 7f877d2d8640 1 Processor -- start 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f877d2d8640 1 -- start start 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f877d2d8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87781029e0 0x7f877819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f877d2d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778103c20 0x7f877819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f877d2d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8778108920 0x7f87781a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f877d2d8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f877810e410 con 0x7f8778103c20 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f877d2d8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f877810e290 con 0x7f87781029e0 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f877d2d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f877810e590 con 0x7f8778108920 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778103c20 0x7f877819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778103c20 0x7f877819a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48528/0 (socket says 172.21.15.98:48528) 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 -- 172.21.15.98:0/3080727029 learned_addr learned my addr 172.21.15.98:0/3080727029 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f8776ffd640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87781029e0 0x7f877819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 -- 172.21.15.98:0/3080727029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8778108920 msgr2=0x7f87781a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8778108920 0x7f87781a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 -- 172.21.15.98:0/3080727029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87781029e0 msgr2=0x7f877819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87781029e0 0x7f877819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 -- 172.21.15.98:0/3080727029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87781a1f90 con 0x7f8778103c20 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.156+0000 7f87767fc640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778103c20 0x7f877819a990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f876800d950 tx=0x7f876800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.157+0000 7f875ffff640 1 -- 172.21.15.98:0/3080727029 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8768017070 con 0x7f8778103c20 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.157+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87781a2280 con 0x7f8778103c20 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.157+0000 7f875ffff640 1 -- 172.21.15.98:0/3080727029 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8768009c30 con 0x7f8778103c20 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.157+0000 7f875ffff640 1 -- 172.21.15.98:0/3080727029 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f876800f620 con 0x7f8778103c20 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.157+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87781a2820 con 0x7f8778103c20 2024-04-25T16:30:22.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.158+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877806c2a0 con 0x7f8778103c20 2024-04-25T16:30:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.159+0000 7f875ffff640 1 -- 172.21.15.98:0/3080727029 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8768010460 con 0x7f8778103c20 2024-04-25T16:30:22.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.159+0000 7f875ffff640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f87600afaf0 0x7f87600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.160+0000 7f8776ffd640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f87600afaf0 0x7f87600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.160+0000 7f875ffff640 1 -- 172.21.15.98:0/3080727029 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f87680dea70 con 0x7f8778103c20 2024-04-25T16:30:22.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.160+0000 7f8776ffd640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f87600afaf0 0x7f87600b2000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f876c0058c0 tx=0x7f876c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:22.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.163+0000 7f875ffff640 1 -- 172.21.15.98:0/3080727029 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f87680a7410 con 0x7f8778103c20 2024-04-25T16:30:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.339+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f87781a2a80 con 0x7f8778103c20 2024-04-25T16:30:22.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.340+0000 7f875ffff640 1 -- 172.21.15.98:0/3080727029 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f8768009de0 con 0x7f8778103c20 2024-04-25T16:30:22.340 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f87600afaf0 msgr2=0x7f87600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f87600afaf0 0x7f87600b2000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f876c0058c0 tx=0x7f876c00a2d0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778103c20 msgr2=0x7f877819a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778103c20 0x7f877819a990 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f876800d950 tx=0x7f876800de10 comp rx=0 tx=0).stop 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 shutdown_connections 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f87600afaf0 0x7f87600b2000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8778108920 0x7f87781a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8778103c20 0x7f877819a990 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 --2- 172.21.15.98:0/3080727029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87781029e0 0x7f877819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 >> 172.21.15.98:0/3080727029 conn(0x7f87780fdbd0 msgr2=0x7f8778069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:22.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 shutdown_connections 2024-04-25T16:30:22.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.342+0000 7f877d2d8640 1 -- 172.21.15.98:0/3080727029 wait complete. 2024-04-25T16:30:22.353 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:30:22.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 -- 172.21.15.98:0/2243018308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 msgr2=0x7f1d08104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 --2- 172.21.15.98:0/2243018308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 0x7f1d08104030 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009900 tx=0x7f1cfc02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:22.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 -- 172.21.15.98:0/2243018308 shutdown_connections 2024-04-25T16:30:22.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 --2- 172.21.15.98:0/2243018308 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d08108920 0x7f1d0810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 --2- 172.21.15.98:0/2243018308 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 0x7f1d08104030 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 --2- 172.21.15.98:0/2243018308 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d081029e0 0x7f1d081036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 -- 172.21.15.98:0/2243018308 >> 172.21.15.98:0/2243018308 conn(0x7f1d080fdbd0 msgr2=0x7f1d08100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 -- 172.21.15.98:0/2243018308 shutdown_connections 2024-04-25T16:30:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 -- 172.21.15.98:0/2243018308 wait complete. 2024-04-25T16:30:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 Processor -- start 2024-04-25T16:30:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.464+0000 7f1d0e442640 1 -- start start 2024-04-25T16:30:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d0e442640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d081029e0 0x7f1d08075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d0e442640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 0x7f1d0819ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d0e442640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d08108920 0x7f1d081a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d0e442640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0810e3b0 con 0x7f1d08103c20 2024-04-25T16:30:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d0e442640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0810e230 con 0x7f1d081029e0 2024-04-25T16:30:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d0e442640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0810e530 con 0x7f1d08108920 2024-04-25T16:30:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 0x7f1d0819ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 0x7f1d0819ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48550/0 (socket says 172.21.15.98:48550) 2024-04-25T16:30:22.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 -- 172.21.15.98:0/1760456408 learned_addr learned my addr 172.21.15.98:0/1760456408 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d07fff640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d081029e0 0x7f1d08075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d0c9df640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d08108920 0x7f1d081a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 -- 172.21.15.98:0/1760456408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d08108920 msgr2=0x7f1d081a14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d08108920 0x7f1d081a14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 -- 172.21.15.98:0/1760456408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d081029e0 msgr2=0x7f1d08075860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d081029e0 0x7f1d08075860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 -- 172.21.15.98:0/1760456408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d081a1cd0 con 0x7f1d08103c20 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.465+0000 7f1d077fe640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 0x7f1d0819ef40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009a20 tx=0x7f1cfc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.466+0000 7f1d057fa640 1 -- 172.21.15.98:0/1760456408 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc03f030 con 0x7f1d08103c20 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.466+0000 7f1d057fa640 1 -- 172.21.15.98:0/1760456408 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1cfc003a80 con 0x7f1d08103c20 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.466+0000 7f1d057fa640 1 -- 172.21.15.98:0/1760456408 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc035070 con 0x7f1d08103c20 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.466+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d081a1f60 con 0x7f1d08103c20 2024-04-25T16:30:22.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.466+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d081a23c0 con 0x7f1d08103c20 2024-04-25T16:30:22.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.467+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d08107730 con 0x7f1d08103c20 2024-04-25T16:30:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.468+0000 7f1d057fa640 1 -- 172.21.15.98:0/1760456408 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1cfc042030 con 0x7f1d08103c20 2024-04-25T16:30:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.469+0000 7f1d057fa640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1cf00afaf0 0x7f1cf00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.469+0000 7f1d07fff640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1cf00afaf0 0x7f1cf00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.469+0000 7f1d057fa640 1 -- 172.21.15.98:0/1760456408 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f1cfc0ff840 con 0x7f1d08103c20 2024-04-25T16:30:22.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.470+0000 7f1d07fff640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1cf00afaf0 0x7f1cf00b2000 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d081035e0 tx=0x7f1cf8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:22.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.473+0000 7f1d057fa640 1 -- 172.21.15.98:0/1760456408 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cfc0c81e0 con 0x7f1d08103c20 2024-04-25T16:30:22.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.650+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1d080637d0 con 0x7f1d08103c20 2024-04-25T16:30:22.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.650+0000 7f1d057fa640 1 -- 172.21.15.98:0/1760456408 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) ==== 104+0+11 (secure 0 0 0) 0x7f1cfc0d0260 con 0x7f1d08103c20 2024-04-25T16:30:22.650 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:30:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1cf00afaf0 msgr2=0x7f1cf00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1cf00afaf0 0x7f1cf00b2000 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1d081035e0 tx=0x7f1cf8005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 msgr2=0x7f1d0819ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 0x7f1d0819ef40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009a20 tx=0x7f1cfc0040c0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 shutdown_connections 2024-04-25T16:30:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1cf00afaf0 0x7f1cf00b2000 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1d08108920 0x7f1d081a14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1d08103c20 0x7f1d0819ef40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 --2- 172.21.15.98:0/1760456408 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1d081029e0 0x7f1d08075860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 >> 172.21.15.98:0/1760456408 conn(0x7f1d080fdbd0 msgr2=0x7f1d08104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 shutdown_connections 2024-04-25T16:30:22.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.652+0000 7f1d0e442640 1 -- 172.21.15.98:0/1760456408 wait complete. 2024-04-25T16:30:22.663 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:30:22.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 -- 172.21.15.98:0/294196416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc101740 msgr2=0x7f45cc101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 --2- 172.21.15.98:0/294196416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc101740 0x7f45cc101bb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009900 tx=0x7f45b802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:22.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 -- 172.21.15.98:0/294196416 shutdown_connections 2024-04-25T16:30:22.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 --2- 172.21.15.98:0/294196416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45cc102240 0x7f45cc10bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 --2- 172.21.15.98:0/294196416 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc101740 0x7f45cc101bb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 --2- 172.21.15.98:0/294196416 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45cc105090 0x7f45cc1054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 -- 172.21.15.98:0/294196416 >> 172.21.15.98:0/294196416 conn(0x7f45cc0fd380 msgr2=0x7f45cc0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 -- 172.21.15.98:0/294196416 shutdown_connections 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 -- 172.21.15.98:0/294196416 wait complete. 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 Processor -- start 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.773+0000 7f45d2e30640 1 -- start start 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d2e30640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45cc101740 0x7f45cc19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d2e30640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc102240 0x7f45cc19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d2e30640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45cc105090 0x7f45cc1a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d2e30640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc10f4d0 con 0x7f45cc102240 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d2e30640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc10f350 con 0x7f45cc105090 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d2e30640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45cc10f650 con 0x7f45cc101740 2024-04-25T16:30:22.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc102240 0x7f45cc19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc102240 0x7f45cc19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48568/0 (socket says 172.21.15.98:48568) 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 -- 172.21.15.98:0/2553558600 learned_addr learned my addr 172.21.15.98:0/2553558600 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d13cd640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45cc105090 0x7f45cc1a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d0bcc640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45cc101740 0x7f45cc19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 -- 172.21.15.98:0/2553558600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45cc101740 msgr2=0x7f45cc19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45cc101740 0x7f45cc19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 -- 172.21.15.98:0/2553558600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45cc105090 msgr2=0x7f45cc1a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45cc105090 0x7f45cc1a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 -- 172.21.15.98:0/2553558600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45cc1a1cd0 con 0x7f45cc102240 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45cbfff640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc102240 0x7f45cc19a6d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009230 tx=0x7f45b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45c9ffb640 1 -- 172.21.15.98:0/2553558600 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b803f030 con 0x7f45cc102240 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45c9ffb640 1 -- 172.21.15.98:0/2553558600 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45b8003c90 con 0x7f45cc102240 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45cc1a1f60 con 0x7f45cc102240 2024-04-25T16:30:22.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.774+0000 7f45c9ffb640 1 -- 172.21.15.98:0/2553558600 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8035070 con 0x7f45cc102240 2024-04-25T16:30:22.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.775+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45cc1a2370 con 0x7f45cc102240 2024-04-25T16:30:22.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.775+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45cc108a20 con 0x7f45cc102240 2024-04-25T16:30:22.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.777+0000 7f45c9ffb640 1 -- 172.21.15.98:0/2553558600 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f45b8042030 con 0x7f45cc102240 2024-04-25T16:30:22.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.778+0000 7f45c9ffb640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45bc0afaf0 0x7f45bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:22.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.778+0000 7f45d0bcc640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45bc0afaf0 0x7f45bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:22.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.778+0000 7f45c9ffb640 1 -- 172.21.15.98:0/2553558600 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f45b80ffac0 con 0x7f45cc102240 2024-04-25T16:30:22.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.778+0000 7f45d0bcc640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45bc0afaf0 0x7f45bc0b2000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f45c00058c0 tx=0x7f45c000a4c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:22.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.781+0000 7f45c9ffb640 1 -- 172.21.15.98:0/2553558600 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45b80c8460 con 0x7f45cc102240 2024-04-25T16:30:22.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.962+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f45cc0637a0 con 0x7f45cc102240 2024-04-25T16:30:22.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.963+0000 7f45c9ffb640 1 -- 172.21.15.98:0/2553558600 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) ==== 100+0+11 (secure 0 0 0) 0x7f45b80c7d60 con 0x7f45cc102240 2024-04-25T16:30:22.963 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:30:22.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45bc0afaf0 msgr2=0x7f45bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45bc0afaf0 0x7f45bc0b2000 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f45c00058c0 tx=0x7f45c000a4c0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc102240 msgr2=0x7f45cc19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:22.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc102240 0x7f45cc19a6d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009230 tx=0x7f45b80042f0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 shutdown_connections 2024-04-25T16:30:22.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45bc0afaf0 0x7f45bc0b2000 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45cc105090 0x7f45cc1a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45cc102240 0x7f45cc19a6d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 --2- 172.21.15.98:0/2553558600 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45cc101740 0x7f45cc19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:22.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 >> 172.21.15.98:0/2553558600 conn(0x7f45cc0fd380 msgr2=0x7f45cc105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:22.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 shutdown_connections 2024-04-25T16:30:22.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:22.965+0000 7f45d2e30640 1 -- 172.21.15.98:0/2553558600 wait complete. 2024-04-25T16:30:22.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:30:23.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 -- 172.21.15.98:0/103248341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870103af0 msgr2=0x7fc8701047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 --2- 172.21.15.98:0/103248341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870103af0 0x7fc8701047f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009900 tx=0x7fc86c02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 -- 172.21.15.98:0/103248341 shutdown_connections 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 --2- 172.21.15.98:0/103248341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc870109a30 0x7fc87010be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 --2- 172.21.15.98:0/103248341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc870104d30 0x7fc870105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 --2- 172.21.15.98:0/103248341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870103af0 0x7fc8701047f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 -- 172.21.15.98:0/103248341 >> 172.21.15.98:0/103248341 conn(0x7fc8700fece0 msgr2=0x7fc870101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 -- 172.21.15.98:0/103248341 shutdown_connections 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.086+0000 7fc87708c640 1 -- 172.21.15.98:0/103248341 wait complete. 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc87708c640 1 Processor -- start 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc87708c640 1 -- start start 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc87708c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc870103af0 0x7fc870077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc87708c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870104d30 0x7fc870076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc87708c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc870109a30 0x7fc870076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc87708c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc87010f4a0 con 0x7fc870104d30 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc87708c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc87010f320 con 0x7fc870103af0 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc87708c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc87010f620 con 0x7fc870109a30 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc867fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870104d30 0x7fc870076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc867fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870104d30 0x7fc870076000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48594/0 (socket says 172.21.15.98:48594) 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc867fff640 1 -- 172.21.15.98:0/3451123198 learned_addr learned my addr 172.21.15.98:0/3451123198 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc875629640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc870109a30 0x7fc870076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.087+0000 7fc874e28640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc870103af0 0x7fc870077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc867fff640 1 -- 172.21.15.98:0/3451123198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc870109a30 msgr2=0x7fc870076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc867fff640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc870109a30 0x7fc870076540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc867fff640 1 -- 172.21.15.98:0/3451123198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc870103af0 msgr2=0x7fc870077960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc867fff640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc870103af0 0x7fc870077960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc867fff640 1 -- 172.21.15.98:0/3451123198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc870076c70 con 0x7fc870104d30 2024-04-25T16:30:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc867fff640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870104d30 0x7fc870076000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc86000e9f0 tx=0x7fc86000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc865ffb640 1 -- 172.21.15.98:0/3451123198 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86000cd80 con 0x7fc870104d30 2024-04-25T16:30:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc865ffb640 1 -- 172.21.15.98:0/3451123198 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8600054b0 con 0x7fc870104d30 2024-04-25T16:30:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8701a4220 con 0x7fc870104d30 2024-04-25T16:30:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc865ffb640 1 -- 172.21.15.98:0/3451123198 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc860010430 con 0x7fc870104d30 2024-04-25T16:30:23.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.088+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8701a4690 con 0x7fc870104d30 2024-04-25T16:30:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.089+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc870108840 con 0x7fc870104d30 2024-04-25T16:30:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.090+0000 7fc865ffb640 1 -- 172.21.15.98:0/3451123198 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc8600046c0 con 0x7fc870104d30 2024-04-25T16:30:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.091+0000 7fc865ffb640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8580afaf0 0x7fc8580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.091+0000 7fc874e28640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8580afaf0 0x7fc8580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.092+0000 7fc865ffb640 1 -- 172.21.15.98:0/3451123198 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fc8600df1d0 con 0x7fc870104d30 2024-04-25T16:30:23.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.092+0000 7fc874e28640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8580afaf0 0x7fc8580b2000 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc86c005760 tx=0x7fc86c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:23.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.096+0000 7fc865ffb640 1 -- 172.21.15.98:0/3451123198 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8600a7b70 con 0x7fc870104d30 2024-04-25T16:30:23.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:22 smithi098 ceph-mon[29162]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-04-25T16:30:23.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:22 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1830539425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:23.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:22 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3080727029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:22 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1760456408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:22 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2553558600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:22 smithi173 ceph-mon[32556]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-04-25T16:30:23.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:22 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1830539425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:23.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:22 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3080727029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:22 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1760456408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:22 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2553558600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.275+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc8700637a0 con 0x7fc870104d30 2024-04-25T16:30:23.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.275+0000 7fc865ffb640 1 -- 172.21.15.98:0/3451123198 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v81) ==== 103+0+11 (secure 0 0 0) 0x7fc8600a7470 con 0x7fc870104d30 2024-04-25T16:30:23.275 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:30:23.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8580afaf0 msgr2=0x7fc8580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8580afaf0 0x7fc8580b2000 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc86c005760 tx=0x7fc86c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870104d30 msgr2=0x7fc870076000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870104d30 0x7fc870076000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc86000e9f0 tx=0x7fc86000edf0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 shutdown_connections 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8580afaf0 0x7fc8580b2000 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc870109a30 0x7fc870076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc870104d30 0x7fc870076000 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 --2- 172.21.15.98:0/3451123198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc870103af0 0x7fc870077960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 >> 172.21.15.98:0/3451123198 conn(0x7fc8700fece0 msgr2=0x7fc870105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 shutdown_connections 2024-04-25T16:30:23.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.277+0000 7fc87708c640 1 -- 172.21.15.98:0/3451123198 wait complete. 2024-04-25T16:30:23.288 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T16:30:23.391 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:22 smithi203 ceph-mon[33115]: pgmap v288: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s wr, 0 op/s 2024-04-25T16:30:23.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:22 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1830539425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:23.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:22 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3080727029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:22 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1760456408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.392 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:22 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2553558600' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- 172.21.15.98:0/2899117108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90103c20 msgr2=0x7f5c90104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/2899117108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90103c20 0x7f5c90104030 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009900 tx=0x7f5c8402a410 comp rx=0 tx=0).stop 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- 172.21.15.98:0/2899117108 shutdown_connections 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/2899117108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c90108920 0x7f5c9010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/2899117108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90103c20 0x7f5c90104030 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/2899117108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c901029e0 0x7f5c901036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- 172.21.15.98:0/2899117108 >> 172.21.15.98:0/2899117108 conn(0x7f5c900fdbd0 msgr2=0x7f5c90100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- 172.21.15.98:0/2899117108 shutdown_connections 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- 172.21.15.98:0/2899117108 wait complete. 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 Processor -- start 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- start start 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c901029e0 0x7f5c9019a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c90103c20 0x7f5c9019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90108920 0x7f5c901a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9010e3c0 con 0x7f5c90108920 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9010e240 con 0x7f5c90103c20 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.397+0000 7f5c96ba8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9010e540 con 0x7f5c901029e0 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90108920 0x7f5c901a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90108920 0x7f5c901a1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48616/0 (socket says 172.21.15.98:48616) 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 -- 172.21.15.98:0/1965848889 learned_addr learned my addr 172.21.15.98:0/1965848889 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c8ffff640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c90103c20 0x7f5c9019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c94944640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c901029e0 0x7f5c9019a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 -- 172.21.15.98:0/1965848889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c901029e0 msgr2=0x7f5c9019a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c901029e0 0x7f5c9019a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 -- 172.21.15.98:0/1965848889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c90103c20 msgr2=0x7f5c9019a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c90103c20 0x7f5c9019a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 -- 172.21.15.98:0/1965848889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c901a1fb0 con 0x7f5c90108920 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c95145640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90108920 0x7f5c901a1940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5c8800eb10 tx=0x7f5c8800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:23.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c8dffb640 1 -- 172.21.15.98:0/1965848889 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88004b20 con 0x7f5c90108920 2024-04-25T16:30:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c901a22a0 con 0x7f5c90108920 2024-04-25T16:30:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c8dffb640 1 -- 172.21.15.98:0/1965848889 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5c88004cc0 con 0x7f5c90108920 2024-04-25T16:30:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c8dffb640 1 -- 172.21.15.98:0/1965848889 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88010690 con 0x7f5c90108920 2024-04-25T16:30:23.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.398+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c901a27f0 con 0x7f5c90108920 2024-04-25T16:30:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.399+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c9006c2a0 con 0x7f5c90108920 2024-04-25T16:30:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.400+0000 7f5c8dffb640 1 -- 172.21.15.98:0/1965848889 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5c88005020 con 0x7f5c90108920 2024-04-25T16:30:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.401+0000 7f5c8dffb640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5c780afaf0 0x7f5c780b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.401+0000 7f5c94944640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5c780afaf0 0x7f5c780b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.401+0000 7f5c8dffb640 1 -- 172.21.15.98:0/1965848889 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f5c880df470 con 0x7f5c90108920 2024-04-25T16:30:23.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.402+0000 7f5c94944640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5c780afaf0 0x7f5c780b2000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5c901035e0 tx=0x7f5c80005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:23.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.406+0000 7f5c8dffb640 1 -- 172.21.15.98:0/1965848889 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c880a7e10 con 0x7f5c90108920 2024-04-25T16:30:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.594+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5c901a2ab0 con 0x7f5c90108920 2024-04-25T16:30:23.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.595+0000 7f5c8dffb640 1 -- 172.21.15.98:0/1965848889 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) ==== 81+0+589 (secure 0 0 0) 0x7f5c880a7710 con 0x7f5c90108920 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1883089564 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/119631250 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/217035597 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2573630616 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/4161723298 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3681298222 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/4161723298 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1429360204 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3090764993 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:23.595 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/217035597 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:23.596 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5c780afaf0 msgr2=0x7f5c780b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5c780afaf0 0x7f5c780b2000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5c901035e0 tx=0x7f5c80005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90108920 msgr2=0x7f5c901a1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90108920 0x7f5c901a1940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5c8800eb10 tx=0x7f5c8800c2d0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 shutdown_connections 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5c780afaf0 0x7f5c780b2000 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5c90108920 0x7f5c901a1940 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c90103c20 0x7f5c9019a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 --2- 172.21.15.98:0/1965848889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5c901029e0 0x7f5c9019a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 >> 172.21.15.98:0/1965848889 conn(0x7f5c900fdbd0 msgr2=0x7f5c90069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 shutdown_connections 2024-04-25T16:30:23.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.597+0000 7f5c96ba8640 1 -- 172.21.15.98:0/1965848889 wait complete. 2024-04-25T16:30:23.608 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:30:23.608 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:30:23.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 -- 172.21.15.98:0/3445966364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 msgr2=0x7fb54810bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 --2- 172.21.15.98:0/3445966364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 0x7fb54810bcc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb540009ca0 tx=0x7fb54002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 -- 172.21.15.98:0/3445966364 shutdown_connections 2024-04-25T16:30:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 --2- 172.21.15.98:0/3445966364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 0x7fb54810bcc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 --2- 172.21.15.98:0/3445966364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb548101760 0x7fb548101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 --2- 172.21.15.98:0/3445966364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5481050b0 0x7fb5481054c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 -- 172.21.15.98:0/3445966364 >> 172.21.15.98:0/3445966364 conn(0x7fb5480fd360 msgr2=0x7fb5480ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:23.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 -- 172.21.15.98:0/3445966364 shutdown_connections 2024-04-25T16:30:23.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.771+0000 7fb54d19e640 1 -- 172.21.15.98:0/3445966364 wait complete. 2024-04-25T16:30:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54d19e640 1 Processor -- start 2024-04-25T16:30:23.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54d19e640 1 -- start start 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54d19e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb548101760 0x7fb54819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54d19e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 0x7fb54819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54d19e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5481050b0 0x7fb5481a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54d19e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb54810f540 con 0x7fb548102260 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54d19e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb54810f3c0 con 0x7fb548101760 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54d19e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb54810f6c0 con 0x7fb5481050b0 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 0x7fb54819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 0x7fb54819a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48626/0 (socket says 172.21.15.98:48626) 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54659c640 1 -- 172.21.15.98:0/1023600621 learned_addr learned my addr 172.21.15.98:0/1023600621 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb546d9d640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb548101760 0x7fb54819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.772+0000 7fb54759e640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5481050b0 0x7fb5481a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb54659c640 1 -- 172.21.15.98:0/1023600621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5481050b0 msgr2=0x7fb5481a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb54659c640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5481050b0 0x7fb5481a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb54659c640 1 -- 172.21.15.98:0/1023600621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb548101760 msgr2=0x7fb54819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb54659c640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb548101760 0x7fb54819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb54659c640 1 -- 172.21.15.98:0/1023600621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5481a1b10 con 0x7fb548102260 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb54659c640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 0x7fb54819a5c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c920 tx=0x7fb53c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb52ffff640 1 -- 172.21.15.98:0/1023600621 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c007c70 con 0x7fb548102260 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb52ffff640 1 -- 172.21.15.98:0/1023600621 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb53c007e10 con 0x7fb548102260 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5481a1e00 con 0x7fb548102260 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb52ffff640 1 -- 172.21.15.98:0/1023600621 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb53c018710 con 0x7fb548102260 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.773+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5481a2270 con 0x7fb548102260 2024-04-25T16:30:23.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.774+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54810aa50 con 0x7fb548102260 2024-04-25T16:30:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.775+0000 7fb52ffff640 1 -- 172.21.15.98:0/1023600621 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fb53c00f450 con 0x7fb548102260 2024-04-25T16:30:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.776+0000 7fb52ffff640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5300afaf0 0x7fb5300b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.776+0000 7fb546d9d640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5300afaf0 0x7fb5300b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:23.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.776+0000 7fb52ffff640 1 -- 172.21.15.98:0/1023600621 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fb53c0df1f0 con 0x7fb548102260 2024-04-25T16:30:23.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.776+0000 7fb546d9d640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5300afaf0 0x7fb5300b2000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb538005fd0 tx=0x7fb53800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:23.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.780+0000 7fb52ffff640 1 -- 172.21.15.98:0/1023600621 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb53c014030 con 0x7fb548102260 2024-04-25T16:30:23.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.958+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb54802dd20 con 0x7fb548102260 2024-04-25T16:30:23.959 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.960+0000 7fb52ffff640 1 -- 172.21.15.98:0/1023600621 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7fb53c0a7c20 con 0x7fb548102260 2024-04-25T16:30:23.960 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:23.960 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5300afaf0 msgr2=0x7fb5300b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5300afaf0 0x7fb5300b2000 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb538005fd0 tx=0x7fb53800b390 comp rx=0 tx=0).stop 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 msgr2=0x7fb54819a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 0x7fb54819a5c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00c920 tx=0x7fb53c00cde0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 shutdown_connections 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5300afaf0 0x7fb5300b2000 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5481050b0 0x7fb5481a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb548102260 0x7fb54819a5c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 --2- 172.21.15.98:0/1023600621 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb548101760 0x7fb54819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 >> 172.21.15.98:0/1023600621 conn(0x7fb5480fd360 msgr2=0x7fb548107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 shutdown_connections 2024-04-25T16:30:23.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:23.962+0000 7fb54d19e640 1 -- 172.21.15.98:0/1023600621 wait complete. 2024-04-25T16:30:23.973 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:30:23.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:30:24.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.084+0000 7f4ea7c92640 1 -- 172.21.15.98:0/1794296692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 msgr2=0x7f4ea010bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.084+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/1794296692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 0x7f4ea010bcc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009ca0 tx=0x7f4e9802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.084+0000 7f4ea7c92640 1 -- 172.21.15.98:0/1794296692 shutdown_connections 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.084+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/1794296692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 0x7f4ea010bcc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.084+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/1794296692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea0101760 0x7f4ea0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.084+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/1794296692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01050b0 0x7f4ea01054c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.084+0000 7f4ea7c92640 1 -- 172.21.15.98:0/1794296692 >> 172.21.15.98:0/1794296692 conn(0x7f4ea00fd380 msgr2=0x7f4ea00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 -- 172.21.15.98:0/1794296692 shutdown_connections 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 -- 172.21.15.98:0/1794296692 wait complete. 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 Processor -- start 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 -- start start 2024-04-25T16:30:24.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea0101760 0x7f4ea019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 0x7f4ea019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01050b0 0x7f4ea01a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea010f540 con 0x7f4ea0102260 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea010f3c0 con 0x7f4ea01050b0 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea7c92640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea010f6c0 con 0x7f4ea0101760 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea522d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 0x7f4ea019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea522d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 0x7f4ea019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48632/0 (socket says 172.21.15.98:48632) 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.085+0000 7f4ea522d640 1 -- 172.21.15.98:0/3876908782 learned_addr learned my addr 172.21.15.98:0/3876908782 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea5a2e640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea0101760 0x7f4ea019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea622f640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01050b0 0x7f4ea01a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea522d640 1 -- 172.21.15.98:0/3876908782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea0101760 msgr2=0x7f4ea019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea522d640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea0101760 0x7f4ea019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea522d640 1 -- 172.21.15.98:0/3876908782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01050b0 msgr2=0x7f4ea01a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea522d640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01050b0 0x7f4ea01a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea522d640 1 -- 172.21.15.98:0/3876908782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea01a1c50 con 0x7f4ea0102260 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea522d640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 0x7f4ea019a6c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9f0 tx=0x7f4e9000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4e96ffd640 1 -- 172.21.15.98:0/3876908782 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e9000cd80 con 0x7f4ea0102260 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4e96ffd640 1 -- 172.21.15.98:0/3876908782 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e900054b0 con 0x7f4ea0102260 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea01a1f40 con 0x7f4ea0102260 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4e96ffd640 1 -- 172.21.15.98:0/3876908782 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e90010430 con 0x7f4ea0102260 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.086+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea01a23a0 con 0x7f4ea0102260 2024-04-25T16:30:24.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.087+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ea010aac0 con 0x7f4ea0102260 2024-04-25T16:30:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.088+0000 7f4e96ffd640 1 -- 172.21.15.98:0/3876908782 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4e900046c0 con 0x7f4ea0102260 2024-04-25T16:30:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.089+0000 7f4e96ffd640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e8c0afaf0 0x7f4e8c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.089+0000 7f4ea5a2e640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e8c0afaf0 0x7f4e8c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.089+0000 7f4e96ffd640 1 -- 172.21.15.98:0/3876908782 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f4e900df3d0 con 0x7f4ea0102260 2024-04-25T16:30:24.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.089+0000 7f4ea5a2e640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e8c0afaf0 0x7f4e8c0b2000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ea006b2d0 tx=0x7f4e9c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:24.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.093+0000 7f4e96ffd640 1 -- 172.21.15.98:0/3876908782 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e90053ff0 con 0x7f4ea0102260 2024-04-25T16:30:24.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:23 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3451123198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:30:24.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:23 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1965848889' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:30:24.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:23 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1023600621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:24.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:23 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3451123198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:30:24.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:23 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1965848889' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:30:24.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:23 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1023600621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.272+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4ea00637d0 con 0x7f4ea0102260 2024-04-25T16:30:24.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.273+0000 7f4e96ffd640 1 -- 172.21.15.98:0/3876908782 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f4e900a7d70 con 0x7f4ea0102260 2024-04-25T16:30:24.272 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:30:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e8c0afaf0 msgr2=0x7f4e8c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e8c0afaf0 0x7f4e8c0b2000 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ea006b2d0 tx=0x7f4e9c005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 msgr2=0x7f4ea019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 0x7f4ea019a6c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4e9000e9f0 tx=0x7f4e9000edf0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 shutdown_connections 2024-04-25T16:30:24.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e8c0afaf0 0x7f4e8c0b2000 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4ea01050b0 0x7f4ea01a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4ea0102260 0x7f4ea019a6c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 --2- 172.21.15.98:0/3876908782 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4ea0101760 0x7f4ea019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 >> 172.21.15.98:0/3876908782 conn(0x7f4ea00fd380 msgr2=0x7f4ea0107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:24.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 shutdown_connections 2024-04-25T16:30:24.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.275+0000 7f4ea7c92640 1 -- 172.21.15.98:0/3876908782 wait complete. 2024-04-25T16:30:24.286 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:30:24.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 -- 172.21.15.98:0/909952784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc109a30 msgr2=0x7f14cc10be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 --2- 172.21.15.98:0/909952784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc109a30 0x7f14cc10be50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009ca0 tx=0x7f14c402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 -- 172.21.15.98:0/909952784 shutdown_connections 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 --2- 172.21.15.98:0/909952784 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc109a30 0x7f14cc10be50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 --2- 172.21.15.98:0/909952784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14cc104d30 0x7f14cc105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 --2- 172.21.15.98:0/909952784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc103af0 0x7f14cc1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 -- 172.21.15.98:0/909952784 >> 172.21.15.98:0/909952784 conn(0x7f14cc0fece0 msgr2=0x7f14cc101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 -- 172.21.15.98:0/909952784 shutdown_connections 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.397+0000 7f14d2906640 1 -- 172.21.15.98:0/909952784 wait complete. 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.398+0000 7f14d2906640 1 Processor -- start 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.398+0000 7f14d2906640 1 -- start start 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.398+0000 7f14d2906640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc103af0 0x7f14cc19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.398+0000 7f14d2906640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc104d30 0x7f14cc19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.398+0000 7f14d2906640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14cc109a30 0x7f14cc1a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.398+0000 7f14d2906640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc10f520 con 0x7f14cc103af0 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.398+0000 7f14d2906640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc10f3a0 con 0x7f14cc109a30 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.398+0000 7f14d2906640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc10f6a0 con 0x7f14cc104d30 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc103af0 0x7f14cc19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc103af0 0x7f14cc19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48646/0 (socket says 172.21.15.98:48646) 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 -- 172.21.15.98:0/963041232 learned_addr learned my addr 172.21.15.98:0/963041232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cb7fe640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc104d30 0x7f14cc19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14d0ea3640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14cc109a30 0x7f14cc1a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 -- 172.21.15.98:0/963041232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc104d30 msgr2=0x7f14cc19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc104d30 0x7f14cc19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 -- 172.21.15.98:0/963041232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14cc109a30 msgr2=0x7f14cc1a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14cc109a30 0x7f14cc1a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 -- 172.21.15.98:0/963041232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc1a1c60 con 0x7f14cc103af0 2024-04-25T16:30:24.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14cbfff640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc103af0 0x7f14cc19a120 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f14c000e9f0 tx=0x7f14c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14c97fa640 1 -- 172.21.15.98:0/963041232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c000ccc0 con 0x7f14cc103af0 2024-04-25T16:30:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14c97fa640 1 -- 172.21.15.98:0/963041232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f14c00035f0 con 0x7f14cc103af0 2024-04-25T16:30:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc1a1f50 con 0x7f14cc103af0 2024-04-25T16:30:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14c97fa640 1 -- 172.21.15.98:0/963041232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c000ce60 con 0x7f14cc103af0 2024-04-25T16:30:24.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.399+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc1a2360 con 0x7f14cc103af0 2024-04-25T16:30:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.400+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14cc06a160 con 0x7f14cc103af0 2024-04-25T16:30:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.401+0000 7f14c97fa640 1 -- 172.21.15.98:0/963041232 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f14c0004000 con 0x7f14cc103af0 2024-04-25T16:30:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.402+0000 7f14c97fa640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f14b40afaf0 0x7f14b40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.402+0000 7f14cb7fe640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f14b40afaf0 0x7f14b40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.402+0000 7f14c97fa640 1 -- 172.21.15.98:0/963041232 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f14c00df190 con 0x7f14cc103af0 2024-04-25T16:30:24.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.403+0000 7f14cb7fe640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f14b40afaf0 0x7f14b40b2000 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f14cc19b1c0 tx=0x7f14bc005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:24.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.407+0000 7f14c97fa640 1 -- 172.21.15.98:0/963041232 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14c00a7b30 con 0x7f14cc103af0 2024-04-25T16:30:24.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:23 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3451123198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:30:24.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:23 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1965848889' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:30:24.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:23 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1023600621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:24.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.585+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f14cc02db50 con 0x7f14cc103af0 2024-04-25T16:30:24.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.586+0000 7f14c97fa640 1 -- 172.21.15.98:0/963041232 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) ==== 104+0+11 (secure 0 0 0) 0x7f14c00a7430 con 0x7f14cc103af0 2024-04-25T16:30:24.586 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f14b40afaf0 msgr2=0x7f14b40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f14b40afaf0 0x7f14b40b2000 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f14cc19b1c0 tx=0x7f14bc005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc103af0 msgr2=0x7f14cc19a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc103af0 0x7f14cc19a120 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f14c000e9f0 tx=0x7f14c000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 shutdown_connections 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f14b40afaf0 0x7f14b40b2000 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14cc109a30 0x7f14cc1a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f14cc104d30 0x7f14cc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 --2- 172.21.15.98:0/963041232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f14cc103af0 0x7f14cc19a120 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 >> 172.21.15.98:0/963041232 conn(0x7f14cc0fece0 msgr2=0x7f14cc105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:24.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 shutdown_connections 2024-04-25T16:30:24.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.588+0000 7f14d2906640 1 -- 172.21.15.98:0/963041232 wait complete. 2024-04-25T16:30:24.599 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:30:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 -- 172.21.15.98:0/692603658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c103c40 msgr2=0x7fcb3c104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 --2- 172.21.15.98:0/692603658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c103c40 0x7fcb3c104050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcb38009900 tx=0x7fcb3802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 -- 172.21.15.98:0/692603658 shutdown_connections 2024-04-25T16:30:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 --2- 172.21.15.98:0/692603658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb3c108940 0x7fcb3c10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 --2- 172.21.15.98:0/692603658 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c103c40 0x7fcb3c104050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 --2- 172.21.15.98:0/692603658 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb3c102a00 0x7fcb3c103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 -- 172.21.15.98:0/692603658 >> 172.21.15.98:0/692603658 conn(0x7fcb3c0fdbf0 msgr2=0x7fcb3c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:24.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 -- 172.21.15.98:0/692603658 shutdown_connections 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 -- 172.21.15.98:0/692603658 wait complete. 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.709+0000 7fcb4357a640 1 Processor -- start 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 -- start start 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102a00 0x7fcb3c075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb3c103c40 0x7fcb3c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb3c108940 0x7fcb3c1a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c10e430 con 0x7fcb3c102a00 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c10e2b0 con 0x7fcb3c108940 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3c10e5b0 con 0x7fcb3c103c40 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102a00 0x7fcb3c075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102a00 0x7fcb3c075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48668/0 (socket says 172.21.15.98:48668) 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 -- 172.21.15.98:0/2444320350 learned_addr learned my addr 172.21.15.98:0/2444320350 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41b17640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb3c108940 0x7fcb3c1a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb40b15640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb3c103c40 0x7fcb3c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 -- 172.21.15.98:0/2444320350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb3c103c40 msgr2=0x7fcb3c19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb3c103c40 0x7fcb3c19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 -- 172.21.15.98:0/2444320350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb3c108940 msgr2=0x7fcb3c1a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb3c108940 0x7fcb3c1a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 -- 172.21.15.98:0/2444320350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb3c1a1d10 con 0x7fcb3c102a00 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb40b15640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb3c103c40 0x7fcb3c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb41316640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102a00 0x7fcb3c075840 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00b450 tx=0x7fcb2c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb327fc640 1 -- 172.21.15.98:0/2444320350 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c015020 con 0x7fcb3c102a00 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb327fc640 1 -- 172.21.15.98:0/2444320350 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcb2c011d20 con 0x7fcb3c102a00 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb3c1a1fa0 con 0x7fcb3c102a00 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb327fc640 1 -- 172.21.15.98:0/2444320350 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c010df0 con 0x7fcb3c102a00 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.710+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb3c1a24e0 con 0x7fcb3c102a00 2024-04-25T16:30:24.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.711+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb3c107750 con 0x7fcb3c102a00 2024-04-25T16:30:24.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.713+0000 7fcb327fc640 1 -- 172.21.15.98:0/2444320350 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fcb2c018020 con 0x7fcb3c102a00 2024-04-25T16:30:24.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.714+0000 7fcb327fc640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcb280afaf0 0x7fcb280b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:24.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.714+0000 7fcb40b15640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcb280afaf0 0x7fcb280b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:24.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.714+0000 7fcb327fc640 1 -- 172.21.15.98:0/2444320350 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fcb2c0deda0 con 0x7fcb3c102a00 2024-04-25T16:30:24.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.714+0000 7fcb40b15640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcb280afaf0 0x7fcb280b2000 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcb380096c0 tx=0x7fcb38005de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:24.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.718+0000 7fcb327fc640 1 -- 172.21.15.98:0/2444320350 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb2c01f020 con 0x7fcb3c102a00 2024-04-25T16:30:24.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.896+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcb3c0637a0 con 0x7fcb3c102a00 2024-04-25T16:30:24.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.897+0000 7fcb327fc640 1 -- 172.21.15.98:0/2444320350 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) ==== 100+0+11 (secure 0 0 0) 0x7fcb2c00d040 con 0x7fcb3c102a00 2024-04-25T16:30:24.897 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcb280afaf0 msgr2=0x7fcb280b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcb280afaf0 0x7fcb280b2000 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcb380096c0 tx=0x7fcb38005de0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102a00 msgr2=0x7fcb3c075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102a00 0x7fcb3c075840 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00b450 tx=0x7fcb2c00b910 comp rx=0 tx=0).stop 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 shutdown_connections 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcb280afaf0 0x7fcb280b2000 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb3c108940 0x7fcb3c1a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb3c103c40 0x7fcb3c19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 --2- 172.21.15.98:0/2444320350 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcb3c102a00 0x7fcb3c075840 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 >> 172.21.15.98:0/2444320350 conn(0x7fcb3c0fdbf0 msgr2=0x7fcb3c104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:24.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 shutdown_connections 2024-04-25T16:30:24.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:24.899+0000 7fcb4357a640 1 -- 172.21.15.98:0/2444320350 wait complete. 2024-04-25T16:30:24.910 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 -- 172.21.15.98:0/4264493451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488109890 msgr2=0x7f348810bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 --2- 172.21.15.98:0/4264493451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488109890 0x7f348810bcb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3480009ca0 tx=0x7f348002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 -- 172.21.15.98:0/4264493451 shutdown_connections 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 --2- 172.21.15.98:0/4264493451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488109890 0x7f348810bcb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 --2- 172.21.15.98:0/4264493451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3488106f30 0x7f3488109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 --2- 172.21.15.98:0/4264493451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34881045d0 0x7f34881069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 -- 172.21.15.98:0/4264493451 >> 172.21.15.98:0/4264493451 conn(0x7f34880fdbd0 msgr2=0x7f3488100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 -- 172.21.15.98:0/4264493451 shutdown_connections 2024-04-25T16:30:25.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 -- 172.21.15.98:0/4264493451 wait complete. 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 Processor -- start 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.023+0000 7f348f5a8640 1 -- start start 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348f5a8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34881045d0 0x7f348819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348f5a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488106f30 0x7f348819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348f5a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3488109890 0x7f34881a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348f5a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f348810f510 con 0x7f3488106f30 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348f5a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f348810f390 con 0x7f3488109890 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348f5a8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f348810f690 con 0x7f34881045d0 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488106f30 0x7f348819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488106f30 0x7f348819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48692/0 (socket says 172.21.15.98:48692) 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 -- 172.21.15.98:0/3762878595 learned_addr learned my addr 172.21.15.98:0/3762878595 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348d344640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34881045d0 0x7f348819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:25.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348db45640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3488109890 0x7f34881a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 -- 172.21.15.98:0/3762878595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34881045d0 msgr2=0x7f348819a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34881045d0 0x7f348819a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 -- 172.21.15.98:0/3762878595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3488109890 msgr2=0x7f34881a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3488109890 0x7f34881a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 -- 172.21.15.98:0/3762878595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34881a1be0 con 0x7f3488106f30 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348d344640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34881045d0 0x7f348819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.024+0000 7f348cb43640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488106f30 0x7f348819a690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f347800e9f0 tx=0x7f347800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.025+0000 7f347e7fc640 1 -- 172.21.15.98:0/3762878595 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f347800cd80 con 0x7f3488106f30 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.025+0000 7f347e7fc640 1 -- 172.21.15.98:0/3762878595 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f34780054b0 con 0x7f3488106f30 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.025+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34881a1ed0 con 0x7f3488106f30 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.025+0000 7f347e7fc640 1 -- 172.21.15.98:0/3762878595 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3478010430 con 0x7f3488106f30 2024-04-25T16:30:25.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.025+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34881a2340 con 0x7f3488106f30 2024-04-25T16:30:25.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.026+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34880ff6f0 con 0x7f3488106f30 2024-04-25T16:30:25.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.027+0000 7f347e7fc640 1 -- 172.21.15.98:0/3762878595 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f34780046c0 con 0x7f3488106f30 2024-04-25T16:30:25.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.028+0000 7f347e7fc640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34740afaf0 0x7f34740b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:25.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.028+0000 7f348d344640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34740afaf0 0x7f34740b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:25.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.028+0000 7f347e7fc640 1 -- 172.21.15.98:0/3762878595 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f34780df4f0 con 0x7f3488106f30 2024-04-25T16:30:25.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.029+0000 7f348d344640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34740afaf0 0x7f34740b2000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f34880757f0 tx=0x7f348400a5c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:25.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.032+0000 7f347e7fc640 1 -- 172.21.15.98:0/3762878595 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3478053ff0 con 0x7f3488106f30 2024-04-25T16:30:25.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.208+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f348802dce0 con 0x7f3488106f30 2024-04-25T16:30:25.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.209+0000 7f347e7fc640 1 -- 172.21.15.98:0/3762878595 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v81) ==== 103+0+11 (secure 0 0 0) 0x7f34780a7e90 con 0x7f3488106f30 2024-04-25T16:30:25.209 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:30:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34740afaf0 msgr2=0x7f34740b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34740afaf0 0x7f34740b2000 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f34880757f0 tx=0x7f348400a5c0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488106f30 msgr2=0x7f348819a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488106f30 0x7f348819a690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f347800e9f0 tx=0x7f347800edf0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 shutdown_connections 2024-04-25T16:30:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34740afaf0 0x7f34740b2000 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3488109890 0x7f34881a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3488106f30 0x7f348819a690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 --2- 172.21.15.98:0/3762878595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34881045d0 0x7f348819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 >> 172.21.15.98:0/3762878595 conn(0x7f34880fdbd0 msgr2=0x7f3488105220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 shutdown_connections 2024-04-25T16:30:25.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.211+0000 7f348f5a8640 1 -- 172.21.15.98:0/3762878595 wait complete. 2024-04-25T16:30:25.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:24 smithi098 ceph-mon[29162]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-04-25T16:30:25.221 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:24 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3876908782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.221 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:24 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/963041232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.221 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:24 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2444320350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T16:30:25.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:24 smithi173 ceph-mon[32556]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-04-25T16:30:25.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:24 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3876908782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:24 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/963041232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:24 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2444320350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 -- 172.21.15.98:0/508870032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c102190 msgr2=0x7fb97c102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 --2- 172.21.15.98:0/508870032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c102190 0x7fb97c102e90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb970009900 tx=0x7fb97002a440 comp rx=0 tx=0).stop 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 -- 172.21.15.98:0/508870032 shutdown_connections 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 --2- 172.21.15.98:0/508870032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb97c1080d0 0x7fb97c10a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 --2- 172.21.15.98:0/508870032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb97c1033d0 0x7fb97c1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 --2- 172.21.15.98:0/508870032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c102190 0x7fb97c102e90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 -- 172.21.15.98:0/508870032 >> 172.21.15.98:0/508870032 conn(0x7fb97c0fd380 msgr2=0x7fb97c0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 -- 172.21.15.98:0/508870032 shutdown_connections 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.331+0000 7fb982dce640 1 -- 172.21.15.98:0/508870032 wait complete. 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb982dce640 1 Processor -- start 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb982dce640 1 -- start start 2024-04-25T16:30:25.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb982dce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb97c102190 0x7fb97c199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb982dce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c1033d0 0x7fb97c1997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb982dce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb97c1080d0 0x7fb97c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb982dce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c10dc60 con 0x7fb97c1033d0 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb982dce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c10dae0 con 0x7fb97c1080d0 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb982dce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb97c10dde0 con 0x7fb97c102190 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c1033d0 0x7fb97c1997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb97bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c1033d0 0x7fb97c1997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48714/0 (socket says 172.21.15.98:48714) 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb97bfff640 1 -- 172.21.15.98:0/4141859557 learned_addr learned my addr 172.21.15.98:0/4141859557 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb98136b640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb97c1080d0 0x7fb97c1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb980b6a640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb97c102190 0x7fb97c199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:25.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb97bfff640 1 -- 172.21.15.98:0/4141859557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb97c102190 msgr2=0x7fb97c199280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb97bfff640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb97c102190 0x7fb97c199280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb97bfff640 1 -- 172.21.15.98:0/4141859557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb97c1080d0 msgr2=0x7fb97c1a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb97bfff640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb97c1080d0 0x7fb97c1a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb97bfff640 1 -- 172.21.15.98:0/4141859557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97c1a0dc0 con 0x7fb97c1033d0 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.332+0000 7fb98136b640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb97c1080d0 0x7fb97c1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.333+0000 7fb97bfff640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c1033d0 0x7fb97c1997c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb96800b450 tx=0x7fb96800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.333+0000 7fb979ffb640 1 -- 172.21.15.98:0/4141859557 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968015020 con 0x7fb97c1033d0 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.333+0000 7fb979ffb640 1 -- 172.21.15.98:0/4141859557 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb968011d20 con 0x7fb97c1033d0 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.333+0000 7fb979ffb640 1 -- 172.21.15.98:0/4141859557 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb968010e30 con 0x7fb97c1033d0 2024-04-25T16:30:25.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.333+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97c1a10b0 con 0x7fb97c1033d0 2024-04-25T16:30:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.333+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb97c1a1630 con 0x7fb97c1033d0 2024-04-25T16:30:25.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.334+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb97c1077d0 con 0x7fb97c1033d0 2024-04-25T16:30:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.335+0000 7fb979ffb640 1 -- 172.21.15.98:0/4141859557 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fb968018020 con 0x7fb97c1033d0 2024-04-25T16:30:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.336+0000 7fb979ffb640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb96c0afb70 0x7fb96c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.336+0000 7fb980b6a640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb96c0afb70 0x7fb96c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.336+0000 7fb979ffb640 1 -- 172.21.15.98:0/4141859557 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fb9680de960 con 0x7fb97c1033d0 2024-04-25T16:30:25.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.336+0000 7fb980b6a640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb96c0afb70 0x7fb96c0b2080 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb9700057c0 tx=0x7fb970000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:25.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.340+0000 7fb979ffb640 1 -- 172.21.15.98:0/4141859557 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb96801f020 con 0x7fb97c1033d0 2024-04-25T16:30:25.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:24 smithi203 ceph-mon[33115]: pgmap v289: 129 pgs: 129 active+clean; 579 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s wr, 0 op/s 2024-04-25T16:30:25.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:24 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3876908782' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:24 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/963041232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:24 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2444320350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:30:25.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.533+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb97c02db50 con 0x7fb97c1033d0 2024-04-25T16:30:25.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.534+0000 7fb979ffb640 1 -- 172.21.15.98:0/4141859557 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) ==== 81+0+589 (secure 0 0 0) 0x7fb96800d040 con 0x7fb97c1033d0 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1883089564 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/119631250 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/217035597 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2573630616 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/4161723298 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3681298222 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/4161723298 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1429360204 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3090764993 2024-04-26T16:22:41.266215+0000 2024-04-25T16:30:25.534 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/217035597 2024-04-26T16:23:07.830236+0000 2024-04-25T16:30:25.535 INFO:teuthology.orchestra.run.smithi098.stderr:listed 10 entries 2024-04-25T16:30:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb96c0afb70 msgr2=0x7fb96c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb96c0afb70 0x7fb96c0b2080 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb9700057c0 tx=0x7fb970000bc0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c1033d0 msgr2=0x7fb97c1997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c1033d0 0x7fb97c1997c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb96800b450 tx=0x7fb96800b910 comp rx=0 tx=0).stop 2024-04-25T16:30:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 shutdown_connections 2024-04-25T16:30:25.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb96c0afb70 0x7fb96c0b2080 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb97c1080d0 0x7fb97c1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb97c1033d0 0x7fb97c1997c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 --2- 172.21.15.98:0/4141859557 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb97c102190 0x7fb97c199280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 >> 172.21.15.98:0/4141859557 conn(0x7fb97c0fd380 msgr2=0x7fb97c0fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 shutdown_connections 2024-04-25T16:30:25.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:25.536+0000 7fb982dce640 1 -- 172.21.15.98:0/4141859557 wait complete. 2024-04-25T16:30:25.547 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-25T16:30:25.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:25.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T16:30:25.569 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi098.stderr:ceph-fuse[72879]: fuse finished with error 0 and tester_r 0 2024-04-25T16:30:25.571 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T16:30:26.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:25 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3762878595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:30:26.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:25 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4141859557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:30:26.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:25 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3762878595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:30:26.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:25 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4141859557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:30:26.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:25 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3762878595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:30:26.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:25 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4141859557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:30:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:26 smithi098 ceph-mon[29162]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:27.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:26 smithi173 ceph-mon[32556]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:27.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:26 smithi203 ceph-mon[33115]: pgmap v290: 129 pgs: 129 active+clean; 579 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T16:30:28.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:27 smithi098 ceph-mon[29162]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:28.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:27 smithi173 ceph-mon[32556]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:28.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:27 smithi203 ceph-mon[33115]: pgmap v291: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:31.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:30 smithi098 ceph-mon[29162]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:31.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:30 smithi173 ceph-mon[32556]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:31.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:30 smithi203 ceph-mon[33115]: pgmap v292: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:31.672 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T16:30:31.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:30:31.673 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T16:30:31.697 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:31.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 -- 172.21.15.98:0/3572754094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c103c10 msgr2=0x7f415c104020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 --2- 172.21.15.98:0/3572754094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c103c10 0x7f415c104020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4158009900 tx=0x7f415802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 -- 172.21.15.98:0/3572754094 shutdown_connections 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 --2- 172.21.15.98:0/3572754094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f415c108910 0x7f415c10ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 --2- 172.21.15.98:0/3572754094 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c103c10 0x7f415c104020 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 --2- 172.21.15.98:0/3572754094 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f415c1029d0 0x7f415c1036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 -- 172.21.15.98:0/3572754094 >> 172.21.15.98:0/3572754094 conn(0x7f415c0fdba0 msgr2=0x7f415c100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 -- 172.21.15.98:0/3572754094 shutdown_connections 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 -- 172.21.15.98:0/3572754094 wait complete. 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 Processor -- start 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.850+0000 7f4163ae0640 1 -- start start 2024-04-25T16:30:31.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4163ae0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c1029d0 0x7f415c06bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4163ae0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f415c103c10 0x7f415c1a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4163ae0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f415c108910 0x7f415c1a3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4163ae0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10e3b0 con 0x7f415c1029d0 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4163ae0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10e230 con 0x7f415c103c10 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4163ae0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10e530 con 0x7f415c108910 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c1029d0 0x7f415c06bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c1029d0 0x7f415c06bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49776/0 (socket says 172.21.15.98:49776) 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 -- 172.21.15.98:0/115272775 learned_addr learned my addr 172.21.15.98:0/115272775 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416107b640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f415c103c10 0x7f415c1a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416207d640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f415c108910 0x7f415c1a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 -- 172.21.15.98:0/115272775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f415c108910 msgr2=0x7f415c1a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f415c108910 0x7f415c1a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:31.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 -- 172.21.15.98:0/115272775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f415c103c10 msgr2=0x7f415c1a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f415c103c10 0x7f415c1a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 -- 172.21.15.98:0/115272775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c1a3e50 con 0x7f415c1029d0 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f416187c640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c1029d0 0x7f415c06bd40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f414c00c970 tx=0x7f414c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4152ffd640 1 -- 172.21.15.98:0/115272775 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c004220 con 0x7f415c1029d0 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4152ffd640 1 -- 172.21.15.98:0/115272775 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f414c007e40 con 0x7f415c1029d0 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415c1a4140 con 0x7f415c1029d0 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.851+0000 7f4152ffd640 1 -- 172.21.15.98:0/115272775 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c00f450 con 0x7f415c1029d0 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.852+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f415c1a4690 con 0x7f415c1029d0 2024-04-25T16:30:31.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.852+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f415c107720 con 0x7f415c1029d0 2024-04-25T16:30:31.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.856+0000 7f4152ffd640 1 -- 172.21.15.98:0/115272775 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f414c0079b0 con 0x7f415c1029d0 2024-04-25T16:30:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.857+0000 7f4152ffd640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41480afaf0 0x7f41480b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.857+0000 7f416107b640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41480afaf0 0x7f41480b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:31.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.857+0000 7f4152ffd640 1 -- 172.21.15.98:0/115272775 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f414c0defb0 con 0x7f415c1029d0 2024-04-25T16:30:31.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.857+0000 7f416107b640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41480afaf0 0x7f41480b2000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4158006010 tx=0x7f4158000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:31.859 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:31.859+0000 7f4152ffd640 1 -- 172.21.15.98:0/115272775 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f414c014030 con 0x7f415c1029d0 2024-04-25T16:30:32.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.044+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f415c0637a0 con 0x7f415c1029d0 2024-04-25T16:30:32.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.046+0000 7f4152ffd640 1 -- 172.21.15.98:0/115272775 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7f414c0a7950 con 0x7f415c1029d0 2024-04-25T16:30:32.046 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:32.046 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:32.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41480afaf0 msgr2=0x7f41480b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41480afaf0 0x7f41480b2000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4158006010 tx=0x7f4158000bc0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c1029d0 msgr2=0x7f415c06bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c1029d0 0x7f415c06bd40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f414c00c970 tx=0x7f414c00ce30 comp rx=0 tx=0).stop 2024-04-25T16:30:32.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 shutdown_connections 2024-04-25T16:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f41480afaf0 0x7f41480b2000 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f415c108910 0x7f415c1a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f415c103c10 0x7f415c1a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 --2- 172.21.15.98:0/115272775 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f415c1029d0 0x7f415c06bd40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 >> 172.21.15.98:0/115272775 conn(0x7f415c0fdba0 msgr2=0x7f415c1048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 shutdown_connections 2024-04-25T16:30:32.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.049+0000 7f4163ae0640 1 -- 172.21.15.98:0/115272775 wait complete. 2024-04-25T16:30:32.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:32.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 -- 172.21.15.98:0/1021608145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81045c0 msgr2=0x7f47e81069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 --2- 172.21.15.98:0/1021608145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81045c0 0x7f47e81069e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f47d8009900 tx=0x7f47d802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:32.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 -- 172.21.15.98:0/1021608145 shutdown_connections 2024-04-25T16:30:32.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 --2- 172.21.15.98:0/1021608145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47e8109880 0x7f47e810bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 --2- 172.21.15.98:0/1021608145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47e8106f20 0x7f47e8109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 --2- 172.21.15.98:0/1021608145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e81045c0 0x7f47e81069e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 -- 172.21.15.98:0/1021608145 >> 172.21.15.98:0/1021608145 conn(0x7f47e80fdba0 msgr2=0x7f47e8100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:32.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 -- 172.21.15.98:0/1021608145 shutdown_connections 2024-04-25T16:30:32.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.175+0000 7f47ee29a640 1 -- 172.21.15.98:0/1021608145 wait complete. 2024-04-25T16:30:32.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ee29a640 1 Processor -- start 2024-04-25T16:30:32.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ee29a640 1 -- start start 2024-04-25T16:30:32.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ee29a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47e81045c0 0x7f47e819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ee29a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e8106f20 0x7f47e819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ee29a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47e8109880 0x7f47e81a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ee29a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f47e810f480 con 0x7f47e8106f20 2024-04-25T16:30:32.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ee29a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f47e810f300 con 0x7f47e8109880 2024-04-25T16:30:32.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ee29a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f47e810f600 con 0x7f47e81045c0 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e8106f20 0x7f47e819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47e77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e8106f20 0x7f47e819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49798/0 (socket says 172.21.15.98:49798) 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47e77fe640 1 -- 172.21.15.98:0/3869225291 learned_addr learned my addr 172.21.15.98:0/3869225291 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47e7fff640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47e81045c0 0x7f47e819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.176+0000 7f47ec837640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47e8109880 0x7f47e81a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e77fe640 1 -- 172.21.15.98:0/3869225291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47e81045c0 msgr2=0x7f47e819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e77fe640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47e81045c0 0x7f47e819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e77fe640 1 -- 172.21.15.98:0/3869225291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47e8109880 msgr2=0x7f47e81a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e77fe640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47e8109880 0x7f47e81a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e77fe640 1 -- 172.21.15.98:0/3869225291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e81a1c80 con 0x7f47e8106f20 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e77fe640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e8106f20 0x7f47e819a6d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c970 tx=0x7f47dc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e57fa640 1 -- 172.21.15.98:0/3869225291 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc007bb0 con 0x7f47e8106f20 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e57fa640 1 -- 172.21.15.98:0/3869225291 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47dc007d50 con 0x7f47e8106f20 2024-04-25T16:30:32.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e81a1f70 con 0x7f47e8106f20 2024-04-25T16:30:32.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47e57fa640 1 -- 172.21.15.98:0/3869225291 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc004e40 con 0x7f47e8106f20 2024-04-25T16:30:32.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.177+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e81a23e0 con 0x7f47e8106f20 2024-04-25T16:30:32.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.178+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e80ff6c0 con 0x7f47e8106f20 2024-04-25T16:30:32.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.180+0000 7f47e57fa640 1 -- 172.21.15.98:0/3869225291 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f47dc005000 con 0x7f47e8106f20 2024-04-25T16:30:32.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.180+0000 7f47e57fa640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47d00afaf0 0x7f47d00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.180+0000 7f47e7fff640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47d00afaf0 0x7f47d00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.180+0000 7f47e57fa640 1 -- 172.21.15.98:0/3869225291 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f47dc0defc0 con 0x7f47e8106f20 2024-04-25T16:30:32.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.181+0000 7f47e7fff640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47d00afaf0 0x7f47d00b2000 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f47d8005ab0 tx=0x7f47d8000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:32.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.184+0000 7f47e57fa640 1 -- 172.21.15.98:0/3869225291 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f47dc014030 con 0x7f47e8106f20 2024-04-25T16:30:32.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.369+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47e802dda0 con 0x7f47e8106f20 2024-04-25T16:30:32.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.371+0000 7f47e57fa640 1 -- 172.21.15.98:0/3869225291 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7f47dc0a7960 con 0x7f47e8106f20 2024-04-25T16:30:32.371 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:32.371 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:32.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47d00afaf0 msgr2=0x7f47d00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47d00afaf0 0x7f47d00b2000 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f47d8005ab0 tx=0x7f47d8000bc0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e8106f20 msgr2=0x7f47e819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e8106f20 0x7f47e819a6d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00c970 tx=0x7f47dc00ce30 comp rx=0 tx=0).stop 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 shutdown_connections 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47d00afaf0 0x7f47d00b2000 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f47e8109880 0x7f47e81a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f47e8106f20 0x7f47e819a6d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 --2- 172.21.15.98:0/3869225291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47e81045c0 0x7f47e819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 >> 172.21.15.98:0/3869225291 conn(0x7f47e80fdba0 msgr2=0x7f47e8106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 shutdown_connections 2024-04-25T16:30:32.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.374+0000 7f47ee29a640 1 -- 172.21.15.98:0/3869225291 wait complete. 2024-04-25T16:30:32.389 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T16:30:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.500+0000 7f948263c640 1 -- 172.21.15.98:0/1144993538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c103c90 msgr2=0x7f947c1040a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.500+0000 7f948263c640 1 --2- 172.21.15.98:0/1144993538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c103c90 0x7f947c1040a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f946c009900 tx=0x7f946c02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.500+0000 7f948263c640 1 -- 172.21.15.98:0/1144993538 shutdown_connections 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.500+0000 7f948263c640 1 --2- 172.21.15.98:0/1144993538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f947c108990 0x7f947c10adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.501+0000 7f948263c640 1 --2- 172.21.15.98:0/1144993538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c103c90 0x7f947c1040a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.501+0000 7f948263c640 1 --2- 172.21.15.98:0/1144993538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f947c102a50 0x7f947c103750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.501+0000 7f948263c640 1 -- 172.21.15.98:0/1144993538 >> 172.21.15.98:0/1144993538 conn(0x7f947c0fdc40 msgr2=0x7f947c1000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.501+0000 7f948263c640 1 -- 172.21.15.98:0/1144993538 shutdown_connections 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.501+0000 7f948263c640 1 -- 172.21.15.98:0/1144993538 wait complete. 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.501+0000 7f948263c640 1 Processor -- start 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.501+0000 7f948263c640 1 -- start start 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f948263c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f947c102a50 0x7f947c075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f948263c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f947c103c90 0x7f947c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f948263c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c108990 0x7f947c1a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f948263c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f947c10e390 con 0x7f947c108990 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f948263c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f947c10e210 con 0x7f947c102a50 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f948263c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f947c10e510 con 0x7f947c103c90 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c108990 0x7f947c1a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c108990 0x7f947c1a14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49828/0 (socket says 172.21.15.98:49828) 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 -- 172.21.15.98:0/3084034993 learned_addr learned my addr 172.21.15.98:0/3084034993 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f947bfff640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f947c102a50 0x7f947c075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f947b7fe640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f947c103c90 0x7f947c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 -- 172.21.15.98:0/3084034993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f947c103c90 msgr2=0x7f947c19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f947c103c90 0x7f947c19ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 -- 172.21.15.98:0/3084034993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f947c102a50 msgr2=0x7f947c075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f947c102a50 0x7f947c075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 -- 172.21.15.98:0/3084034993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c1a1cd0 con 0x7f947c108990 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f947bfff640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f947c102a50 0x7f947c075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f9480bd9640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c108990 0x7f947c1a14b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f947400eaa0 tx=0x7f947400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f94797fa640 1 -- 172.21.15.98:0/3084034993 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947400cdb0 con 0x7f947c108990 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f94797fa640 1 -- 172.21.15.98:0/3084034993 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9474005020 con 0x7f947c108990 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947c1a1fc0 con 0x7f947c108990 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f94797fa640 1 -- 172.21.15.98:0/3084034993 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94740116d0 con 0x7f947c108990 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.502+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947c1a2420 con 0x7f947c108990 2024-04-25T16:30:32.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.504+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f947c1077a0 con 0x7f947c108990 2024-04-25T16:30:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.505+0000 7f94797fa640 1 -- 172.21.15.98:0/3084034993 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f94740051d0 con 0x7f947c108990 2024-04-25T16:30:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.506+0000 7f94797fa640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94640afaf0 0x7f94640b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.506+0000 7f947bfff640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94640afaf0 0x7f94640b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:32.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.506+0000 7f94797fa640 1 -- 172.21.15.98:0/3084034993 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f94740df040 con 0x7f947c108990 2024-04-25T16:30:32.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.506+0000 7f947bfff640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94640afaf0 0x7f94640b2000 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9470005d00 tx=0x7f9470005c70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:32.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.510+0000 7f94797fa640 1 -- 172.21.15.98:0/3084034993 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9474016030 con 0x7f947c108990 2024-04-25T16:30:32.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.730+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f947c063840 con 0x7f947c108990 2024-04-25T16:30:32.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.990+0000 7f94797fa640 1 -- 172.21.15.98:0/3084034993 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f94740a79c0 con 0x7f947c108990 2024-04-25T16:30:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94640afaf0 msgr2=0x7f94640b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94640afaf0 0x7f94640b2000 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9470005d00 tx=0x7f9470005c70 comp rx=0 tx=0).stop 2024-04-25T16:30:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c108990 msgr2=0x7f947c1a14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:32.992 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c108990 0x7f947c1a14b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f947400eaa0 tx=0x7f947400ef60 comp rx=0 tx=0).stop 2024-04-25T16:30:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 shutdown_connections 2024-04-25T16:30:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94640afaf0 0x7f94640b2000 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f947c108990 0x7f947c1a14b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f947c103c90 0x7f947c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 --2- 172.21.15.98:0/3084034993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f947c102a50 0x7f947c075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 >> 172.21.15.98:0/3084034993 conn(0x7f947c0fdc40 msgr2=0x7f947c104970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 shutdown_connections 2024-04-25T16:30:32.993 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:32.992+0000 7f948263c640 1 -- 172.21.15.98:0/3084034993 wait complete. 2024-04-25T16:30:33.004 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-04-25T16:30:33.004 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-25T16:30:33.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:32 smithi098 ceph-mon[29162]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:33.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:32 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/115272775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:33.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:32 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3869225291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:33.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:32 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3084034993' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 -- 172.21.15.98:0/1443438423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b4108920 msgr2=0x7f49b410ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 --2- 172.21.15.98:0/1443438423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b4108920 0x7f49b410ad40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f49b0009900 tx=0x7f49b002a410 comp rx=0 tx=0).stop 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 -- 172.21.15.98:0/1443438423 shutdown_connections 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 --2- 172.21.15.98:0/1443438423 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b4108920 0x7f49b410ad40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 --2- 172.21.15.98:0/1443438423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b4103c20 0x7f49b4104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 --2- 172.21.15.98:0/1443438423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49b41029e0 0x7f49b41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 -- 172.21.15.98:0/1443438423 >> 172.21.15.98:0/1443438423 conn(0x7f49b40fdbd0 msgr2=0x7f49b4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 -- 172.21.15.98:0/1443438423 shutdown_connections 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 -- 172.21.15.98:0/1443438423 wait complete. 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 Processor -- start 2024-04-25T16:30:33.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.116+0000 7f49bb292640 1 -- start start 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49bb292640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b41029e0 0x7f49b4199320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49bb292640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49b4103c20 0x7f49b4199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49bb292640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b4108920 0x7f49b41a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49bb292640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49b410e3c0 con 0x7f49b41029e0 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49bb292640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49b410e240 con 0x7f49b4108920 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49bb292640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f49b410e540 con 0x7f49b4103c20 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b41029e0 0x7f49b4199320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b41029e0 0x7f49b4199320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49850/0 (socket says 172.21.15.98:49850) 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 -- 172.21.15.98:0/3864793509 learned_addr learned my addr 172.21.15.98:0/3864793509 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b882d640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49b4103c20 0x7f49b4199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 -- 172.21.15.98:0/3864793509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49b4103c20 msgr2=0x7f49b4199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49b4103c20 0x7f49b4199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 -- 172.21.15.98:0/3864793509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b4108920 msgr2=0x7f49b41a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:33.119 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b4108920 0x7f49b41a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 -- 172.21.15.98:0/3864793509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b41a0e60 con 0x7f49b41029e0 2024-04-25T16:30:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49b902e640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b41029e0 0x7f49b4199320 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00ead0 tx=0x7f49ac00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49aa7fc640 1 -- 172.21.15.98:0/3864793509 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac00cde0 con 0x7f49b41029e0 2024-04-25T16:30:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49aa7fc640 1 -- 172.21.15.98:0/3864793509 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f49ac003e00 con 0x7f49b41029e0 2024-04-25T16:30:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b41a1150 con 0x7f49b41029e0 2024-04-25T16:30:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49aa7fc640 1 -- 172.21.15.98:0/3864793509 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49ac0116d0 con 0x7f49b41029e0 2024-04-25T16:30:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.117+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b41a1650 con 0x7f49b41029e0 2024-04-25T16:30:33.120 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.118+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b4107730 con 0x7f49b41029e0 2024-04-25T16:30:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.120+0000 7f49aa7fc640 1 -- 172.21.15.98:0/3864793509 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f49ac003970 con 0x7f49b41029e0 2024-04-25T16:30:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.120+0000 7f49aa7fc640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49a00afaf0 0x7f49a00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.121+0000 7f49b882d640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49a00afaf0 0x7f49a00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.121+0000 7f49aa7fc640 1 -- 172.21.15.98:0/3864793509 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f49ac0df5f0 con 0x7f49b41029e0 2024-04-25T16:30:33.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.121+0000 7f49b882d640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49a00afaf0 0x7f49a00b2000 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f49a4005fd0 tx=0x7f49a4005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:33.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.124+0000 7f49aa7fc640 1 -- 172.21.15.98:0/3864793509 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49ac015030 con 0x7f49b41029e0 2024-04-25T16:30:33.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:32 smithi173 ceph-mon[32556]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:33.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:32 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/115272775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:33.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:32 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3869225291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:33.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:32 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3084034993' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T16:30:33.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:33.340+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f49b402dd90 con 0x7f49b41029e0 2024-04-25T16:30:33.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:32 smithi203 ceph-mon[33115]: pgmap v293: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:33.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:32 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/115272775' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:33.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:32 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3869225291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:33.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:32 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3084034993' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T16:30:33.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.000+0000 7f49aa7fc640 1 -- 172.21.15.98:0/3864793509 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f49ac0a7f90 con 0x7f49b41029e0 2024-04-25T16:30:34.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49a00afaf0 msgr2=0x7f49a00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:34.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49a00afaf0 0x7f49a00b2000 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f49a4005fd0 tx=0x7f49a4005850 comp rx=0 tx=0).stop 2024-04-25T16:30:34.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b41029e0 msgr2=0x7f49b4199320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:34.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b41029e0 0x7f49b4199320 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f49ac00ead0 tx=0x7f49ac00ef90 comp rx=0 tx=0).stop 2024-04-25T16:30:34.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 shutdown_connections 2024-04-25T16:30:34.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49a00afaf0 0x7f49a00b2000 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49b4108920 0x7f49b41a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49b4103c20 0x7f49b4199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 --2- 172.21.15.98:0/3864793509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49b41029e0 0x7f49b4199320 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 >> 172.21.15.98:0/3864793509 conn(0x7f49b40fdbd0 msgr2=0x7f49b4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:34.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 shutdown_connections 2024-04-25T16:30:34.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.002+0000 7f49bb292640 1 -- 172.21.15.98:0/3864793509 wait complete. 2024-04-25T16:30:34.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T16:30:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 -- 172.21.15.98:0/2885928788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004108940 msgr2=0x7f900410ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 --2- 172.21.15.98:0/2885928788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004108940 0x7f900410ad60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8ffc009ca0 tx=0x7f8ffc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 -- 172.21.15.98:0/2885928788 shutdown_connections 2024-04-25T16:30:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 --2- 172.21.15.98:0/2885928788 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004108940 0x7f900410ad60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 --2- 172.21.15.98:0/2885928788 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9004103c40 0x7f9004104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 --2- 172.21.15.98:0/2885928788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9004102a00 0x7f9004103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 -- 172.21.15.98:0/2885928788 >> 172.21.15.98:0/2885928788 conn(0x7f90040fdbf0 msgr2=0x7f90041000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 -- 172.21.15.98:0/2885928788 shutdown_connections 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.170+0000 7f900c215640 1 -- 172.21.15.98:0/2885928788 wait complete. 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900c215640 1 Processor -- start 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900c215640 1 -- start start 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900c215640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004102a00 0x7f9004198fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900c215640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9004103c40 0x7f90041994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900c215640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9004108940 0x7f90041a0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900c215640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f900410e410 con 0x7f9004102a00 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900c215640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f900410e290 con 0x7f9004103c40 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900c215640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f900410e590 con 0x7f9004108940 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f9009fb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004102a00 0x7f9004198fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:34.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f9009fb1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004102a00 0x7f9004198fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49870/0 (socket says 172.21.15.98:49870) 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f9009fb1640 1 -- 172.21.15.98:0/3478796435 learned_addr learned my addr 172.21.15.98:0/3478796435 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f900a7b2640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9004108940 0x7f90041a0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f90097b0640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9004103c40 0x7f90041994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f9009fb1640 1 -- 172.21.15.98:0/3478796435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9004108940 msgr2=0x7f90041a0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f9009fb1640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9004108940 0x7f90041a0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f9009fb1640 1 -- 172.21.15.98:0/3478796435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9004103c40 msgr2=0x7f90041994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f9009fb1640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9004103c40 0x7f90041994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.171+0000 7f9009fb1640 1 -- 172.21.15.98:0/3478796435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90041a0af0 con 0x7f9004102a00 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.172+0000 7f9009fb1640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004102a00 0x7f9004198fb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800d740 tx=0x7f8ff800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.172+0000 7f8ff6ffd640 1 -- 172.21.15.98:0/3478796435 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8017070 con 0x7f9004102a00 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.172+0000 7f8ff6ffd640 1 -- 172.21.15.98:0/3478796435 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8ff800de50 con 0x7f9004102a00 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.172+0000 7f8ff6ffd640 1 -- 172.21.15.98:0/3478796435 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ff8003bb0 con 0x7f9004102a00 2024-04-25T16:30:34.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.172+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90041a0de0 con 0x7f9004102a00 2024-04-25T16:30:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.172+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90041a12a0 con 0x7f9004102a00 2024-04-25T16:30:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.173+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9004107750 con 0x7f9004102a00 2024-04-25T16:30:34.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.174+0000 7f8ff6ffd640 1 -- 172.21.15.98:0/3478796435 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8ff8003d50 con 0x7f9004102a00 2024-04-25T16:30:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.175+0000 7f8ff6ffd640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8ff00afb70 0x7f8ff00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.175+0000 7f90097b0640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8ff00afb70 0x7f8ff00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.175+0000 7f8ff6ffd640 1 -- 172.21.15.98:0/3478796435 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f8ff80debd0 con 0x7f9004102a00 2024-04-25T16:30:34.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.175+0000 7f90097b0640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8ff00afb70 0x7f8ff00b2080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f900419a050 tx=0x7f900000b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:34.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.179+0000 7f8ff6ffd640 1 -- 172.21.15.98:0/3478796435 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ff80a75f0 con 0x7f9004102a00 2024-04-25T16:30:34.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:33 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3084034993' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T16:30:34.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:33 smithi098 ceph-mon[29162]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:34.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:33 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3864793509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T16:30:34.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:33 smithi098 ceph-mon[29162]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:34.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:33 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3084034993' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T16:30:34.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:33 smithi173 ceph-mon[32556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:34.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:33 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3864793509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T16:30:34.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:33 smithi173 ceph-mon[32556]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:34.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:34.397+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f900402db50 con 0x7f9004102a00 2024-04-25T16:30:34.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:33 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3084034993' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T16:30:34.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:33 smithi203 ceph-mon[33115]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:34.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:33 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3864793509' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T16:30:34.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:33 smithi203 ceph-mon[33115]: pgmap v294: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:35.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.008+0000 7f8ff6ffd640 1 -- 172.21.15.98:0/3478796435 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f8ff80a6ef0 con 0x7f9004102a00 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8ff00afb70 msgr2=0x7f8ff00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8ff00afb70 0x7f8ff00b2080 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f900419a050 tx=0x7f900000b5b0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004102a00 msgr2=0x7f9004198fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004102a00 0x7f9004198fb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8ff800d740 tx=0x7f8ff800dc00 comp rx=0 tx=0).stop 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 shutdown_connections 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8ff00afb70 0x7f8ff00b2080 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9004108940 0x7f90041a0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9004103c40 0x7f90041994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 --2- 172.21.15.98:0/3478796435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9004102a00 0x7f9004198fb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 >> 172.21.15.98:0/3478796435 conn(0x7f90040fdbf0 msgr2=0x7f9004104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:35.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 shutdown_connections 2024-04-25T16:30:35.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.011+0000 7f900c215640 1 -- 172.21.15.98:0/3478796435 wait complete. 2024-04-25T16:30:35.027 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T16:30:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 -- 172.21.15.98:0/4261142224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 msgr2=0x7fa2a410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 --2- 172.21.15.98:0/4261142224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 0x7fa2a410be50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009be0 tx=0x7fa29c02a6c0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 -- 172.21.15.98:0/4261142224 shutdown_connections 2024-04-25T16:30:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 --2- 172.21.15.98:0/4261142224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 0x7fa2a410be50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 --2- 172.21.15.98:0/4261142224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2a4104d30 0x7fa2a4105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 --2- 172.21.15.98:0/4261142224 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2a4103af0 0x7fa2a41047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.145 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 -- 172.21.15.98:0/4261142224 >> 172.21.15.98:0/4261142224 conn(0x7fa2a40fece0 msgr2=0x7fa2a4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 -- 172.21.15.98:0/4261142224 shutdown_connections 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.143+0000 7fa2ac346640 1 -- 172.21.15.98:0/4261142224 wait complete. 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2ac346640 1 Processor -- start 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2ac346640 1 -- start start 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2ac346640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2a4103af0 0x7fa2a419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2ac346640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2a4104d30 0x7fa2a419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2ac346640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 0x7fa2a41a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2ac346640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a410f4d0 con 0x7fa2a4104d30 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2ac346640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a410f350 con 0x7fa2a4109a30 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2ac346640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a410f650 con 0x7fa2a4103af0 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2a4104d30 0x7fa2a419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2a4104d30 0x7fa2a419a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:49900/0 (socket says 172.21.15.98:49900) 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 -- 172.21.15.98:0/3806891309 learned_addr learned my addr 172.21.15.98:0/3806891309 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:35.146 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2aa8e3640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 0x7fa2a41a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2aa0e2640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2a4103af0 0x7fa2a419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 -- 172.21.15.98:0/3806891309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2a4103af0 msgr2=0x7fa2a419a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2a4103af0 0x7fa2a419a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 -- 172.21.15.98:0/3806891309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 msgr2=0x7fa2a41a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 0x7fa2a41a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 -- 172.21.15.98:0/3806891309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a41a1c80 con 0x7fa2a4104d30 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2aa8e3640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 0x7fa2a41a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.144+0000 7fa2a98e1640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2a4104d30 0x7fa2a419a680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000d770 tx=0x7fa2a000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.145+0000 7fa29b7fe640 1 -- 172.21.15.98:0/3806891309 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0017070 con 0x7fa2a4104d30 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.145+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a41a1f70 con 0x7fa2a4104d30 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.145+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2a41a23e0 con 0x7fa2a4104d30 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.145+0000 7fa29b7fe640 1 -- 172.21.15.98:0/3806891309 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa2a0009c30 con 0x7fa2a4104d30 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.145+0000 7fa29b7fe640 1 -- 172.21.15.98:0/3806891309 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0020060 con 0x7fa2a4104d30 2024-04-25T16:30:35.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.146+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2880052c0 con 0x7fa2a4104d30 2024-04-25T16:30:35.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.147+0000 7fa29b7fe640 1 -- 172.21.15.98:0/3806891309 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa2a0010460 con 0x7fa2a4104d30 2024-04-25T16:30:35.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.148+0000 7fa29b7fe640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa2900afaf0 0x7fa2900b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:35.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.148+0000 7fa2aa0e2640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa2900afaf0 0x7fa2900b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.148+0000 7fa29b7fe640 1 -- 172.21.15.98:0/3806891309 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fa2a00de7b0 con 0x7fa2a4104d30 2024-04-25T16:30:35.149 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.148+0000 7fa2aa0e2640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa2900afaf0 0x7fa2900b2000 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa2940096c0 tx=0x7fa294009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:35.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.152+0000 7fa29b7fe640 1 -- 172.21.15.98:0/3806891309 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2a00a7150 con 0x7fa2a4104d30 2024-04-25T16:30:35.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:34 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T16:30:35.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:34 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3864793509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T16:30:35.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:34 smithi098 ceph-mon[29162]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T16:30:35.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:34 smithi098 ceph-mon[29162]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:35.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:34 smithi098 ceph-mon[29162]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-25T16:30:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:34 smithi098 ceph-mon[29162]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-25T16:30:35.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:34 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3478796435' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T16:30:35.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:34 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T16:30:35.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:34 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3864793509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T16:30:35.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:34 smithi173 ceph-mon[32556]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T16:30:35.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:34 smithi173 ceph-mon[32556]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:35.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:34 smithi173 ceph-mon[32556]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-25T16:30:35.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:34 smithi173 ceph-mon[32556]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-25T16:30:35.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:34 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3478796435' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T16:30:35.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:35.369+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fa2880055b0 con 0x7fa2a4104d30 2024-04-25T16:30:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:34 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T16:30:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:34 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3864793509' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T16:30:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:34 smithi203 ceph-mon[33115]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T16:30:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:34 smithi203 ceph-mon[33115]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T16:30:35.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:34 smithi203 ceph-mon[33115]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-04-25T16:30:35.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:34 smithi203 ceph-mon[33115]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-04-25T16:30:35.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:34 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3478796435' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T16:30:36.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.020+0000 7fa29b7fe640 1 -- 172.21.15.98:0/3806891309 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fa2a00a6a50 con 0x7fa2a4104d30 2024-04-25T16:30:36.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa2900afaf0 msgr2=0x7fa2900b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa2900afaf0 0x7fa2900b2000 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa2940096c0 tx=0x7fa294009260 comp rx=0 tx=0).stop 2024-04-25T16:30:36.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2a4104d30 msgr2=0x7fa2a419a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2a4104d30 0x7fa2a419a680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa2a000d770 tx=0x7fa2a000dc30 comp rx=0 tx=0).stop 2024-04-25T16:30:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 shutdown_connections 2024-04-25T16:30:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa2900afaf0 0x7fa2900b2000 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2a4109a30 0x7fa2a41a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa2a4104d30 0x7fa2a419a680 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 --2- 172.21.15.98:0/3806891309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa2a4103af0 0x7fa2a419a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.022+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 >> 172.21.15.98:0/3806891309 conn(0x7fa2a40fece0 msgr2=0x7fa2a4105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.023+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 shutdown_connections 2024-04-25T16:30:36.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.023+0000 7fa2ac346640 1 -- 172.21.15.98:0/3806891309 wait complete. 2024-04-25T16:30:36.038 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-04-25T16:30:36.039 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T16:30:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 -- 172.21.15.98:0/1887726538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 msgr2=0x7f7ca81047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 --2- 172.21.15.98:0/1887726538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 0x7f7ca81047e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7c98009900 tx=0x7f7c9802a410 comp rx=0 tx=0).stop 2024-04-25T16:30:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 -- 172.21.15.98:0/1887726538 shutdown_connections 2024-04-25T16:30:36.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 --2- 172.21.15.98:0/1887726538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8109a20 0x7f7ca810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 --2- 172.21.15.98:0/1887726538 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8104d20 0x7f7ca8105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 --2- 172.21.15.98:0/1887726538 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 0x7f7ca81047e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 -- 172.21.15.98:0/1887726538 >> 172.21.15.98:0/1887726538 conn(0x7f7ca80fed10 msgr2=0x7f7ca8101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 -- 172.21.15.98:0/1887726538 shutdown_connections 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.152+0000 7f7cad51f640 1 -- 172.21.15.98:0/1887726538 wait complete. 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7cad51f640 1 Processor -- start 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7cad51f640 1 -- start start 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7cad51f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 0x7f7ca819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7cad51f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8104d20 0x7f7ca819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7cad51f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8109a20 0x7f7ca81a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7cad51f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca810f4f0 con 0x7f7ca8103ae0 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7cad51f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca810f370 con 0x7f7ca8109a20 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7cad51f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7ca810f670 con 0x7f7ca8104d20 2024-04-25T16:30:36.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7ca6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 0x7f7ca819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7ca6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 0x7f7ca819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36926/0 (socket says 172.21.15.98:36926) 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7ca6ffd640 1 -- 172.21.15.98:0/3639530965 learned_addr learned my addr 172.21.15.98:0/3639530965 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7ca77fe640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8109a20 0x7f7ca81a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7ca6ffd640 1 -- 172.21.15.98:0/3639530965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8104d20 msgr2=0x7f7ca819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.153+0000 7f7c9ffff640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8104d20 0x7f7ca819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7ca6ffd640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8104d20 0x7f7ca819a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7ca6ffd640 1 -- 172.21.15.98:0/3639530965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8109a20 msgr2=0x7f7ca81a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7ca6ffd640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8109a20 0x7f7ca81a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7ca6ffd640 1 -- 172.21.15.98:0/3639530965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ca81a1d30 con 0x7f7ca8103ae0 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7ca6ffd640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 0x7f7ca819a1f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7c98005760 tx=0x7f7c98004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7ca4ff9640 1 -- 172.21.15.98:0/3639530965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9803f030 con 0x7f7ca8103ae0 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7ca4ff9640 1 -- 172.21.15.98:0/3639530965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c98004440 con 0x7f7ca8103ae0 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ca81a1fc0 con 0x7f7ca8103ae0 2024-04-25T16:30:36.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7ca4ff9640 1 -- 172.21.15.98:0/3639530965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c98035070 con 0x7f7ca8103ae0 2024-04-25T16:30:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.154+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ca81a24a0 con 0x7f7ca8103ae0 2024-04-25T16:30:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.155+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ca8108830 con 0x7f7ca8103ae0 2024-04-25T16:30:36.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.156+0000 7f7ca4ff9640 1 -- 172.21.15.98:0/3639530965 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f7c98042030 con 0x7f7ca8103ae0 2024-04-25T16:30:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.157+0000 7f7ca4ff9640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c940afb70 0x7f7c940b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.157+0000 7f7c9ffff640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c940afb70 0x7f7c940b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.158+0000 7f7ca4ff9640 1 -- 172.21.15.98:0/3639530965 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f7c980ff5e0 con 0x7f7ca8103ae0 2024-04-25T16:30:36.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.158+0000 7f7c9ffff640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c940afb70 0x7f7c940b2080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7ca819b290 tx=0x7f7c90008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:36.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.161+0000 7f7ca4ff9640 1 -- 172.21.15.98:0/3639530965 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c980c7f80 con 0x7f7ca8103ae0 2024-04-25T16:30:36.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3478796435' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T16:30:36.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:36 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] up:active 2024-04-25T16:30:36.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:36 smithi173 ceph-mon[32556]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T16:30:36.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:36 smithi173 ceph-mon[32556]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-25T16:30:36.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:36 smithi173 ceph-mon[32556]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-25T16:30:36.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:36 smithi173 ceph-mon[32556]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:30:36.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3806891309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T16:30:36.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:36 smithi173 ceph-mon[32556]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:36.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.365+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7ca8062fe0 con 0x7f7c940afb70 2024-04-25T16:30:36.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3478796435' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T16:30:36.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] up:active 2024-04-25T16:30:36.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-mon[29162]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T16:30:36.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-mon[29162]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-25T16:30:36.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-mon[29162]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-25T16:30:36.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-mon[29162]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:30:36.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3806891309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T16:30:36.371 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-mon[29162]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:36.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.394+0000 7f7ca4ff9640 1 -- 172.21.15.98:0/3639530965 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7ca8062fe0 con 0x7f7c940afb70 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c940afb70 msgr2=0x7f7c940b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c940afb70 0x7f7c940b2080 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7ca819b290 tx=0x7f7c90008040 comp rx=0 tx=0).stop 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 msgr2=0x7f7ca819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 0x7f7ca819a1f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7c98005760 tx=0x7f7c98004040 comp rx=0 tx=0).stop 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 shutdown_connections 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c940afb70 0x7f7c940b2080 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ca8109a20 0x7f7ca81a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ca8104d20 0x7f7ca819a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 --2- 172.21.15.98:0/3639530965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7ca8103ae0 0x7f7ca819a1f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 >> 172.21.15.98:0/3639530965 conn(0x7f7ca80fed10 msgr2=0x7f7ca8105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 shutdown_connections 2024-04-25T16:30:36.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.396+0000 7f7cad51f640 1 -- 172.21.15.98:0/3639530965 wait complete. 2024-04-25T16:30:36.407 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T16:30:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3478796435' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T16:30:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:36 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] up:active 2024-04-25T16:30:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:36 smithi203 ceph-mon[33115]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T16:30:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:36 smithi203 ceph-mon[33115]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-04-25T16:30:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:36 smithi203 ceph-mon[33115]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-04-25T16:30:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:36 smithi203 ceph-mon[33115]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:30:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3806891309' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T16:30:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:36 smithi203 ceph-mon[33115]: pgmap v295: 129 pgs: 129 active+clean; 581 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-25T16:30:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.519+0000 7f3076b38640 1 -- 172.21.15.98:0/54350158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070102250 msgr2=0x7f307010bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.519+0000 7f3076b38640 1 --2- 172.21.15.98:0/54350158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070102250 0x7f307010bcb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3068009ca0 tx=0x7f306802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.519+0000 7f3076b38640 1 -- 172.21.15.98:0/54350158 shutdown_connections 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.519+0000 7f3076b38640 1 --2- 172.21.15.98:0/54350158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070102250 0x7f307010bcb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.519+0000 7f3076b38640 1 --2- 172.21.15.98:0/54350158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3070101750 0x7f3070101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 --2- 172.21.15.98:0/54350158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30701050a0 0x7f30701054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 -- 172.21.15.98:0/54350158 >> 172.21.15.98:0/54350158 conn(0x7f30700fd350 msgr2=0x7f30700ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 -- 172.21.15.98:0/54350158 shutdown_connections 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 -- 172.21.15.98:0/54350158 wait complete. 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 Processor -- start 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 -- start start 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070101750 0x7f3070104250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3070102250 0x7f30701028f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30701050a0 0x7f3070102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f307010f530 con 0x7f3070101750 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f307010f3b0 con 0x7f3070102250 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f3076b38640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f307010f6b0 con 0x7f30701050a0 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.520+0000 7f30748d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070101750 0x7f3070104250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30748d4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070101750 0x7f3070104250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36964/0 (socket says 172.21.15.98:36964) 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30748d4640 1 -- 172.21.15.98:0/4013463010 learned_addr learned my addr 172.21.15.98:0/4013463010 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f3067fff640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3070102250 0x7f30701028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30750d5640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30701050a0 0x7f3070102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30748d4640 1 -- 172.21.15.98:0/4013463010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30701050a0 msgr2=0x7f3070102e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30748d4640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30701050a0 0x7f3070102e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30748d4640 1 -- 172.21.15.98:0/4013463010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3070102250 msgr2=0x7f30701028f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30748d4640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3070102250 0x7f30701028f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30748d4640 1 -- 172.21.15.98:0/4013463010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3070103560 con 0x7f3070101750 2024-04-25T16:30:36.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f30748d4640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070101750 0x7f3070104250 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f306c00e9f0 tx=0x7f306c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f3065ffb640 1 -- 172.21.15.98:0/4013463010 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c00ccc0 con 0x7f3070101750 2024-04-25T16:30:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f3065ffb640 1 -- 172.21.15.98:0/4013463010 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f306c0035f0 con 0x7f3070101750 2024-04-25T16:30:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30701a41a0 con 0x7f3070101750 2024-04-25T16:30:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f3065ffb640 1 -- 172.21.15.98:0/4013463010 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f306c00ce60 con 0x7f3070101750 2024-04-25T16:30:36.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.521+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30701a4660 con 0x7f3070101750 2024-04-25T16:30:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.522+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f307010aa40 con 0x7f3070101750 2024-04-25T16:30:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.524+0000 7f3065ffb640 1 -- 172.21.15.98:0/4013463010 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f306c004000 con 0x7f3070101750 2024-04-25T16:30:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.524+0000 7f3065ffb640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30580afb70 0x7f30580b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.524+0000 7f3067fff640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30580afb70 0x7f30580b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.524+0000 7f3065ffb640 1 -- 172.21.15.98:0/4013463010 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f306c0df260 con 0x7f3070101750 2024-04-25T16:30:36.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.525+0000 7f3067fff640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30580afb70 0x7f30580b2080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3070103bf0 tx=0x7f3060005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:36.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.528+0000 7f3065ffb640 1 -- 172.21.15.98:0/4013463010 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f306c0a7c00 con 0x7f3070101750 2024-04-25T16:30:36.731 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.732+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3070062fe0 con 0x7f30580afb70 2024-04-25T16:30:36.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:30:36.370+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:36.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:30:36.370+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:36.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:30:36.370+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:36.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:30:36.370+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:36.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:30:36 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:30:36.370+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:30:36.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.760+0000 7f3065ffb640 1 -- 172.21.15.98:0/4013463010 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3070062fe0 con 0x7f30580afb70 2024-04-25T16:30:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30580afb70 msgr2=0x7f30580b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30580afb70 0x7f30580b2080 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3070103bf0 tx=0x7f3060005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070101750 msgr2=0x7f3070104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070101750 0x7f3070104250 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f306c00e9f0 tx=0x7f306c00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 shutdown_connections 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30580afb70 0x7f30580b2080 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30701050a0 0x7f3070102e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3070102250 0x7f30701028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 --2- 172.21.15.98:0/4013463010 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3070101750 0x7f3070104250 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 >> 172.21.15.98:0/4013463010 conn(0x7f30700fd350 msgr2=0x7f3070107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 shutdown_connections 2024-04-25T16:30:36.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.763+0000 7f3076b38640 1 -- 172.21.15.98:0/4013463010 wait complete. 2024-04-25T16:30:36.773 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T16:30:36.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 -- 172.21.15.98:0/2700825601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc1046a0 msgr2=0x7f08bc10adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 --2- 172.21.15.98:0/2700825601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc1046a0 0x7f08bc10adf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f08b4009ca0 tx=0x7f08b402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 -- 172.21.15.98:0/2700825601 shutdown_connections 2024-04-25T16:30:36.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 --2- 172.21.15.98:0/2700825601 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc1046a0 0x7f08bc10adf0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 --2- 172.21.15.98:0/2700825601 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08bc103b40 0x7f08bc104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 --2- 172.21.15.98:0/2700825601 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc102fe0 0x7f08bc1033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 -- 172.21.15.98:0/2700825601 >> 172.21.15.98:0/2700825601 conn(0x7f08bc0fd380 msgr2=0x7f08bc0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 -- 172.21.15.98:0/2700825601 shutdown_connections 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.886+0000 7f08c4706640 1 -- 172.21.15.98:0/2700825601 wait complete. 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c4706640 1 Processor -- start 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c4706640 1 -- start start 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c4706640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc102fe0 0x7f08bc199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c4706640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08bc103b40 0x7f08bc19a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c4706640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc19f480 0x7f08bc1a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c4706640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc10e6c0 con 0x7f08bc19f480 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c4706640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc10e540 con 0x7f08bc103b40 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c4706640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f08bc10e840 con 0x7f08bc102fe0 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c2ca3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc19f480 0x7f08bc1a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c2ca3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc19f480 0x7f08bc1a18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36984/0 (socket says 172.21.15.98:36984) 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c2ca3640 1 -- 172.21.15.98:0/2511231495 learned_addr learned my addr 172.21.15.98:0/2511231495 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c1ca1640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08bc103b40 0x7f08bc19a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c24a2640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc102fe0 0x7f08bc199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c2ca3640 1 -- 172.21.15.98:0/2511231495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc102fe0 msgr2=0x7f08bc199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c2ca3640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc102fe0 0x7f08bc199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c2ca3640 1 -- 172.21.15.98:0/2511231495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08bc103b40 msgr2=0x7f08bc19a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c2ca3640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08bc103b40 0x7f08bc19a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.887+0000 7f08c2ca3640 1 -- 172.21.15.98:0/2511231495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08bc1a1f10 con 0x7f08bc19f480 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.888+0000 7f08c2ca3640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc19f480 0x7f08bc1a18a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f08b4005e30 tx=0x7f08b4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.888+0000 7f08b37fe640 1 -- 172.21.15.98:0/2511231495 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4032e90 con 0x7f08bc19f480 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.888+0000 7f08b37fe640 1 -- 172.21.15.98:0/2511231495 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f08b40044d0 con 0x7f08bc19f480 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.888+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08bc1a21a0 con 0x7f08bc19f480 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.888+0000 7f08b37fe640 1 -- 172.21.15.98:0/2511231495 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08b4030ce0 con 0x7f08bc19f480 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.888+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08bc1a2740 con 0x7f08bc19f480 2024-04-25T16:30:36.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.889+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08bc06b350 con 0x7f08bc19f480 2024-04-25T16:30:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.890+0000 7f08b37fe640 1 -- 172.21.15.98:0/2511231495 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f08b4004040 con 0x7f08bc19f480 2024-04-25T16:30:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.891+0000 7f08b37fe640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f08a80afb70 0x7f08a80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:36.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.891+0000 7f08c24a2640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f08a80afb70 0x7f08a80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.891+0000 7f08b37fe640 1 -- 172.21.15.98:0/2511231495 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f08b4100500 con 0x7f08bc19f480 2024-04-25T16:30:36.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.891+0000 7f08c24a2640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f08a80afb70 0x7f08a80b2080 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f08bc103a40 tx=0x7f08b8008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:36.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:36.894+0000 7f08b37fe640 1 -- 172.21.15.98:0/2511231495 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08b40c8ea0 con 0x7f08bc19f480 2024-04-25T16:30:37.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.105+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f08bc101f20 con 0x7f08a80afb70 2024-04-25T16:30:37.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.110+0000 7f08b37fe640 1 -- 172.21.15.98:0/2511231495 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f08bc101f20 con 0x7f08a80afb70 2024-04-25T16:30:37.110 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_0/a4566d7d-75ee-46c1-9e38-d177d26b4a22 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.112+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f08a80afb70 msgr2=0x7f08a80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.112+0000 7f08c4706640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f08a80afb70 0x7f08a80b2080 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f08bc103a40 tx=0x7f08b8008040 comp rx=0 tx=0).stop 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.112+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc19f480 msgr2=0x7f08bc1a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.112+0000 7f08c4706640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc19f480 0x7f08bc1a18a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f08b4005e30 tx=0x7f08b4003ba0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.112+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 shutdown_connections 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.112+0000 7f08c4706640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f08a80afb70 0x7f08a80b2080 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.112+0000 7f08c4706640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f08bc19f480 0x7f08bc1a18a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.113+0000 7f08c4706640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08bc103b40 0x7f08bc19a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.113+0000 7f08c4706640 1 --2- 172.21.15.98:0/2511231495 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc102fe0 0x7f08bc199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.113+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 >> 172.21.15.98:0/2511231495 conn(0x7f08bc0fd380 msgr2=0x7f08bc101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.113+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 shutdown_connections 2024-04-25T16:30:37.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.113+0000 7f08c4706640 1 -- 172.21.15.98:0/2511231495 wait complete. 2024-04-25T16:30:37.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:37 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3806891309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T16:30:37.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:37 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] up:active 2024-04-25T16:30:37.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:37 smithi098 ceph-mon[29162]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T16:30:37.123 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:37 smithi098 ceph-mon[29162]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-25T16:30:37.124 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:37 smithi098 ceph-mon[29162]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-04-25T16:30:37.124 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:37 smithi098 ceph-mon[29162]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-25T16:30:37.124 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:37 smithi098 ceph-mon[29162]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:37.124 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:37 smithi098 ceph-mon[29162]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:37.125 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T16:30:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 -- 172.21.15.98:0/1601725355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 msgr2=0x7efd701047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 --2- 172.21.15.98:0/1601725355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 0x7efd701047f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7efd6c009900 tx=0x7efd6c02a410 comp rx=0 tx=0).stop 2024-04-25T16:30:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 -- 172.21.15.98:0/1601725355 shutdown_connections 2024-04-25T16:30:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 --2- 172.21.15.98:0/1601725355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd70109a30 0x7efd7010be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 --2- 172.21.15.98:0/1601725355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd70104d30 0x7efd70105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 --2- 172.21.15.98:0/1601725355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 0x7efd701047f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 -- 172.21.15.98:0/1601725355 >> 172.21.15.98:0/1601725355 conn(0x7efd700fece0 msgr2=0x7efd70101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 -- 172.21.15.98:0/1601725355 shutdown_connections 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.238+0000 7efd77092640 1 -- 172.21.15.98:0/1601725355 wait complete. 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd77092640 1 Processor -- start 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd77092640 1 -- start start 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd77092640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 0x7efd7019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd77092640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd70104d30 0x7efd7019a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd77092640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd70109a30 0x7efd701a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd77092640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efd7010f4a0 con 0x7efd70103af0 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd77092640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efd7010f320 con 0x7efd70104d30 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd77092640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efd7010f620 con 0x7efd70109a30 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd74e2e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 0x7efd7019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd74e2e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 0x7efd7019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:36998/0 (socket says 172.21.15.98:36998) 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd74e2e640 1 -- 172.21.15.98:0/2431555103 learned_addr learned my addr 172.21.15.98:0/2431555103 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd67fff640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd70104d30 0x7efd7019a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.239+0000 7efd7562f640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd70109a30 0x7efd701a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd74e2e640 1 -- 172.21.15.98:0/2431555103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd70109a30 msgr2=0x7efd701a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd74e2e640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd70109a30 0x7efd701a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd74e2e640 1 -- 172.21.15.98:0/2431555103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd70104d30 msgr2=0x7efd7019a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd74e2e640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd70104d30 0x7efd7019a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd74e2e640 1 -- 172.21.15.98:0/2431555103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd701a1cd0 con 0x7efd70103af0 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd74e2e640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 0x7efd7019a190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efd6c005760 tx=0x7efd6c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd65ffb640 1 -- 172.21.15.98:0/2431555103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c03f030 con 0x7efd70103af0 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd65ffb640 1 -- 172.21.15.98:0/2431555103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efd6c004440 con 0x7efd70103af0 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd701a1f60 con 0x7efd70103af0 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd65ffb640 1 -- 172.21.15.98:0/2431555103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd6c035070 con 0x7efd70103af0 2024-04-25T16:30:37.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.240+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd701a2360 con 0x7efd70103af0 2024-04-25T16:30:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.241+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd70108840 con 0x7efd70103af0 2024-04-25T16:30:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.243+0000 7efd65ffb640 1 -- 172.21.15.98:0/2431555103 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7efd6c042030 con 0x7efd70103af0 2024-04-25T16:30:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.243+0000 7efd65ffb640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efd580afaf0 0x7efd580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.244+0000 7efd67fff640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efd580afaf0 0x7efd580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.244+0000 7efd65ffb640 1 -- 172.21.15.98:0/2431555103 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7efd6c0ff570 con 0x7efd70103af0 2024-04-25T16:30:37.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.244+0000 7efd67fff640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efd580afaf0 0x7efd580b2000 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efd7019b230 tx=0x7efd60005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.247+0000 7efd65ffb640 1 -- 172.21.15.98:0/2431555103 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd6c0c78c0 con 0x7efd70103af0 2024-04-25T16:30:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:37 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3806891309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T16:30:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:37 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] up:active 2024-04-25T16:30:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:37 smithi203 ceph-mon[33115]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T16:30:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:37 smithi203 ceph-mon[33115]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-25T16:30:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:37 smithi203 ceph-mon[33115]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-04-25T16:30:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:37 smithi203 ceph-mon[33115]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-25T16:30:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:37 smithi203 ceph-mon[33115]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:37.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:37 smithi203 ceph-mon[33115]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:37.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.455+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7efd70063070 con 0x7efd580afaf0 2024-04-25T16:30:37.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.483+0000 7efd65ffb640 1 -- 172.21.15.98:0/2431555103 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7efd70063070 con 0x7efd580afaf0 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efd580afaf0 msgr2=0x7efd580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efd580afaf0 0x7efd580b2000 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efd7019b230 tx=0x7efd60005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 msgr2=0x7efd7019a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 0x7efd7019a190 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7efd6c005760 tx=0x7efd6c004040 comp rx=0 tx=0).stop 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 shutdown_connections 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efd580afaf0 0x7efd580b2000 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd70109a30 0x7efd701a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd70104d30 0x7efd7019a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 --2- 172.21.15.98:0/2431555103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efd70103af0 0x7efd7019a190 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 >> 172.21.15.98:0/2431555103 conn(0x7efd700fece0 msgr2=0x7efd70105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 shutdown_connections 2024-04-25T16:30:37.486 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.485+0000 7efd77092640 1 -- 172.21.15.98:0/2431555103 wait complete. 2024-04-25T16:30:37.495 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T16:30:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:37 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3806891309' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T16:30:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:37 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] up:active 2024-04-25T16:30:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:37 smithi173 ceph-mon[32556]: daemon mds.e assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T16:30:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:37 smithi173 ceph-mon[32556]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-04-25T16:30:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:37 smithi173 ceph-mon[32556]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:creating} 8 up:standby 2024-04-25T16:30:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:37 smithi173 ceph-mon[32556]: daemon mds.e is now active in filesystem cephfs as rank 3 2024-04-25T16:30:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:37 smithi173 ceph-mon[32556]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:37.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:37 smithi173 ceph-mon[32556]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 -- 172.21.15.98:0/1855349944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4102f60 msgr2=0x7fbfa4103370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/1855349944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4102f60 0x7fbfa4103370 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009ca0 tx=0x7fbf9402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 -- 172.21.15.98:0/1855349944 shutdown_connections 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/1855349944 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa4104460 0x7fbfa410abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/1855349944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa4103940 0x7fbfa4103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/1855349944 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4102f60 0x7fbfa4103370 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 -- 172.21.15.98:0/1855349944 >> 172.21.15.98:0/1855349944 conn(0x7fbfa40fd340 msgr2=0x7fbfa40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 -- 172.21.15.98:0/1855349944 shutdown_connections 2024-04-25T16:30:37.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 -- 172.21.15.98:0/1855349944 wait complete. 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 Processor -- start 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.608+0000 7fbfabb0e640 1 -- start start 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfabb0e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa4102f60 0x7fbfa4199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfabb0e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4103940 0x7fbfa41995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfabb0e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa4104460 0x7fbfa41a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfabb0e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410e400 con 0x7fbfa4103940 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfabb0e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410e280 con 0x7fbfa4102f60 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfabb0e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa410e580 con 0x7fbfa4104460 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4103940 0x7fbfa41995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4103940 0x7fbfa41995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37026/0 (socket says 172.21.15.98:37026) 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 -- 172.21.15.98:0/3310469630 learned_addr learned my addr 172.21.15.98:0/3310469630 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa98aa640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa4102f60 0x7fbfa4199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfaa0ab640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa4104460 0x7fbfa41a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 -- 172.21.15.98:0/3310469630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa4104460 msgr2=0x7fbfa41a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa4104460 0x7fbfa41a0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 -- 172.21.15.98:0/3310469630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa4102f60 msgr2=0x7fbfa4199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa4102f60 0x7fbfa4199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 -- 172.21.15.98:0/3310469630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa41a0ba0 con 0x7fbfa4103940 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfaa0ab640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa4104460 0x7fbfa41a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa90a9640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4103940 0x7fbfa41995a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800e9f0 tx=0x7fbf9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa2ffd640 1 -- 172.21.15.98:0/3310469630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf9800cd80 con 0x7fbfa4103940 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa2ffd640 1 -- 172.21.15.98:0/3310469630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf980108c0 con 0x7fbfa4103940 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa41a0e90 con 0x7fbfa4103940 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfa2ffd640 1 -- 172.21.15.98:0/3310469630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf980187c0 con 0x7fbfa4103940 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.609+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa41a1300 con 0x7fbfa4103940 2024-04-25T16:30:37.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.610+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa4107900 con 0x7fbfa4103940 2024-04-25T16:30:37.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.612+0000 7fbfa2ffd640 1 -- 172.21.15.98:0/3310469630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbf98010430 con 0x7fbfa4103940 2024-04-25T16:30:37.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.612+0000 7fbfa2ffd640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbf8c0afaf0 0x7fbf8c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.612+0000 7fbfa98aa640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbf8c0afaf0 0x7fbf8c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.613+0000 7fbfa2ffd640 1 -- 172.21.15.98:0/3310469630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fbf980df730 con 0x7fbfa4103940 2024-04-25T16:30:37.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.613+0000 7fbfa98aa640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbf8c0afaf0 0x7fbf8c0b2000 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbf94000f80 tx=0x7fbf940318e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.616+0000 7fbfa2ffd640 1 -- 172.21.15.98:0/3310469630 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf980a80d0 con 0x7fbfa4103940 2024-04-25T16:30:37.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.824+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fbfa4063070 con 0x7fbf8c0afaf0 2024-04-25T16:30:37.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.829+0000 7fbfa2ffd640 1 -- 172.21.15.98:0/3310469630 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fbfa4063070 con 0x7fbf8c0afaf0 2024-04-25T16:30:37.830 INFO:teuthology.orchestra.run.smithi098.stdout:/volumes/qa/sv_1/5d86ca33-da43-44d4-be72-59156325c073 2024-04-25T16:30:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.831+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbf8c0afaf0 msgr2=0x7fbf8c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.831+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbf8c0afaf0 0x7fbf8c0b2000 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbf94000f80 tx=0x7fbf940318e0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.831+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4103940 msgr2=0x7fbfa41995a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.831+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4103940 0x7fbfa41995a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fbf9800e9f0 tx=0x7fbf9800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.832+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 shutdown_connections 2024-04-25T16:30:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.832+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbf8c0afaf0 0x7fbf8c0b2000 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.832+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbfa4104460 0x7fbfa41a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.832+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbfa4103940 0x7fbfa41995a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.832+0000 7fbfabb0e640 1 --2- 172.21.15.98:0/3310469630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa4102f60 0x7fbfa4199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.832+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 >> 172.21.15.98:0/3310469630 conn(0x7fbfa40fd340 msgr2=0x7fbfa4104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.832+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 shutdown_connections 2024-04-25T16:30:37.833 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.832+0000 7fbfabb0e640 1 -- 172.21.15.98:0/3310469630 wait complete. 2024-04-25T16:30:37.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:30:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 -- 172.21.15.98:0/993913458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe4108900 msgr2=0x7f9fe410ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/993913458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe4108900 0x7f9fe410ad20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009ca0 tx=0x7f9fdc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 -- 172.21.15.98:0/993913458 shutdown_connections 2024-04-25T16:30:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/993913458 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe4108900 0x7f9fe410ad20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/993913458 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe4103c00 0x7f9fe4104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/993913458 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fe41029c0 0x7f9fe41036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.954 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 -- 172.21.15.98:0/993913458 >> 172.21.15.98:0/993913458 conn(0x7f9fe40fdbd0 msgr2=0x7f9fe4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 -- 172.21.15.98:0/993913458 shutdown_connections 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 -- 172.21.15.98:0/993913458 wait complete. 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 Processor -- start 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.952+0000 7f9fec5b7640 1 -- start start 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fec5b7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe41029c0 0x7f9fe4198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fec5b7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fe4103c00 0x7f9fe4199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fec5b7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe4108900 0x7f9fe41a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fec5b7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe410e3f0 con 0x7f9fe41029c0 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fec5b7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe410e270 con 0x7f9fe4103c00 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fec5b7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe410e570 con 0x7f9fe4108900 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe41029c0 0x7f9fe4198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe41029c0 0x7f9fe4198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37044/0 (socket says 172.21.15.98:37044) 2024-04-25T16:30:37.955 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 -- 172.21.15.98:0/1248976264 learned_addr learned my addr 172.21.15.98:0/1248976264 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fe9b52640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fe4103c00 0x7f9fe4199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9feab54640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe4108900 0x7f9fe41a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 -- 172.21.15.98:0/1248976264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe4108900 msgr2=0x7f9fe41a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe4108900 0x7f9fe41a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 -- 172.21.15.98:0/1248976264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fe4103c00 msgr2=0x7f9fe4199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fe4103c00 0x7f9fe4199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 -- 172.21.15.98:0/1248976264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe41a0b30 con 0x7f9fe41029c0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9feab54640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe4108900 0x7f9fe41a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fea353640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe41029c0 0x7f9fe4198ff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000e9f0 tx=0x7f9fe000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fdb7fe640 1 -- 172.21.15.98:0/1248976264 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe000ccc0 con 0x7f9fe41029c0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fdb7fe640 1 -- 172.21.15.98:0/1248976264 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9fe00044b0 con 0x7f9fe41029c0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe41a0e20 con 0x7f9fe41029c0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fdb7fe640 1 -- 172.21.15.98:0/1248976264 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0010430 con 0x7f9fe41029c0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.953+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe41a1280 con 0x7f9fe41029c0 2024-04-25T16:30:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.954+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fe406afa0 con 0x7f9fe41029c0 2024-04-25T16:30:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.956+0000 7f9fdb7fe640 1 -- 172.21.15.98:0/1248976264 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9fe0003f90 con 0x7f9fe41029c0 2024-04-25T16:30:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.957+0000 7f9fdb7fe640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9fd00afaf0 0x7f9fd00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:37.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.957+0000 7f9fe9b52640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9fd00afaf0 0x7f9fd00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.957+0000 7f9fdb7fe640 1 -- 172.21.15.98:0/1248976264 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f9fe00df220 con 0x7f9fe41029c0 2024-04-25T16:30:37.958 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.958+0000 7f9fe9b52640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9fd00afaf0 0x7f9fd00b2000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9fe419a090 tx=0x7f9fd4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:37.960 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:37.961+0000 7f9fdb7fe640 1 -- 172.21.15.98:0/1248976264 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9fe00a7cf0 con 0x7f9fe41029c0 2024-04-25T16:30:38.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.049+0000 7f9fdb7fe640 1 -- 172.21.15.98:0/1248976264 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f9fe00a3880 con 0x7f9fe41029c0 2024-04-25T16:30:38.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.189+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fe402dc10 con 0x7f9fe41029c0 2024-04-25T16:30:38.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.190+0000 7f9fdb7fe640 1 -- 172.21.15.98:0/1248976264 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7f9fe00da040 con 0x7f9fe41029c0 2024-04-25T16:30:38.189 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:38.190 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:30:38.051368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24484,24539,24457,24473,24494]},"id":1}]} 2024-04-25T16:30:38.190 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9fd00afaf0 msgr2=0x7f9fd00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9fd00afaf0 0x7f9fd00b2000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9fe419a090 tx=0x7f9fd4005f90 comp rx=0 tx=0).stop 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe41029c0 msgr2=0x7f9fe4198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe41029c0 0x7f9fe4198ff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9fe000e9f0 tx=0x7f9fe000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 shutdown_connections 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9fd00afaf0 0x7f9fd00b2000 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fe4108900 0x7f9fe41a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fe4103c00 0x7f9fe4199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 --2- 172.21.15.98:0/1248976264 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9fe41029c0 0x7f9fe4198ff0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 >> 172.21.15.98:0/1248976264 conn(0x7f9fe40fdbd0 msgr2=0x7f9fe41048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 shutdown_connections 2024-04-25T16:30:38.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.192+0000 7f9fec5b7640 1 -- 172.21.15.98:0/1248976264 wait complete. 2024-04-25T16:30:38.201 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] up:active 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:38 smithi098 ceph-mon[29162]: pgmap v296: 129 pgs: 129 active+clean; 582 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-04-25T16:30:38.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.310+0000 7f5e8dad8640 1 -- 172.21.15.98:0/3967964376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e88109870 msgr2=0x7f5e8810bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.310+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/3967964376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e88109870 0x7f5e8810bc90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5e80009be0 tx=0x7f5e8002a6c0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 -- 172.21.15.98:0/3967964376 shutdown_connections 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/3967964376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e88109870 0x7f5e8810bc90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/3967964376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88106f10 0x7f5e88109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/3967964376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e881045b0 0x7f5e881069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 -- 172.21.15.98:0/3967964376 >> 172.21.15.98:0/3967964376 conn(0x7f5e880fdbd0 msgr2=0x7f5e88100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 -- 172.21.15.98:0/3967964376 shutdown_connections 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 -- 172.21.15.98:0/3967964376 wait complete. 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 Processor -- start 2024-04-25T16:30:38.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.311+0000 7f5e8dad8640 1 -- start start 2024-04-25T16:30:38.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e8dad8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e881045b0 0x7f5e8819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e8dad8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e88106f10 0x7f5e8819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e8dad8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88109870 0x7f5e881a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e8dad8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810f4a0 con 0x7f5e88109870 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e8dad8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810f320 con 0x7f5e881045b0 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e8dad8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8810f620 con 0x7f5e88106f10 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88109870 0x7f5e881a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88109870 0x7f5e881a15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37060/0 (socket says 172.21.15.98:37060) 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e87fff640 1 -- 172.21.15.98:0/1018412835 learned_addr learned my addr 172.21.15.98:0/1018412835 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e877fe640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e881045b0 0x7f5e8819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e86ffd640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e88106f10 0x7f5e8819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e877fe640 1 -- 172.21.15.98:0/1018412835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e88106f10 msgr2=0x7f5e8819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e877fe640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e88106f10 0x7f5e8819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e877fe640 1 -- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88109870 msgr2=0x7f5e881a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e877fe640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88109870 0x7f5e881a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e877fe640 1 -- 172.21.15.98:0/1018412835 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e881a1c20 con 0x7f5e881045b0 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e87fff640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88109870 0x7f5e881a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e877fe640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e881045b0 0x7f5e8819a0e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e8806a8b0 tx=0x7f5e7800dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e84ff9640 1 -- 172.21.15.98:0/1018412835 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78017070 con 0x7f5e881045b0 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e84ff9640 1 -- 172.21.15.98:0/1018412835 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5e780099d0 con 0x7f5e881045b0 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e84ff9640 1 -- 172.21.15.98:0/1018412835 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e78005620 con 0x7f5e881045b0 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e881a1f10 con 0x7f5e881045b0 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.312+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e881a2320 con 0x7f5e881045b0 2024-04-25T16:30:38.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.313+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e8806afa0 con 0x7f5e881045b0 2024-04-25T16:30:38.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.315+0000 7f5e84ff9640 1 -- 172.21.15.98:0/1018412835 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5e78009b80 con 0x7f5e881045b0 2024-04-25T16:30:38.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.315+0000 7f5e84ff9640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5e700afb80 0x7f5e700b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.315+0000 7f5e86ffd640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5e700afb80 0x7f5e700b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.315+0000 7f5e84ff9640 1 -- 172.21.15.98:0/1018412835 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f5e780deeb0 con 0x7f5e881045b0 2024-04-25T16:30:38.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.316+0000 7f5e86ffd640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5e700afb80 0x7f5e700b2090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5e8819b180 tx=0x7f5e7c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.319+0000 7f5e84ff9640 1 -- 172.21.15.98:0/1018412835 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5e78020020 con 0x7f5e881045b0 2024-04-25T16:30:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] up:active 2024-04-25T16:30:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T16:30:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T16:30:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:30:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-04-25T16:30:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T16:30:38.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-25T16:30:38.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:38 smithi203 ceph-mon[33115]: pgmap v296: 129 pgs: 129 active+clean; 582 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-04-25T16:30:38.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.499+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e8802dc20 con 0x7f5e881045b0 2024-04-25T16:30:38.500 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.501+0000 7f5e84ff9640 1 -- 172.21.15.98:0/1018412835 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7f5e780a7690 con 0x7f5e881045b0 2024-04-25T16:30:38.501 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:38.501 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5e700afb80 msgr2=0x7f5e700b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5e700afb80 0x7f5e700b2090 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5e8819b180 tx=0x7f5e7c009260 comp rx=0 tx=0).stop 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e881045b0 msgr2=0x7f5e8819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e881045b0 0x7f5e8819a0e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e8806a8b0 tx=0x7f5e7800dbb0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 shutdown_connections 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5e700afb80 0x7f5e700b2090 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5e88109870 0x7f5e881a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e88106f10 0x7f5e8819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 --2- 172.21.15.98:0/1018412835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e881045b0 0x7f5e8819a0e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 >> 172.21.15.98:0/1018412835 conn(0x7f5e880fdbd0 msgr2=0x7f5e88105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 shutdown_connections 2024-04-25T16:30:38.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.503+0000 7f5e8dad8640 1 -- 172.21.15.98:0/1018412835 wait complete. 2024-04-25T16:30:38.514 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T16:30:38.514 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] up:active 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=e=up:active} 8 up:standby 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: from='client.14928 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:30:38.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:38 smithi173 ceph-mon[32556]: pgmap v296: 129 pgs: 129 active+clean; 582 KiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 4 op/s 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 -- 172.21.15.98:0/1336254125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840109a30 msgr2=0x7fc84010be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 --2- 172.21.15.98:0/1336254125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840109a30 0x7fc84010be50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc838009ca0 tx=0x7fc83802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 -- 172.21.15.98:0/1336254125 shutdown_connections 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 --2- 172.21.15.98:0/1336254125 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840109a30 0x7fc84010be50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 --2- 172.21.15.98:0/1336254125 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc840104d30 0x7fc840105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 --2- 172.21.15.98:0/1336254125 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc840103af0 0x7fc8401047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 -- 172.21.15.98:0/1336254125 >> 172.21.15.98:0/1336254125 conn(0x7fc8400fed40 msgr2=0x7fc840101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 -- 172.21.15.98:0/1336254125 shutdown_connections 2024-04-25T16:30:38.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 -- 172.21.15.98:0/1336254125 wait complete. 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 Processor -- start 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.623+0000 7fc84657d640 1 -- start start 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc84657d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc840103af0 0x7fc84019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc84657d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840104d30 0x7fc84019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc84657d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc840109a30 0x7fc8401a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc84657d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc84010f520 con 0x7fc840104d30 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc84657d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc84010f3a0 con 0x7fc840109a30 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc84657d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc84010f6a0 con 0x7fc840103af0 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840104d30 0x7fc84019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840104d30 0x7fc84019a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37076/0 (socket says 172.21.15.98:37076) 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 -- 172.21.15.98:0/308929615 learned_addr learned my addr 172.21.15.98:0/308929615 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:38.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc844b1a640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc840109a30 0x7fc8401a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83ffff640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc840103af0 0x7fc84019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 -- 172.21.15.98:0/308929615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc840103af0 msgr2=0x7fc84019a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc840103af0 0x7fc84019a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 -- 172.21.15.98:0/308929615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc840109a30 msgr2=0x7fc8401a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc840109a30 0x7fc8401a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 -- 172.21.15.98:0/308929615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8401a1c90 con 0x7fc840104d30 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83ffff640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc840103af0 0x7fc84019a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc844b1a640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc840109a30 0x7fc8401a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.624+0000 7fc83f7fe640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840104d30 0x7fc84019a690 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc83400e9f0 tx=0x7fc83400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.625+0000 7fc83d7fa640 1 -- 172.21.15.98:0/308929615 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc83400cd80 con 0x7fc840104d30 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.625+0000 7fc83d7fa640 1 -- 172.21.15.98:0/308929615 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc8340108c0 con 0x7fc840104d30 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.625+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8401a1f80 con 0x7fc840104d30 2024-04-25T16:30:38.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.625+0000 7fc83d7fa640 1 -- 172.21.15.98:0/308929615 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8340187c0 con 0x7fc840104d30 2024-04-25T16:30:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.625+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8401a2440 con 0x7fc840104d30 2024-04-25T16:30:38.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.626+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc840108840 con 0x7fc840104d30 2024-04-25T16:30:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.627+0000 7fc83d7fa640 1 -- 172.21.15.98:0/308929615 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fc834010430 con 0x7fc840104d30 2024-04-25T16:30:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.628+0000 7fc83d7fa640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8280afc80 0x7fc8280b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.628+0000 7fc83ffff640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8280afc80 0x7fc8280b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.628+0000 7fc83d7fa640 1 -- 172.21.15.98:0/308929615 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fc8340df860 con 0x7fc840104d30 2024-04-25T16:30:38.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.628+0000 7fc83ffff640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8280afc80 0x7fc8280b2190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc8300058c0 tx=0x7fc830005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.631+0000 7fc83d7fa640 1 -- 172.21.15.98:0/308929615 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8340a8040 con 0x7fc840104d30 2024-04-25T16:30:38.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.805+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc840062fe0 con 0x7fc8280afc80 2024-04-25T16:30:38.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.815+0000 7fc83d7fa640 1 -- 172.21.15.98:0/308929615 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340243 (secure 0 0 0) 0x7fc840062fe0 con 0x7fc8280afc80 2024-04-25T16:30:38.818 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:38.820 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T16:30:38.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8280afc80 msgr2=0x7fc8280b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8280afc80 0x7fc8280b2190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc8300058c0 tx=0x7fc830005850 comp rx=0 tx=0).stop 2024-04-25T16:30:38.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840104d30 msgr2=0x7fc84019a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840104d30 0x7fc84019a690 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc83400e9f0 tx=0x7fc83400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 shutdown_connections 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc8280afc80 0x7fc8280b2190 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc840109a30 0x7fc8401a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc840104d30 0x7fc84019a690 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 --2- 172.21.15.98:0/308929615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc840103af0 0x7fc84019a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 >> 172.21.15.98:0/308929615 conn(0x7fc8400fed40 msgr2=0x7fc840105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 shutdown_connections 2024-04-25T16:30:38.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.818+0000 7fc84657d640 1 -- 172.21.15.98:0/308929615 wait complete. 2024-04-25T16:30:38.830 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-04-25T16:30:37.928447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596464,"num_objects":46,"num_object_clones":0,"num_object_copies":139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":218,"num_write_kb":2184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386468,"kb_used_data":12332,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555356,"statfs":{"total":1151940427776,"available":1151544684544,"internally_reserved":0,"allocated":12627968,"data_stored":6975645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3686,"num_objects":22,"num_object_clones":0,"num_object_copies":67,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003930"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011452+0000","last_change":"2024-04-25T16:30:09.990700+0000","last_active":"2024-04-25T16:30:15.011452+0000","last_peered":"2024-04-25T16:30:15.011452+0000","last_clean":"2024-04-25T16:30:15.011452+0000","last_became_active":"2024-04-25T16:30:09.990225+0000","last_became_peered":"2024-04-25T16:30:09.990225+0000","last_unstale":"2024-04-25T16:30:15.011452+0000","last_undegraded":"2024-04-25T16:30:15.011452+0000","last_fullsized":"2024-04-25T16:30:15.011452+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:33.897277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391502+0000","last_change":"2024-04-25T16:30:09.989618+0000","last_active":"2024-04-25T16:30:15.391502+0000","last_peered":"2024-04-25T16:30:15.391502+0000","last_clean":"2024-04-25T16:30:15.391502+0000","last_became_active":"2024-04-25T16:30:09.989270+0000","last_became_peered":"2024-04-25T16:30:09.989270+0000","last_unstale":"2024-04-25T16:30:15.391502+0000","last_undegraded":"2024-04-25T16:30:15.391502+0000","last_fullsized":"2024-04-25T16:30:15.391502+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:03.952233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391028+0000","last_change":"2024-04-25T16:30:09.992442+0000","last_active":"2024-04-25T16:30:15.391028+0000","last_peered":"2024-04-25T16:30:15.391028+0000","last_clean":"2024-04-25T16:30:15.391028+0000","last_became_active":"2024-04-25T16:30:09.992115+0000","last_became_peered":"2024-04-25T16:30:09.992115+0000","last_unstale":"2024-04-25T16:30:15.391028+0000","last_undegraded":"2024-04-25T16:30:15.391028+0000","last_fullsized":"2024-04-25T16:30:15.391028+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:00.668427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391344+0000","last_change":"2024-04-25T16:30:10.001366+0000","last_active":"2024-04-25T16:30:15.391344+0000","last_peered":"2024-04-25T16:30:15.391344+0000","last_clean":"2024-04-25T16:30:15.391344+0000","last_became_active":"2024-04-25T16:30:10.001000+0000","last_became_peered":"2024-04-25T16:30:10.001000+0000","last_unstale":"2024-04-25T16:30:15.391344+0000","last_undegraded":"2024-04-25T16:30:15.391344+0000","last_fullsized":"2024-04-25T16:30:15.391344+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:32.269811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391245+0000","last_change":"2024-04-25T16:30:09.991372+0000","last_active":"2024-04-25T16:30:15.391245+0000","last_peered":"2024-04-25T16:30:15.391245+0000","last_clean":"2024-04-25T16:30:15.391245+0000","last_became_active":"2024-04-25T16:30:09.991053+0000","last_became_peered":"2024-04-25T16:30:09.991053+0000","last_unstale":"2024-04-25T16:30:15.391245+0000","last_undegraded":"2024-04-25T16:30:15.391245+0000","last_fullsized":"2024-04-25T16:30:15.391245+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:28.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.015408+0000","last_change":"2024-04-25T16:30:09.997563+0000","last_active":"2024-04-25T16:30:15.015408+0000","last_peered":"2024-04-25T16:30:15.015408+0000","last_clean":"2024-04-25T16:30:15.015408+0000","last_became_active":"2024-04-25T16:30:09.997158+0000","last_became_peered":"2024-04-25T16:30:09.997158+0000","last_unstale":"2024-04-25T16:30:15.015408+0000","last_undegraded":"2024-04-25T16:30:15.015408+0000","last_fullsized":"2024-04-25T16:30:15.015408+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:21.441957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173528+0000","last_change":"2024-04-25T16:30:09.989859+0000","last_active":"2024-04-25T16:30:15.173528+0000","last_peered":"2024-04-25T16:30:15.173528+0000","last_clean":"2024-04-25T16:30:15.173528+0000","last_became_active":"2024-04-25T16:30:09.989291+0000","last_became_peered":"2024-04-25T16:30:09.989291+0000","last_unstale":"2024-04-25T16:30:15.173528+0000","last_undegraded":"2024-04-25T16:30:15.173528+0000","last_fullsized":"2024-04-25T16:30:15.173528+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:07.833941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390141+0000","last_change":"2024-04-25T16:30:09.994820+0000","last_active":"2024-04-25T16:30:15.390141+0000","last_peered":"2024-04-25T16:30:15.390141+0000","last_clean":"2024-04-25T16:30:15.390141+0000","last_became_active":"2024-04-25T16:30:09.994494+0000","last_became_peered":"2024-04-25T16:30:09.994494+0000","last_unstale":"2024-04-25T16:30:15.390141+0000","last_undegraded":"2024-04-25T16:30:15.390141+0000","last_fullsized":"2024-04-25T16:30:15.390141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:39.373266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390710+0000","last_change":"2024-04-25T16:30:09.994870+0000","last_active":"2024-04-25T16:30:15.390710+0000","last_peered":"2024-04-25T16:30:15.390710+0000","last_clean":"2024-04-25T16:30:15.390710+0000","last_became_active":"2024-04-25T16:30:09.994521+0000","last_became_peered":"2024-04-25T16:30:09.994521+0000","last_unstale":"2024-04-25T16:30:15.390710+0000","last_undegraded":"2024-04-25T16:30:15.390710+0000","last_fullsized":"2024-04-25T16:30:15.390710+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:18.093904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.015924+0000","last_change":"2024-04-25T16:30:09.999199+0000","last_active":"2024-04-25T16:30:15.015924+0000","last_peered":"2024-04-25T16:30:15.015924+0000","last_clean":"2024-04-25T16:30:15.015924+0000","last_became_active":"2024-04-25T16:30:09.998822+0000","last_became_peered":"2024-04-25T16:30:09.998822+0000","last_unstale":"2024-04-25T16:30:15.015924+0000","last_undegraded":"2024-04-25T16:30:15.015924+0000","last_fullsized":"2024-04-25T16:30:15.015924+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:28.955045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174178+0000","last_change":"2024-04-25T16:30:09.999519+0000","last_active":"2024-04-25T16:30:15.174178+0000","last_peered":"2024-04-25T16:30:15.174178+0000","last_clean":"2024-04-25T16:30:15.174178+0000","last_became_active":"2024-04-25T16:30:09.999158+0000","last_became_peered":"2024-04-25T16:30:09.999158+0000","last_unstale":"2024-04-25T16:30:15.174178+0000","last_undegraded":"2024-04-25T16:30:15.174178+0000","last_fullsized":"2024-04-25T16:30:15.174178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:38:06.686541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391442+0000","last_change":"2024-04-25T16:30:10.005455+0000","last_active":"2024-04-25T16:30:15.391442+0000","last_peered":"2024-04-25T16:30:15.391442+0000","last_clean":"2024-04-25T16:30:15.391442+0000","last_became_active":"2024-04-25T16:30:10.004891+0000","last_became_peered":"2024-04-25T16:30:10.004891+0000","last_unstale":"2024-04-25T16:30:15.391442+0000","last_undegraded":"2024-04-25T16:30:15.391442+0000","last_fullsized":"2024-04-25T16:30:15.391442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.598321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010625+0000","last_change":"2024-04-25T16:30:09.992870+0000","last_active":"2024-04-25T16:30:15.010625+0000","last_peered":"2024-04-25T16:30:15.010625+0000","last_clean":"2024-04-25T16:30:15.010625+0000","last_became_active":"2024-04-25T16:30:09.992399+0000","last_became_peered":"2024-04-25T16:30:09.992399+0000","last_unstale":"2024-04-25T16:30:15.010625+0000","last_undegraded":"2024-04-25T16:30:15.010625+0000","last_fullsized":"2024-04-25T16:30:15.010625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.127287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390761+0000","last_change":"2024-04-25T16:30:09.988707+0000","last_active":"2024-04-25T16:30:15.390761+0000","last_peered":"2024-04-25T16:30:15.390761+0000","last_clean":"2024-04-25T16:30:15.390761+0000","last_became_active":"2024-04-25T16:30:09.988386+0000","last_became_peered":"2024-04-25T16:30:09.988386+0000","last_unstale":"2024-04-25T16:30:15.390761+0000","last_undegraded":"2024-04-25T16:30:15.390761+0000","last_fullsized":"2024-04-25T16:30:15.390761+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:03:15.973591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391413+0000","last_change":"2024-04-25T16:30:10.002780+0000","last_active":"2024-04-25T16:30:15.391413+0000","last_peered":"2024-04-25T16:30:15.391413+0000","last_clean":"2024-04-25T16:30:15.391413+0000","last_became_active":"2024-04-25T16:30:10.002055+0000","last_became_peered":"2024-04-25T16:30:10.002055+0000","last_unstale":"2024-04-25T16:30:15.391413+0000","last_undegraded":"2024-04-25T16:30:15.391413+0000","last_fullsized":"2024-04-25T16:30:15.391413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:23.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390092+0000","last_change":"2024-04-25T16:30:09.998270+0000","last_active":"2024-04-25T16:30:15.390092+0000","last_peered":"2024-04-25T16:30:15.390092+0000","last_clean":"2024-04-25T16:30:15.390092+0000","last_became_active":"2024-04-25T16:30:09.997903+0000","last_became_peered":"2024-04-25T16:30:09.997903+0000","last_unstale":"2024-04-25T16:30:15.390092+0000","last_undegraded":"2024-04-25T16:30:15.390092+0000","last_fullsized":"2024-04-25T16:30:15.390092+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:14.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011302+0000","last_change":"2024-04-25T16:30:09.988894+0000","last_active":"2024-04-25T16:30:15.011302+0000","last_peered":"2024-04-25T16:30:15.011302+0000","last_clean":"2024-04-25T16:30:15.011302+0000","last_became_active":"2024-04-25T16:30:09.988508+0000","last_became_peered":"2024-04-25T16:30:09.988508+0000","last_unstale":"2024-04-25T16:30:15.011302+0000","last_undegraded":"2024-04-25T16:30:15.011302+0000","last_fullsized":"2024-04-25T16:30:15.011302+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:48.226193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391238+0000","last_change":"2024-04-25T16:30:09.989910+0000","last_active":"2024-04-25T16:30:15.391238+0000","last_peered":"2024-04-25T16:30:15.391238+0000","last_clean":"2024-04-25T16:30:15.391238+0000","last_became_active":"2024-04-25T16:30:09.989560+0000","last_became_peered":"2024-04-25T16:30:09.989560+0000","last_unstale":"2024-04-25T16:30:15.391238+0000","last_undegraded":"2024-04-25T16:30:15.391238+0000","last_fullsized":"2024-04-25T16:30:15.391238+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:12.834034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392709+0000","last_change":"2024-04-25T16:30:09.987930+0000","last_active":"2024-04-25T16:30:15.392709+0000","last_peered":"2024-04-25T16:30:15.392709+0000","last_clean":"2024-04-25T16:30:15.392709+0000","last_became_active":"2024-04-25T16:30:09.987595+0000","last_became_peered":"2024-04-25T16:30:09.987595+0000","last_unstale":"2024-04-25T16:30:15.392709+0000","last_undegraded":"2024-04-25T16:30:15.392709+0000","last_fullsized":"2024-04-25T16:30:15.392709+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:04.778172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391702+0000","last_change":"2024-04-25T16:30:14.006360+0000","last_active":"2024-04-25T16:30:15.391702+0000","last_peered":"2024-04-25T16:30:15.391702+0000","last_clean":"2024-04-25T16:30:15.391702+0000","last_became_active":"2024-04-25T16:30:14.005894+0000","last_became_peered":"2024-04-25T16:30:14.005894+0000","last_unstale":"2024-04-25T16:30:15.391702+0000","last_undegraded":"2024-04-25T16:30:15.391702+0000","last_fullsized":"2024-04-25T16:30:15.391702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:33.150381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"76'4","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391975+0000","last_change":"2024-04-25T16:30:08.992779+0000","last_active":"2024-04-25T16:30:15.391975+0000","last_peered":"2024-04-25T16:30:15.391975+0000","last_clean":"2024-04-25T16:30:15.391975+0000","last_became_active":"2024-04-25T16:30:08.992458+0000","last_became_peered":"2024-04-25T16:30:08.992458+0000","last_unstale":"2024-04-25T16:30:15.391975+0000","last_undegraded":"2024-04-25T16:30:15.391975+0000","last_fullsized":"2024-04-25T16:30:15.391975+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:21.430855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390554+0000","last_change":"2024-04-25T16:30:09.995353+0000","last_active":"2024-04-25T16:30:15.390554+0000","last_peered":"2024-04-25T16:30:15.390554+0000","last_clean":"2024-04-25T16:30:15.390554+0000","last_became_active":"2024-04-25T16:30:09.995011+0000","last_became_peered":"2024-04-25T16:30:09.995011+0000","last_unstale":"2024-04-25T16:30:15.390554+0000","last_undegraded":"2024-04-25T16:30:15.390554+0000","last_fullsized":"2024-04-25T16:30:15.390554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:33.367268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390178+0000","last_change":"2024-04-25T16:30:14.005848+0000","last_active":"2024-04-25T16:30:15.390178+0000","last_peered":"2024-04-25T16:30:15.390178+0000","last_clean":"2024-04-25T16:30:15.390178+0000","last_became_active":"2024-04-25T16:30:14.005207+0000","last_became_peered":"2024-04-25T16:30:14.005207+0000","last_unstale":"2024-04-25T16:30:15.390178+0000","last_undegraded":"2024-04-25T16:30:15.390178+0000","last_fullsized":"2024-04-25T16:30:15.390178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:01.534652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390770+0000","last_change":"2024-04-25T16:30:08.997663+0000","last_active":"2024-04-25T16:30:15.390770+0000","last_peered":"2024-04-25T16:30:15.390770+0000","last_clean":"2024-04-25T16:30:15.390770+0000","last_became_active":"2024-04-25T16:30:08.997107+0000","last_became_peered":"2024-04-25T16:30:08.997107+0000","last_unstale":"2024-04-25T16:30:15.390770+0000","last_undegraded":"2024-04-25T16:30:15.390770+0000","last_fullsized":"2024-04-25T16:30:15.390770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:25.856556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391752+0000","last_change":"2024-04-25T16:30:09.987947+0000","last_active":"2024-04-25T16:30:15.391752+0000","last_peered":"2024-04-25T16:30:15.391752+0000","last_clean":"2024-04-25T16:30:15.391752+0000","last_became_active":"2024-04-25T16:30:09.987575+0000","last_became_peered":"2024-04-25T16:30:09.987575+0000","last_unstale":"2024-04-25T16:30:15.391752+0000","last_undegraded":"2024-04-25T16:30:15.391752+0000","last_fullsized":"2024-04-25T16:30:15.391752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:28.443249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391491+0000","last_change":"2024-04-25T16:30:14.006711+0000","last_active":"2024-04-25T16:30:15.391491+0000","last_peered":"2024-04-25T16:30:15.391491+0000","last_clean":"2024-04-25T16:30:15.391491+0000","last_became_active":"2024-04-25T16:30:14.005500+0000","last_became_peered":"2024-04-25T16:30:14.005500+0000","last_unstale":"2024-04-25T16:30:15.391491+0000","last_undegraded":"2024-04-25T16:30:15.391491+0000","last_fullsized":"2024-04-25T16:30:15.391491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:58:26.288410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392394+0000","last_change":"2024-04-25T16:30:08.993302+0000","last_active":"2024-04-25T16:30:15.392394+0000","last_peered":"2024-04-25T16:30:15.392394+0000","last_clean":"2024-04-25T16:30:15.392394+0000","last_became_active":"2024-04-25T16:30:08.992918+0000","last_became_peered":"2024-04-25T16:30:08.992918+0000","last_unstale":"2024-04-25T16:30:15.392394+0000","last_undegraded":"2024-04-25T16:30:15.392394+0000","last_fullsized":"2024-04-25T16:30:15.392394+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:18.843803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391027+0000","last_change":"2024-04-25T16:30:09.988216+0000","last_active":"2024-04-25T16:30:15.391027+0000","last_peered":"2024-04-25T16:30:15.391027+0000","last_clean":"2024-04-25T16:30:15.391027+0000","last_became_active":"2024-04-25T16:30:09.987752+0000","last_became_peered":"2024-04-25T16:30:09.987752+0000","last_unstale":"2024-04-25T16:30:15.391027+0000","last_undegraded":"2024-04-25T16:30:15.391027+0000","last_fullsized":"2024-04-25T16:30:15.391027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:57:35.583000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390498+0000","last_change":"2024-04-25T16:30:14.001329+0000","last_active":"2024-04-25T16:30:15.390498+0000","last_peered":"2024-04-25T16:30:15.390498+0000","last_clean":"2024-04-25T16:30:15.390498+0000","last_became_active":"2024-04-25T16:30:14.000878+0000","last_became_peered":"2024-04-25T16:30:14.000878+0000","last_unstale":"2024-04-25T16:30:15.390498+0000","last_undegraded":"2024-04-25T16:30:15.390498+0000","last_fullsized":"2024-04-25T16:30:15.390498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:07.946611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391652+0000","last_change":"2024-04-25T16:30:08.989065+0000","last_active":"2024-04-25T16:30:15.391652+0000","last_peered":"2024-04-25T16:30:15.391652+0000","last_clean":"2024-04-25T16:30:15.391652+0000","last_became_active":"2024-04-25T16:30:08.988709+0000","last_became_peered":"2024-04-25T16:30:08.988709+0000","last_unstale":"2024-04-25T16:30:15.391652+0000","last_undegraded":"2024-04-25T16:30:15.391652+0000","last_fullsized":"2024-04-25T16:30:15.391652+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:34.306581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392480+0000","last_change":"2024-04-25T16:30:09.999641+0000","last_active":"2024-04-25T16:30:15.392480+0000","last_peered":"2024-04-25T16:30:15.392480+0000","last_clean":"2024-04-25T16:30:15.392480+0000","last_became_active":"2024-04-25T16:30:09.999319+0000","last_became_peered":"2024-04-25T16:30:09.999319+0000","last_unstale":"2024-04-25T16:30:15.392480+0000","last_undegraded":"2024-04-25T16:30:15.392480+0000","last_fullsized":"2024-04-25T16:30:15.392480+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:14.043201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392554+0000","last_change":"2024-04-25T16:30:13.999609+0000","last_active":"2024-04-25T16:30:15.392554+0000","last_peered":"2024-04-25T16:30:15.392554+0000","last_clean":"2024-04-25T16:30:15.392554+0000","last_became_active":"2024-04-25T16:30:13.999172+0000","last_became_peered":"2024-04-25T16:30:13.999172+0000","last_unstale":"2024-04-25T16:30:15.392554+0000","last_undegraded":"2024-04-25T16:30:15.392554+0000","last_fullsized":"2024-04-25T16:30:15.392554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:49.878153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"76'1","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391832+0000","last_change":"2024-04-25T16:30:08.988919+0000","last_active":"2024-04-25T16:30:15.391832+0000","last_peered":"2024-04-25T16:30:15.391832+0000","last_clean":"2024-04-25T16:30:15.391832+0000","last_became_active":"2024-04-25T16:30:08.988218+0000","last_became_peered":"2024-04-25T16:30:08.988218+0000","last_unstale":"2024-04-25T16:30:15.391832+0000","last_undegraded":"2024-04-25T16:30:15.391832+0000","last_fullsized":"2024-04-25T16:30:15.391832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:57:57.046055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391080+0000","last_change":"2024-04-25T16:30:09.992498+0000","last_active":"2024-04-25T16:30:15.391080+0000","last_peered":"2024-04-25T16:30:15.391080+0000","last_clean":"2024-04-25T16:30:15.391080+0000","last_became_active":"2024-04-25T16:30:09.992190+0000","last_became_peered":"2024-04-25T16:30:09.992190+0000","last_unstale":"2024-04-25T16:30:15.391080+0000","last_undegraded":"2024-04-25T16:30:15.391080+0000","last_fullsized":"2024-04-25T16:30:15.391080+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:34.407963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391491+0000","last_change":"2024-04-25T16:30:14.007371+0000","last_active":"2024-04-25T16:30:15.391491+0000","last_peered":"2024-04-25T16:30:15.391491+0000","last_clean":"2024-04-25T16:30:15.391491+0000","last_became_active":"2024-04-25T16:30:14.006811+0000","last_became_peered":"2024-04-25T16:30:14.006811+0000","last_unstale":"2024-04-25T16:30:15.391491+0000","last_undegraded":"2024-04-25T16:30:15.391491+0000","last_fullsized":"2024-04-25T16:30:15.391491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:01.473142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:36.035768+0000","last_change":"2024-04-25T16:30:08.984080+0000","last_active":"2024-04-25T16:30:36.035768+0000","last_peered":"2024-04-25T16:30:36.035768+0000","last_clean":"2024-04-25T16:30:36.035768+0000","last_became_active":"2024-04-25T16:30:08.983702+0000","last_became_peered":"2024-04-25T16:30:08.983702+0000","last_unstale":"2024-04-25T16:30:36.035768+0000","last_undegraded":"2024-04-25T16:30:36.035768+0000","last_fullsized":"2024-04-25T16:30:36.035768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:22.311986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392067+0000","last_change":"2024-04-25T16:30:09.997913+0000","last_active":"2024-04-25T16:30:15.392067+0000","last_peered":"2024-04-25T16:30:15.392067+0000","last_clean":"2024-04-25T16:30:15.392067+0000","last_became_active":"2024-04-25T16:30:09.997567+0000","last_became_peered":"2024-04-25T16:30:09.997567+0000","last_unstale":"2024-04-25T16:30:15.392067+0000","last_undegraded":"2024-04-25T16:30:15.392067+0000","last_fullsized":"2024-04-25T16:30:15.392067+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:20.829798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011394+0000","last_change":"2024-04-25T16:30:14.003218+0000","last_active":"2024-04-25T16:30:15.011394+0000","last_peered":"2024-04-25T16:30:15.011394+0000","last_clean":"2024-04-25T16:30:15.011394+0000","last_became_active":"2024-04-25T16:30:14.002782+0000","last_became_peered":"2024-04-25T16:30:14.002782+0000","last_unstale":"2024-04-25T16:30:15.011394+0000","last_undegraded":"2024-04-25T16:30:15.011394+0000","last_fullsized":"2024-04-25T16:30:15.011394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010397+0000","last_change":"2024-04-25T16:30:08.993447+0000","last_active":"2024-04-25T16:30:15.010397+0000","last_peered":"2024-04-25T16:30:15.010397+0000","last_clean":"2024-04-25T16:30:15.010397+0000","last_became_active":"2024-04-25T16:30:08.993108+0000","last_became_peered":"2024-04-25T16:30:08.993108+0000","last_unstale":"2024-04-25T16:30:15.010397+0000","last_undegraded":"2024-04-25T16:30:15.010397+0000","last_fullsized":"2024-04-25T16:30:15.010397+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:00.511392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391022+0000","last_change":"2024-04-25T16:30:09.989863+0000","last_active":"2024-04-25T16:30:15.391022+0000","last_peered":"2024-04-25T16:30:15.391022+0000","last_clean":"2024-04-25T16:30:15.391022+0000","last_became_active":"2024-04-25T16:30:09.989520+0000","last_became_peered":"2024-04-25T16:30:09.989520+0000","last_unstale":"2024-04-25T16:30:15.391022+0000","last_undegraded":"2024-04-25T16:30:15.391022+0000","last_fullsized":"2024-04-25T16:30:15.391022+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:54.178298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391280+0000","last_change":"2024-04-25T16:30:13.999904+0000","last_active":"2024-04-25T16:30:15.391280+0000","last_peered":"2024-04-25T16:30:15.391280+0000","last_clean":"2024-04-25T16:30:15.391280+0000","last_became_active":"2024-04-25T16:30:13.999468+0000","last_became_peered":"2024-04-25T16:30:13.999468+0000","last_unstale":"2024-04-25T16:30:15.391280+0000","last_undegraded":"2024-04-25T16:30:15.391280+0000","last_fullsized":"2024-04-25T16:30:15.391280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:09.653365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390359+0000","last_change":"2024-04-25T16:30:08.987550+0000","last_active":"2024-04-25T16:30:15.390359+0000","last_peered":"2024-04-25T16:30:15.390359+0000","last_clean":"2024-04-25T16:30:15.390359+0000","last_became_active":"2024-04-25T16:30:08.987231+0000","last_became_peered":"2024-04-25T16:30:08.987231+0000","last_unstale":"2024-04-25T16:30:15.390359+0000","last_undegraded":"2024-04-25T16:30:15.390359+0000","last_fullsized":"2024-04-25T16:30:15.390359+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:01.110960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390870+0000","last_change":"2024-04-25T16:30:09.991985+0000","last_active":"2024-04-25T16:30:15.390870+0000","last_peered":"2024-04-25T16:30:15.390870+0000","last_clean":"2024-04-25T16:30:15.390870+0000","last_became_active":"2024-04-25T16:30:09.991645+0000","last_became_peered":"2024-04-25T16:30:09.991645+0000","last_unstale":"2024-04-25T16:30:15.390870+0000","last_undegraded":"2024-04-25T16:30:15.390870+0000","last_fullsized":"2024-04-25T16:30:15.390870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:30.419416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390409+0000","last_change":"2024-04-25T16:30:13.999931+0000","last_active":"2024-04-25T16:30:15.390409+0000","last_peered":"2024-04-25T16:30:15.390409+0000","last_clean":"2024-04-25T16:30:15.390409+0000","last_became_active":"2024-04-25T16:30:13.997093+0000","last_became_peered":"2024-04-25T16:30:13.997093+0000","last_unstale":"2024-04-25T16:30:15.390409+0000","last_undegraded":"2024-04-25T16:30:15.390409+0000","last_fullsized":"2024-04-25T16:30:15.390409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:31.973370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:34.016840+0000","last_change":"2024-04-25T16:30:08.985721+0000","last_active":"2024-04-25T16:30:34.016840+0000","last_peered":"2024-04-25T16:30:34.016840+0000","last_clean":"2024-04-25T16:30:34.016840+0000","last_became_active":"2024-04-25T16:30:08.985097+0000","last_became_peered":"2024-04-25T16:30:08.985097+0000","last_unstale":"2024-04-25T16:30:34.016840+0000","last_undegraded":"2024-04-25T16:30:34.016840+0000","last_fullsized":"2024-04-25T16:30:34.016840+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:00.713027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391185+0000","last_change":"2024-04-25T16:30:09.992659+0000","last_active":"2024-04-25T16:30:15.391185+0000","last_peered":"2024-04-25T16:30:15.391185+0000","last_clean":"2024-04-25T16:30:15.391185+0000","last_became_active":"2024-04-25T16:30:09.985936+0000","last_became_peered":"2024-04-25T16:30:09.985936+0000","last_unstale":"2024-04-25T16:30:15.391185+0000","last_undegraded":"2024-04-25T16:30:15.391185+0000","last_fullsized":"2024-04-25T16:30:15.391185+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:10.808082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390316+0000","last_change":"2024-04-25T16:30:14.009352+0000","last_active":"2024-04-25T16:30:15.390316+0000","last_peered":"2024-04-25T16:30:15.390316+0000","last_clean":"2024-04-25T16:30:15.390316+0000","last_became_active":"2024-04-25T16:30:14.005250+0000","last_became_peered":"2024-04-25T16:30:14.005250+0000","last_unstale":"2024-04-25T16:30:15.390316+0000","last_undegraded":"2024-04-25T16:30:15.390316+0000","last_fullsized":"2024-04-25T16:30:15.390316+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:23.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174092+0000","last_change":"2024-04-25T16:30:08.990547+0000","last_active":"2024-04-25T16:30:15.174092+0000","last_peered":"2024-04-25T16:30:15.174092+0000","last_clean":"2024-04-25T16:30:15.174092+0000","last_became_active":"2024-04-25T16:30:08.989927+0000","last_became_peered":"2024-04-25T16:30:08.989927+0000","last_unstale":"2024-04-25T16:30:15.174092+0000","last_undegraded":"2024-04-25T16:30:15.174092+0000","last_fullsized":"2024-04-25T16:30:15.174092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:05.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391842+0000","last_change":"2024-04-25T16:30:10.002459+0000","last_active":"2024-04-25T16:30:15.391842+0000","last_peered":"2024-04-25T16:30:15.391842+0000","last_clean":"2024-04-25T16:30:15.391842+0000","last_became_active":"2024-04-25T16:30:10.002005+0000","last_became_peered":"2024-04-25T16:30:10.002005+0000","last_unstale":"2024-04-25T16:30:15.391842+0000","last_undegraded":"2024-04-25T16:30:15.391842+0000","last_fullsized":"2024-04-25T16:30:15.391842+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:01.220371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391637+0000","last_change":"2024-04-25T16:30:14.005800+0000","last_active":"2024-04-25T16:30:15.391637+0000","last_peered":"2024-04-25T16:30:15.391637+0000","last_clean":"2024-04-25T16:30:15.391637+0000","last_became_active":"2024-04-25T16:30:14.005153+0000","last_became_peered":"2024-04-25T16:30:14.005153+0000","last_unstale":"2024-04-25T16:30:15.391637+0000","last_undegraded":"2024-04-25T16:30:15.391637+0000","last_fullsized":"2024-04-25T16:30:15.391637+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:40.015615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"76'3","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392325+0000","last_change":"2024-04-25T16:30:08.985264+0000","last_active":"2024-04-25T16:30:15.392325+0000","last_peered":"2024-04-25T16:30:15.392325+0000","last_clean":"2024-04-25T16:30:15.392325+0000","last_became_active":"2024-04-25T16:30:08.984901+0000","last_became_peered":"2024-04-25T16:30:08.984901+0000","last_unstale":"2024-04-25T16:30:15.392325+0000","last_undegraded":"2024-04-25T16:30:15.392325+0000","last_fullsized":"2024-04-25T16:30:15.392325+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:15.271981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390914+0000","last_change":"2024-04-25T16:30:09.997002+0000","last_active":"2024-04-25T16:30:15.390914+0000","last_peered":"2024-04-25T16:30:15.390914+0000","last_clean":"2024-04-25T16:30:15.390914+0000","last_became_active":"2024-04-25T16:30:09.996636+0000","last_became_peered":"2024-04-25T16:30:09.996636+0000","last_unstale":"2024-04-25T16:30:15.390914+0000","last_undegraded":"2024-04-25T16:30:15.390914+0000","last_fullsized":"2024-04-25T16:30:15.390914+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:32.522050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390634+0000","last_change":"2024-04-25T16:30:14.006403+0000","last_active":"2024-04-25T16:30:15.390634+0000","last_peered":"2024-04-25T16:30:15.390634+0000","last_clean":"2024-04-25T16:30:15.390634+0000","last_became_active":"2024-04-25T16:30:14.004816+0000","last_became_peered":"2024-04-25T16:30:14.004816+0000","last_unstale":"2024-04-25T16:30:15.390634+0000","last_undegraded":"2024-04-25T16:30:15.390634+0000","last_fullsized":"2024-04-25T16:30:15.390634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:51:10.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"81'12","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:34.016640+0000","last_change":"2024-04-25T16:30:08.986119+0000","last_active":"2024-04-25T16:30:34.016640+0000","last_peered":"2024-04-25T16:30:34.016640+0000","last_clean":"2024-04-25T16:30:34.016640+0000","last_became_active":"2024-04-25T16:30:08.985806+0000","last_became_peered":"2024-04-25T16:30:08.985806+0000","last_unstale":"2024-04-25T16:30:34.016640+0000","last_undegraded":"2024-04-25T16:30:34.016640+0000","last_fullsized":"2024-04-25T16:30:34.016640+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:17.125886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392139+0000","last_change":"2024-04-25T16:30:09.996891+0000","last_active":"2024-04-25T16:30:15.392139+0000","last_peered":"2024-04-25T16:30:15.392139+0000","last_clean":"2024-04-25T16:30:15.392139+0000","last_became_active":"2024-04-25T16:30:09.995851+0000","last_became_peered":"2024-04-25T16:30:09.995851+0000","last_unstale":"2024-04-25T16:30:15.392139+0000","last_undegraded":"2024-04-25T16:30:15.392139+0000","last_fullsized":"2024-04-25T16:30:15.392139+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:42.602854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392769+0000","last_change":"2024-04-25T16:30:13.998277+0000","last_active":"2024-04-25T16:30:15.392769+0000","last_peered":"2024-04-25T16:30:15.392769+0000","last_clean":"2024-04-25T16:30:15.392769+0000","last_became_active":"2024-04-25T16:30:13.997313+0000","last_became_peered":"2024-04-25T16:30:13.997313+0000","last_unstale":"2024-04-25T16:30:15.392769+0000","last_undegraded":"2024-04-25T16:30:15.392769+0000","last_fullsized":"2024-04-25T16:30:15.392769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:56.379806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390803+0000","last_change":"2024-04-25T16:30:08.986377+0000","last_active":"2024-04-25T16:30:15.390803+0000","last_peered":"2024-04-25T16:30:15.390803+0000","last_clean":"2024-04-25T16:30:15.390803+0000","last_became_active":"2024-04-25T16:30:08.985892+0000","last_became_peered":"2024-04-25T16:30:08.985892+0000","last_unstale":"2024-04-25T16:30:15.390803+0000","last_undegraded":"2024-04-25T16:30:15.390803+0000","last_fullsized":"2024-04-25T16:30:15.390803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:53.651789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010479+0000","last_change":"2024-04-25T16:30:09.996724+0000","last_active":"2024-04-25T16:30:15.010479+0000","last_peered":"2024-04-25T16:30:15.010479+0000","last_clean":"2024-04-25T16:30:15.010479+0000","last_became_active":"2024-04-25T16:30:09.996413+0000","last_became_peered":"2024-04-25T16:30:09.996413+0000","last_unstale":"2024-04-25T16:30:15.010479+0000","last_undegraded":"2024-04-25T16:30:15.010479+0000","last_fullsized":"2024-04-25T16:30:15.010479+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:38.403476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011333+0000","last_change":"2024-04-25T16:30:14.006064+0000","last_active":"2024-04-25T16:30:15.011333+0000","last_peered":"2024-04-25T16:30:15.011333+0000","last_clean":"2024-04-25T16:30:15.011333+0000","last_became_active":"2024-04-25T16:30:14.005545+0000","last_became_peered":"2024-04-25T16:30:14.005545+0000","last_unstale":"2024-04-25T16:30:15.011333+0000","last_undegraded":"2024-04-25T16:30:15.011333+0000","last_fullsized":"2024-04-25T16:30:15.011333+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:36.611008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391384+0000","last_change":"2024-04-25T16:30:08.988413+0000","last_active":"2024-04-25T16:30:15.391384+0000","last_peered":"2024-04-25T16:30:15.391384+0000","last_clean":"2024-04-25T16:30:15.391384+0000","last_became_active":"2024-04-25T16:30:08.988075+0000","last_became_peered":"2024-04-25T16:30:08.988075+0000","last_unstale":"2024-04-25T16:30:15.391384+0000","last_undegraded":"2024-04-25T16:30:15.391384+0000","last_fullsized":"2024-04-25T16:30:15.391384+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:16.635876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173926+0000","last_change":"2024-04-25T16:30:09.997697+0000","last_active":"2024-04-25T16:30:15.173926+0000","last_peered":"2024-04-25T16:30:15.173926+0000","last_clean":"2024-04-25T16:30:15.173926+0000","last_became_active":"2024-04-25T16:30:09.997086+0000","last_became_peered":"2024-04-25T16:30:09.997086+0000","last_unstale":"2024-04-25T16:30:15.173926+0000","last_undegraded":"2024-04-25T16:30:15.173926+0000","last_fullsized":"2024-04-25T16:30:15.173926+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:35.345902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391124+0000","last_change":"2024-04-25T16:30:14.010486+0000","last_active":"2024-04-25T16:30:15.391124+0000","last_peered":"2024-04-25T16:30:15.391124+0000","last_clean":"2024-04-25T16:30:15.391124+0000","last_became_active":"2024-04-25T16:30:14.009273+0000","last_became_peered":"2024-04-25T16:30:14.009273+0000","last_unstale":"2024-04-25T16:30:15.391124+0000","last_undegraded":"2024-04-25T16:30:15.391124+0000","last_fullsized":"2024-04-25T16:30:15.391124+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:11.863575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:36.034025+0000","last_change":"2024-04-25T16:30:08.986661+0000","last_active":"2024-04-25T16:30:36.034025+0000","last_peered":"2024-04-25T16:30:36.034025+0000","last_clean":"2024-04-25T16:30:36.034025+0000","last_became_active":"2024-04-25T16:30:08.986306+0000","last_became_peered":"2024-04-25T16:30:08.986306+0000","last_unstale":"2024-04-25T16:30:36.034025+0000","last_undegraded":"2024-04-25T16:30:36.034025+0000","last_fullsized":"2024-04-25T16:30:36.034025+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:44.363693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173602+0000","last_change":"2024-04-25T16:30:09.993443+0000","last_active":"2024-04-25T16:30:15.173602+0000","last_peered":"2024-04-25T16:30:15.173602+0000","last_clean":"2024-04-25T16:30:15.173602+0000","last_became_active":"2024-04-25T16:30:09.993031+0000","last_became_peered":"2024-04-25T16:30:09.993031+0000","last_unstale":"2024-04-25T16:30:15.173602+0000","last_undegraded":"2024-04-25T16:30:15.173602+0000","last_fullsized":"2024-04-25T16:30:15.173602+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:22.908713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010976+0000","last_change":"2024-04-25T16:30:14.003183+0000","last_active":"2024-04-25T16:30:15.010976+0000","last_peered":"2024-04-25T16:30:15.010976+0000","last_clean":"2024-04-25T16:30:15.010976+0000","last_became_active":"2024-04-25T16:30:14.002182+0000","last_became_peered":"2024-04-25T16:30:14.002182+0000","last_unstale":"2024-04-25T16:30:15.010976+0000","last_undegraded":"2024-04-25T16:30:15.010976+0000","last_fullsized":"2024-04-25T16:30:15.010976+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:08.616556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"76'8","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390597+0000","last_change":"2024-04-25T16:30:08.995860+0000","last_active":"2024-04-25T16:30:15.390597+0000","last_peered":"2024-04-25T16:30:15.390597+0000","last_clean":"2024-04-25T16:30:15.390597+0000","last_became_active":"2024-04-25T16:30:08.995487+0000","last_became_peered":"2024-04-25T16:30:08.995487+0000","last_unstale":"2024-04-25T16:30:15.390597+0000","last_undegraded":"2024-04-25T16:30:15.390597+0000","last_fullsized":"2024-04-25T16:30:15.390597+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:48.206502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390893+0000","last_change":"2024-04-25T16:30:09.998820+0000","last_active":"2024-04-25T16:30:15.390893+0000","last_peered":"2024-04-25T16:30:15.390893+0000","last_clean":"2024-04-25T16:30:15.390893+0000","last_became_active":"2024-04-25T16:30:09.998448+0000","last_became_peered":"2024-04-25T16:30:09.998448+0000","last_unstale":"2024-04-25T16:30:15.390893+0000","last_undegraded":"2024-04-25T16:30:15.390893+0000","last_fullsized":"2024-04-25T16:30:15.390893+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:54.361250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390750+0000","last_change":"2024-04-25T16:30:13.992379+0000","last_active":"2024-04-25T16:30:15.390750+0000","last_peered":"2024-04-25T16:30:15.390750+0000","last_clean":"2024-04-25T16:30:15.390750+0000","last_became_active":"2024-04-25T16:30:13.991940+0000","last_became_peered":"2024-04-25T16:30:13.991940+0000","last_unstale":"2024-04-25T16:30:15.390750+0000","last_undegraded":"2024-04-25T16:30:15.390750+0000","last_fullsized":"2024-04-25T16:30:15.390750+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:59.152384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'1","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391883+0000","last_change":"2024-04-25T16:30:08.987701+0000","last_active":"2024-04-25T16:30:15.391883+0000","last_peered":"2024-04-25T16:30:15.391883+0000","last_clean":"2024-04-25T16:30:15.391883+0000","last_became_active":"2024-04-25T16:30:08.987376+0000","last_became_peered":"2024-04-25T16:30:08.987376+0000","last_unstale":"2024-04-25T16:30:15.391883+0000","last_undegraded":"2024-04-25T16:30:15.391883+0000","last_fullsized":"2024-04-25T16:30:15.391883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:21.810100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011317+0000","last_change":"2024-04-25T16:30:09.993005+0000","last_active":"2024-04-25T16:30:15.011317+0000","last_peered":"2024-04-25T16:30:15.011317+0000","last_clean":"2024-04-25T16:30:15.011317+0000","last_became_active":"2024-04-25T16:30:09.992619+0000","last_became_peered":"2024-04-25T16:30:09.992619+0000","last_unstale":"2024-04-25T16:30:15.011317+0000","last_undegraded":"2024-04-25T16:30:15.011317+0000","last_fullsized":"2024-04-25T16:30:15.011317+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:39:40.998241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391135+0000","last_change":"2024-04-25T16:30:14.147895+0000","last_active":"2024-04-25T16:30:15.391135+0000","last_peered":"2024-04-25T16:30:15.391135+0000","last_clean":"2024-04-25T16:30:15.391135+0000","last_became_active":"2024-04-25T16:30:14.147298+0000","last_became_peered":"2024-04-25T16:30:14.147298+0000","last_unstale":"2024-04-25T16:30:15.391135+0000","last_undegraded":"2024-04-25T16:30:15.391135+0000","last_fullsized":"2024-04-25T16:30:15.391135+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:06.269587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391402+0000","last_change":"2024-04-25T16:30:08.982683+0000","last_active":"2024-04-25T16:30:15.391402+0000","last_peered":"2024-04-25T16:30:15.391402+0000","last_clean":"2024-04-25T16:30:15.391402+0000","last_became_active":"2024-04-25T16:30:08.982236+0000","last_became_peered":"2024-04-25T16:30:08.982236+0000","last_unstale":"2024-04-25T16:30:15.391402+0000","last_undegraded":"2024-04-25T16:30:15.391402+0000","last_fullsized":"2024-04-25T16:30:15.391402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:48.781791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.015701+0000","last_change":"2024-04-25T16:30:09.986734+0000","last_active":"2024-04-25T16:30:15.015701+0000","last_peered":"2024-04-25T16:30:15.015701+0000","last_clean":"2024-04-25T16:30:15.015701+0000","last_became_active":"2024-04-25T16:30:09.986337+0000","last_became_peered":"2024-04-25T16:30:09.986337+0000","last_unstale":"2024-04-25T16:30:15.015701+0000","last_undegraded":"2024-04-25T16:30:15.015701+0000","last_fullsized":"2024-04-25T16:30:15.015701+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.678009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010890+0000","last_change":"2024-04-25T16:30:14.003324+0000","last_active":"2024-04-25T16:30:15.010890+0000","last_peered":"2024-04-25T16:30:15.010890+0000","last_clean":"2024-04-25T16:30:15.010890+0000","last_became_active":"2024-04-25T16:30:14.002904+0000","last_became_peered":"2024-04-25T16:30:14.002904+0000","last_unstale":"2024-04-25T16:30:15.010890+0000","last_undegraded":"2024-04-25T16:30:15.010890+0000","last_fullsized":"2024-04-25T16:30:15.010890+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:28.750102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173660+0000","last_change":"2024-04-25T16:30:08.993469+0000","last_active":"2024-04-25T16:30:15.173660+0000","last_peered":"2024-04-25T16:30:15.173660+0000","last_clean":"2024-04-25T16:30:15.173660+0000","last_became_active":"2024-04-25T16:30:08.993126+0000","last_became_peered":"2024-04-25T16:30:08.993126+0000","last_unstale":"2024-04-25T16:30:15.173660+0000","last_undegraded":"2024-04-25T16:30:15.173660+0000","last_fullsized":"2024-04-25T16:30:15.173660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:10.274645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.015338+0000","last_change":"2024-04-25T16:30:10.000597+0000","last_active":"2024-04-25T16:30:15.015338+0000","last_peered":"2024-04-25T16:30:15.015338+0000","last_clean":"2024-04-25T16:30:15.015338+0000","last_became_active":"2024-04-25T16:30:09.998892+0000","last_became_peered":"2024-04-25T16:30:09.998892+0000","last_unstale":"2024-04-25T16:30:15.015338+0000","last_undegraded":"2024-04-25T16:30:15.015338+0000","last_fullsized":"2024-04-25T16:30:15.015338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:09.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010682+0000","last_change":"2024-04-25T16:30:14.147240+0000","last_active":"2024-04-25T16:30:15.010682+0000","last_peered":"2024-04-25T16:30:15.010682+0000","last_clean":"2024-04-25T16:30:15.010682+0000","last_became_active":"2024-04-25T16:30:14.146793+0000","last_became_peered":"2024-04-25T16:30:14.146793+0000","last_unstale":"2024-04-25T16:30:15.010682+0000","last_undegraded":"2024-04-25T16:30:15.010682+0000","last_fullsized":"2024-04-25T16:30:15.010682+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:55.713386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390593+0000","last_change":"2024-04-25T16:30:09.994595+0000","last_active":"2024-04-25T16:30:15.390593+0000","last_peered":"2024-04-25T16:30:15.390593+0000","last_clean":"2024-04-25T16:30:15.390593+0000","last_became_active":"2024-04-25T16:30:09.994305+0000","last_became_peered":"2024-04-25T16:30:09.994305+0000","last_unstale":"2024-04-25T16:30:15.390593+0000","last_undegraded":"2024-04-25T16:30:15.390593+0000","last_fullsized":"2024-04-25T16:30:15.390593+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:27.839500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010206+0000","last_change":"2024-04-25T16:30:08.984203+0000","last_active":"2024-04-25T16:30:15.010206+0000","last_peered":"2024-04-25T16:30:15.010206+0000","last_clean":"2024-04-25T16:30:15.010206+0000","last_became_active":"2024-04-25T16:30:08.982883+0000","last_became_peered":"2024-04-25T16:30:08.982883+0000","last_unstale":"2024-04-25T16:30:15.010206+0000","last_undegraded":"2024-04-25T16:30:15.010206+0000","last_fullsized":"2024-04-25T16:30:15.010206+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:18.447624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174145+0000","last_change":"2024-04-25T16:30:14.007676+0000","last_active":"2024-04-25T16:30:15.174145+0000","last_peered":"2024-04-25T16:30:15.174145+0000","last_clean":"2024-04-25T16:30:15.174145+0000","last_became_active":"2024-04-25T16:30:14.007124+0000","last_became_peered":"2024-04-25T16:30:14.007124+0000","last_unstale":"2024-04-25T16:30:15.174145+0000","last_undegraded":"2024-04-25T16:30:15.174145+0000","last_fullsized":"2024-04-25T16:30:15.174145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:54.142339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392147+0000","last_change":"2024-04-25T16:30:09.990998+0000","last_active":"2024-04-25T16:30:15.392147+0000","last_peered":"2024-04-25T16:30:15.392147+0000","last_clean":"2024-04-25T16:30:15.392147+0000","last_became_active":"2024-04-25T16:30:09.990644+0000","last_became_peered":"2024-04-25T16:30:09.990644+0000","last_unstale":"2024-04-25T16:30:15.392147+0000","last_undegraded":"2024-04-25T16:30:15.392147+0000","last_fullsized":"2024-04-25T16:30:15.392147+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:42.175520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:36.034450+0000","last_change":"2024-04-25T16:30:08.988315+0000","last_active":"2024-04-25T16:30:36.034450+0000","last_peered":"2024-04-25T16:30:36.034450+0000","last_clean":"2024-04-25T16:30:36.034450+0000","last_became_active":"2024-04-25T16:30:08.987930+0000","last_became_peered":"2024-04-25T16:30:08.987930+0000","last_unstale":"2024-04-25T16:30:36.034450+0000","last_undegraded":"2024-04-25T16:30:36.034450+0000","last_fullsized":"2024-04-25T16:30:36.034450+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:44.884602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":97,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173409+0000","last_change":"2024-04-25T16:28:58.219510+0000","last_active":"2024-04-25T16:30:15.173409+0000","last_peered":"2024-04-25T16:30:15.173409+0000","last_clean":"2024-04-25T16:30:15.173409+0000","last_became_active":"2024-04-25T16:28:58.213888+0000","last_became_peered":"2024-04-25T16:28:58.213888+0000","last_unstale":"2024-04-25T16:30:15.173409+0000","last_undegraded":"2024-04-25T16:30:15.173409+0000","last_fullsized":"2024-04-25T16:30:15.173409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_clean_scrub_stamp":"2024-04-25T16:25:33.939186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:15.378406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392417+0000","last_change":"2024-04-25T16:30:14.149020+0000","last_active":"2024-04-25T16:30:15.392417+0000","last_peered":"2024-04-25T16:30:15.392417+0000","last_clean":"2024-04-25T16:30:15.392417+0000","last_became_active":"2024-04-25T16:30:14.148456+0000","last_became_peered":"2024-04-25T16:30:14.148456+0000","last_unstale":"2024-04-25T16:30:15.392417+0000","last_undegraded":"2024-04-25T16:30:15.392417+0000","last_fullsized":"2024-04-25T16:30:15.392417+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:41.011480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010200+0000","last_change":"2024-04-25T16:30:10.005210+0000","last_active":"2024-04-25T16:30:15.010200+0000","last_peered":"2024-04-25T16:30:15.010200+0000","last_clean":"2024-04-25T16:30:15.010200+0000","last_became_active":"2024-04-25T16:30:10.004936+0000","last_became_peered":"2024-04-25T16:30:10.004936+0000","last_unstale":"2024-04-25T16:30:15.010200+0000","last_undegraded":"2024-04-25T16:30:15.010200+0000","last_fullsized":"2024-04-25T16:30:15.010200+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:53.292675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173539+0000","last_change":"2024-04-25T16:30:08.990162+0000","last_active":"2024-04-25T16:30:15.173539+0000","last_peered":"2024-04-25T16:30:15.173539+0000","last_clean":"2024-04-25T16:30:15.173539+0000","last_became_active":"2024-04-25T16:30:08.989613+0000","last_became_peered":"2024-04-25T16:30:08.989613+0000","last_unstale":"2024-04-25T16:30:15.173539+0000","last_undegraded":"2024-04-25T16:30:15.173539+0000","last_fullsized":"2024-04-25T16:30:15.173539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:41.595381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011257+0000","last_change":"2024-04-25T16:30:14.001850+0000","last_active":"2024-04-25T16:30:15.011257+0000","last_peered":"2024-04-25T16:30:15.011257+0000","last_clean":"2024-04-25T16:30:15.011257+0000","last_became_active":"2024-04-25T16:30:14.001348+0000","last_became_peered":"2024-04-25T16:30:14.001348+0000","last_unstale":"2024-04-25T16:30:15.011257+0000","last_undegraded":"2024-04-25T16:30:15.011257+0000","last_fullsized":"2024-04-25T16:30:15.011257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:56:41.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392209+0000","last_change":"2024-04-25T16:30:08.995833+0000","last_active":"2024-04-25T16:30:15.392209+0000","last_peered":"2024-04-25T16:30:15.392209+0000","last_clean":"2024-04-25T16:30:15.392209+0000","last_became_active":"2024-04-25T16:30:08.995545+0000","last_became_peered":"2024-04-25T16:30:08.995545+0000","last_unstale":"2024-04-25T16:30:15.392209+0000","last_undegraded":"2024-04-25T16:30:15.392209+0000","last_fullsized":"2024-04-25T16:30:15.392209+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:31.452219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011120+0000","last_change":"2024-04-25T16:30:09.990225+0000","last_active":"2024-04-25T16:30:15.011120+0000","last_peered":"2024-04-25T16:30:15.011120+0000","last_clean":"2024-04-25T16:30:15.011120+0000","last_became_active":"2024-04-25T16:30:09.989859+0000","last_became_peered":"2024-04-25T16:30:09.989859+0000","last_unstale":"2024-04-25T16:30:15.011120+0000","last_undegraded":"2024-04-25T16:30:15.011120+0000","last_fullsized":"2024-04-25T16:30:15.011120+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:40.589395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390379+0000","last_change":"2024-04-25T16:30:14.147223+0000","last_active":"2024-04-25T16:30:15.390379+0000","last_peered":"2024-04-25T16:30:15.390379+0000","last_clean":"2024-04-25T16:30:15.390379+0000","last_became_active":"2024-04-25T16:30:14.146821+0000","last_became_peered":"2024-04-25T16:30:14.146821+0000","last_unstale":"2024-04-25T16:30:15.390379+0000","last_undegraded":"2024-04-25T16:30:15.390379+0000","last_fullsized":"2024-04-25T16:30:15.390379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:01.289165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391245+0000","last_change":"2024-04-25T16:30:08.993175+0000","last_active":"2024-04-25T16:30:15.391245+0000","last_peered":"2024-04-25T16:30:15.391245+0000","last_clean":"2024-04-25T16:30:15.391245+0000","last_became_active":"2024-04-25T16:30:08.992145+0000","last_became_peered":"2024-04-25T16:30:08.992145+0000","last_unstale":"2024-04-25T16:30:15.391245+0000","last_undegraded":"2024-04-25T16:30:15.391245+0000","last_fullsized":"2024-04-25T16:30:15.391245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:06.951435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390722+0000","last_change":"2024-04-25T16:30:09.986088+0000","last_active":"2024-04-25T16:30:15.390722+0000","last_peered":"2024-04-25T16:30:15.390722+0000","last_clean":"2024-04-25T16:30:15.390722+0000","last_became_active":"2024-04-25T16:30:09.985256+0000","last_became_peered":"2024-04-25T16:30:09.985256+0000","last_unstale":"2024-04-25T16:30:15.390722+0000","last_undegraded":"2024-04-25T16:30:15.390722+0000","last_fullsized":"2024-04-25T16:30:15.390722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:29.197558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391394+0000","last_change":"2024-04-25T16:30:14.000699+0000","last_active":"2024-04-25T16:30:15.391394+0000","last_peered":"2024-04-25T16:30:15.391394+0000","last_clean":"2024-04-25T16:30:15.391394+0000","last_became_active":"2024-04-25T16:30:14.000255+0000","last_became_peered":"2024-04-25T16:30:14.000255+0000","last_unstale":"2024-04-25T16:30:15.391394+0000","last_undegraded":"2024-04-25T16:30:15.391394+0000","last_fullsized":"2024-04-25T16:30:15.391394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:33.436693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:37.042310+0000","last_change":"2024-04-25T16:30:08.981926+0000","last_active":"2024-04-25T16:30:37.042310+0000","last_peered":"2024-04-25T16:30:37.042310+0000","last_clean":"2024-04-25T16:30:37.042310+0000","last_became_active":"2024-04-25T16:30:08.981590+0000","last_became_peered":"2024-04-25T16:30:08.981590+0000","last_unstale":"2024-04-25T16:30:37.042310+0000","last_undegraded":"2024-04-25T16:30:37.042310+0000","last_fullsized":"2024-04-25T16:30:37.042310+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:57.818774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391395+0000","last_change":"2024-04-25T16:30:10.003998+0000","last_active":"2024-04-25T16:30:15.391395+0000","last_peered":"2024-04-25T16:30:15.391395+0000","last_clean":"2024-04-25T16:30:15.391395+0000","last_became_active":"2024-04-25T16:30:10.003626+0000","last_became_peered":"2024-04-25T16:30:10.003626+0000","last_unstale":"2024-04-25T16:30:15.391395+0000","last_undegraded":"2024-04-25T16:30:15.391395+0000","last_fullsized":"2024-04-25T16:30:15.391395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:10.686092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391014+0000","last_change":"2024-04-25T16:30:14.149278+0000","last_active":"2024-04-25T16:30:15.391014+0000","last_peered":"2024-04-25T16:30:15.391014+0000","last_clean":"2024-04-25T16:30:15.391014+0000","last_became_active":"2024-04-25T16:30:14.148654+0000","last_became_peered":"2024-04-25T16:30:14.148654+0000","last_unstale":"2024-04-25T16:30:15.391014+0000","last_undegraded":"2024-04-25T16:30:15.391014+0000","last_fullsized":"2024-04-25T16:30:15.391014+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:24.823222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.389940+0000","last_change":"2024-04-25T16:30:08.981612+0000","last_active":"2024-04-25T16:30:15.389940+0000","last_peered":"2024-04-25T16:30:15.389940+0000","last_clean":"2024-04-25T16:30:15.389940+0000","last_became_active":"2024-04-25T16:30:08.981222+0000","last_became_peered":"2024-04-25T16:30:08.981222+0000","last_unstale":"2024-04-25T16:30:15.389940+0000","last_undegraded":"2024-04-25T16:30:15.389940+0000","last_fullsized":"2024-04-25T16:30:15.389940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:14.914658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390765+0000","last_change":"2024-04-25T16:30:09.989592+0000","last_active":"2024-04-25T16:30:15.390765+0000","last_peered":"2024-04-25T16:30:15.390765+0000","last_clean":"2024-04-25T16:30:15.390765+0000","last_became_active":"2024-04-25T16:30:09.989222+0000","last_became_peered":"2024-04-25T16:30:09.989222+0000","last_unstale":"2024-04-25T16:30:15.390765+0000","last_undegraded":"2024-04-25T16:30:15.390765+0000","last_fullsized":"2024-04-25T16:30:15.390765+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:52.403239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391835+0000","last_change":"2024-04-25T16:30:14.010480+0000","last_active":"2024-04-25T16:30:15.391835+0000","last_peered":"2024-04-25T16:30:15.391835+0000","last_clean":"2024-04-25T16:30:15.391835+0000","last_became_active":"2024-04-25T16:30:14.009023+0000","last_became_peered":"2024-04-25T16:30:14.009023+0000","last_unstale":"2024-04-25T16:30:15.391835+0000","last_undegraded":"2024-04-25T16:30:15.391835+0000","last_fullsized":"2024-04-25T16:30:15.391835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:10.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'5","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:36.034483+0000","last_change":"2024-04-25T16:30:08.986849+0000","last_active":"2024-04-25T16:30:36.034483+0000","last_peered":"2024-04-25T16:30:36.034483+0000","last_clean":"2024-04-25T16:30:36.034483+0000","last_became_active":"2024-04-25T16:30:08.986479+0000","last_became_peered":"2024-04-25T16:30:08.986479+0000","last_unstale":"2024-04-25T16:30:36.034483+0000","last_undegraded":"2024-04-25T16:30:36.034483+0000","last_fullsized":"2024-04-25T16:30:36.034483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010791+0000","last_change":"2024-04-25T16:30:09.990927+0000","last_active":"2024-04-25T16:30:15.010791+0000","last_peered":"2024-04-25T16:30:15.010791+0000","last_clean":"2024-04-25T16:30:15.010791+0000","last_became_active":"2024-04-25T16:30:09.990482+0000","last_became_peered":"2024-04-25T16:30:09.990482+0000","last_unstale":"2024-04-25T16:30:15.010791+0000","last_undegraded":"2024-04-25T16:30:15.010791+0000","last_fullsized":"2024-04-25T16:30:15.010791+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:18.649721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:17.915026+0000","last_change":"2024-04-25T16:30:14.006159+0000","last_active":"2024-04-25T16:30:17.915026+0000","last_peered":"2024-04-25T16:30:17.915026+0000","last_clean":"2024-04-25T16:30:17.915026+0000","last_became_active":"2024-04-25T16:30:14.005413+0000","last_became_peered":"2024-04-25T16:30:14.005413+0000","last_unstale":"2024-04-25T16:30:17.915026+0000","last_undegraded":"2024-04-25T16:30:17.915026+0000","last_fullsized":"2024-04-25T16:30:17.915026+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:52.934385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390457+0000","last_change":"2024-04-25T16:30:08.986207+0000","last_active":"2024-04-25T16:30:15.390457+0000","last_peered":"2024-04-25T16:30:15.390457+0000","last_clean":"2024-04-25T16:30:15.390457+0000","last_became_active":"2024-04-25T16:30:08.985857+0000","last_became_peered":"2024-04-25T16:30:08.985857+0000","last_unstale":"2024-04-25T16:30:15.390457+0000","last_undegraded":"2024-04-25T16:30:15.390457+0000","last_fullsized":"2024-04-25T16:30:15.390457+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:45.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390450+0000","last_change":"2024-04-25T16:30:09.993566+0000","last_active":"2024-04-25T16:30:15.390450+0000","last_peered":"2024-04-25T16:30:15.390450+0000","last_clean":"2024-04-25T16:30:15.390450+0000","last_became_active":"2024-04-25T16:30:09.992250+0000","last_became_peered":"2024-04-25T16:30:09.992250+0000","last_unstale":"2024-04-25T16:30:15.390450+0000","last_undegraded":"2024-04-25T16:30:15.390450+0000","last_fullsized":"2024-04-25T16:30:15.390450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:33.279300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391315+0000","last_change":"2024-04-25T16:30:14.147443+0000","last_active":"2024-04-25T16:30:15.391315+0000","last_peered":"2024-04-25T16:30:15.391315+0000","last_clean":"2024-04-25T16:30:15.391315+0000","last_became_active":"2024-04-25T16:30:14.146838+0000","last_became_peered":"2024-04-25T16:30:14.146838+0000","last_unstale":"2024-04-25T16:30:15.391315+0000","last_undegraded":"2024-04-25T16:30:15.391315+0000","last_fullsized":"2024-04-25T16:30:15.391315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:51.532685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391369+0000","last_change":"2024-04-25T16:30:08.990123+0000","last_active":"2024-04-25T16:30:15.391369+0000","last_peered":"2024-04-25T16:30:15.391369+0000","last_clean":"2024-04-25T16:30:15.391369+0000","last_became_active":"2024-04-25T16:30:08.989769+0000","last_became_peered":"2024-04-25T16:30:08.989769+0000","last_unstale":"2024-04-25T16:30:15.391369+0000","last_undegraded":"2024-04-25T16:30:15.391369+0000","last_fullsized":"2024-04-25T16:30:15.391369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:39.970292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173729+0000","last_change":"2024-04-25T16:30:09.998486+0000","last_active":"2024-04-25T16:30:15.173729+0000","last_peered":"2024-04-25T16:30:15.173729+0000","last_clean":"2024-04-25T16:30:15.173729+0000","last_became_active":"2024-04-25T16:30:09.998154+0000","last_became_peered":"2024-04-25T16:30:09.998154+0000","last_unstale":"2024-04-25T16:30:15.173729+0000","last_undegraded":"2024-04-25T16:30:15.173729+0000","last_fullsized":"2024-04-25T16:30:15.173729+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:55.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390297+0000","last_change":"2024-04-25T16:30:13.995864+0000","last_active":"2024-04-25T16:30:15.390297+0000","last_peered":"2024-04-25T16:30:15.390297+0000","last_clean":"2024-04-25T16:30:15.390297+0000","last_became_active":"2024-04-25T16:30:13.995326+0000","last_became_peered":"2024-04-25T16:30:13.995326+0000","last_unstale":"2024-04-25T16:30:15.390297+0000","last_undegraded":"2024-04-25T16:30:15.390297+0000","last_fullsized":"2024-04-25T16:30:15.390297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:11.718826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391723+0000","last_change":"2024-04-25T16:30:08.984189+0000","last_active":"2024-04-25T16:30:15.391723+0000","last_peered":"2024-04-25T16:30:15.391723+0000","last_clean":"2024-04-25T16:30:15.391723+0000","last_became_active":"2024-04-25T16:30:08.983821+0000","last_became_peered":"2024-04-25T16:30:08.983821+0000","last_unstale":"2024-04-25T16:30:15.391723+0000","last_undegraded":"2024-04-25T16:30:15.391723+0000","last_fullsized":"2024-04-25T16:30:15.391723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:51.315397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174203+0000","last_change":"2024-04-25T16:30:09.998763+0000","last_active":"2024-04-25T16:30:15.174203+0000","last_peered":"2024-04-25T16:30:15.174203+0000","last_clean":"2024-04-25T16:30:15.174203+0000","last_became_active":"2024-04-25T16:30:09.998439+0000","last_became_peered":"2024-04-25T16:30:09.998439+0000","last_unstale":"2024-04-25T16:30:15.174203+0000","last_undegraded":"2024-04-25T16:30:15.174203+0000","last_fullsized":"2024-04-25T16:30:15.174203+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:14.279883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391766+0000","last_change":"2024-04-25T16:30:14.005427+0000","last_active":"2024-04-25T16:30:15.391766+0000","last_peered":"2024-04-25T16:30:15.391766+0000","last_clean":"2024-04-25T16:30:15.391766+0000","last_became_active":"2024-04-25T16:30:14.005006+0000","last_became_peered":"2024-04-25T16:30:14.005006+0000","last_unstale":"2024-04-25T16:30:15.391766+0000","last_undegraded":"2024-04-25T16:30:15.391766+0000","last_fullsized":"2024-04-25T16:30:15.391766+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:49.563365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392462+0000","last_change":"2024-04-25T16:30:08.985557+0000","last_active":"2024-04-25T16:30:15.392462+0000","last_peered":"2024-04-25T16:30:15.392462+0000","last_clean":"2024-04-25T16:30:15.392462+0000","last_became_active":"2024-04-25T16:30:08.985215+0000","last_became_peered":"2024-04-25T16:30:08.985215+0000","last_unstale":"2024-04-25T16:30:15.392462+0000","last_undegraded":"2024-04-25T16:30:15.392462+0000","last_fullsized":"2024-04-25T16:30:15.392462+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:51.504089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173893+0000","last_change":"2024-04-25T16:30:09.991336+0000","last_active":"2024-04-25T16:30:15.173893+0000","last_peered":"2024-04-25T16:30:15.173893+0000","last_clean":"2024-04-25T16:30:15.173893+0000","last_became_active":"2024-04-25T16:30:09.990819+0000","last_became_peered":"2024-04-25T16:30:09.990819+0000","last_unstale":"2024-04-25T16:30:15.173893+0000","last_undegraded":"2024-04-25T16:30:15.173893+0000","last_fullsized":"2024-04-25T16:30:15.173893+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:45.988061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390125+0000","last_change":"2024-04-25T16:30:13.998463+0000","last_active":"2024-04-25T16:30:15.390125+0000","last_peered":"2024-04-25T16:30:15.390125+0000","last_clean":"2024-04-25T16:30:15.390125+0000","last_became_active":"2024-04-25T16:30:13.997952+0000","last_became_peered":"2024-04-25T16:30:13.997952+0000","last_unstale":"2024-04-25T16:30:15.390125+0000","last_undegraded":"2024-04-25T16:30:15.390125+0000","last_fullsized":"2024-04-25T16:30:15.390125+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:58.507942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010889+0000","last_change":"2024-04-25T16:30:08.987232+0000","last_active":"2024-04-25T16:30:15.010889+0000","last_peered":"2024-04-25T16:30:15.010889+0000","last_clean":"2024-04-25T16:30:15.010889+0000","last_became_active":"2024-04-25T16:30:08.986873+0000","last_became_peered":"2024-04-25T16:30:08.986873+0000","last_unstale":"2024-04-25T16:30:15.010889+0000","last_undegraded":"2024-04-25T16:30:15.010889+0000","last_fullsized":"2024-04-25T16:30:15.010889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:40.746479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174197+0000","last_change":"2024-04-25T16:30:09.988566+0000","last_active":"2024-04-25T16:30:15.174197+0000","last_peered":"2024-04-25T16:30:15.174197+0000","last_clean":"2024-04-25T16:30:15.174197+0000","last_became_active":"2024-04-25T16:30:09.988164+0000","last_became_peered":"2024-04-25T16:30:09.988164+0000","last_unstale":"2024-04-25T16:30:15.174197+0000","last_undegraded":"2024-04-25T16:30:15.174197+0000","last_fullsized":"2024-04-25T16:30:15.174197+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:40.476766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011052+0000","last_change":"2024-04-25T16:30:09.996349+0000","last_active":"2024-04-25T16:30:15.011052+0000","last_peered":"2024-04-25T16:30:15.011052+0000","last_clean":"2024-04-25T16:30:15.011052+0000","last_became_active":"2024-04-25T16:30:09.994974+0000","last_became_peered":"2024-04-25T16:30:09.994974+0000","last_unstale":"2024-04-25T16:30:15.011052+0000","last_undegraded":"2024-04-25T16:30:15.011052+0000","last_fullsized":"2024-04-25T16:30:15.011052+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:46.929057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391959+0000","last_change":"2024-04-25T16:30:09.997837+0000","last_active":"2024-04-25T16:30:15.391959+0000","last_peered":"2024-04-25T16:30:15.391959+0000","last_clean":"2024-04-25T16:30:15.391959+0000","last_became_active":"2024-04-25T16:30:09.997361+0000","last_became_peered":"2024-04-25T16:30:09.997361+0000","last_unstale":"2024-04-25T16:30:15.391959+0000","last_undegraded":"2024-04-25T16:30:15.391959+0000","last_fullsized":"2024-04-25T16:30:15.391959+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:35.698528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390589+0000","last_change":"2024-04-25T16:30:09.991794+0000","last_active":"2024-04-25T16:30:15.390589+0000","last_peered":"2024-04-25T16:30:15.390589+0000","last_clean":"2024-04-25T16:30:15.390589+0000","last_became_active":"2024-04-25T16:30:09.991440+0000","last_became_peered":"2024-04-25T16:30:09.991440+0000","last_unstale":"2024-04-25T16:30:15.390589+0000","last_undegraded":"2024-04-25T16:30:15.390589+0000","last_fullsized":"2024-04-25T16:30:15.390589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:11.322254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391942+0000","last_change":"2024-04-25T16:30:10.002364+0000","last_active":"2024-04-25T16:30:15.391942+0000","last_peered":"2024-04-25T16:30:15.391942+0000","last_clean":"2024-04-25T16:30:15.391942+0000","last_became_active":"2024-04-25T16:30:10.001957+0000","last_became_peered":"2024-04-25T16:30:10.001957+0000","last_unstale":"2024-04-25T16:30:15.391942+0000","last_undegraded":"2024-04-25T16:30:15.391942+0000","last_fullsized":"2024-04-25T16:30:15.391942+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:19.150175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011117+0000","last_change":"2024-04-25T16:30:10.004150+0000","last_active":"2024-04-25T16:30:15.011117+0000","last_peered":"2024-04-25T16:30:15.011117+0000","last_clean":"2024-04-25T16:30:15.011117+0000","last_became_active":"2024-04-25T16:30:10.003733+0000","last_became_peered":"2024-04-25T16:30:10.003733+0000","last_unstale":"2024-04-25T16:30:15.011117+0000","last_undegraded":"2024-04-25T16:30:15.011117+0000","last_fullsized":"2024-04-25T16:30:15.011117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:11.850366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011362+0000","last_change":"2024-04-25T16:30:09.991056+0000","last_active":"2024-04-25T16:30:15.011362+0000","last_peered":"2024-04-25T16:30:15.011362+0000","last_clean":"2024-04-25T16:30:15.011362+0000","last_became_active":"2024-04-25T16:30:09.988708+0000","last_became_peered":"2024-04-25T16:30:09.988708+0000","last_unstale":"2024-04-25T16:30:15.011362+0000","last_undegraded":"2024-04-25T16:30:15.011362+0000","last_fullsized":"2024-04-25T16:30:15.011362+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:13.062269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390938+0000","last_change":"2024-04-25T16:30:09.996571+0000","last_active":"2024-04-25T16:30:15.390938+0000","last_peered":"2024-04-25T16:30:15.390938+0000","last_clean":"2024-04-25T16:30:15.390938+0000","last_became_active":"2024-04-25T16:30:09.996206+0000","last_became_peered":"2024-04-25T16:30:09.996206+0000","last_unstale":"2024-04-25T16:30:15.390938+0000","last_undegraded":"2024-04-25T16:30:15.390938+0000","last_fullsized":"2024-04-25T16:30:15.390938+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:07.107949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390912+0000","last_change":"2024-04-25T16:30:09.995067+0000","last_active":"2024-04-25T16:30:15.390912+0000","last_peered":"2024-04-25T16:30:15.390912+0000","last_clean":"2024-04-25T16:30:15.390912+0000","last_became_active":"2024-04-25T16:30:09.994678+0000","last_became_peered":"2024-04-25T16:30:09.994678+0000","last_unstale":"2024-04-25T16:30:15.390912+0000","last_undegraded":"2024-04-25T16:30:15.390912+0000","last_fullsized":"2024-04-25T16:30:15.390912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:09.000869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390478+0000","last_change":"2024-04-25T16:30:09.996821+0000","last_active":"2024-04-25T16:30:15.390478+0000","last_peered":"2024-04-25T16:30:15.390478+0000","last_clean":"2024-04-25T16:30:15.390478+0000","last_became_active":"2024-04-25T16:30:09.996464+0000","last_became_peered":"2024-04-25T16:30:09.996464+0000","last_unstale":"2024-04-25T16:30:15.390478+0000","last_undegraded":"2024-04-25T16:30:15.390478+0000","last_fullsized":"2024-04-25T16:30:15.390478+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:54.961544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391106+0000","last_change":"2024-04-25T16:30:09.991733+0000","last_active":"2024-04-25T16:30:15.391106+0000","last_peered":"2024-04-25T16:30:15.391106+0000","last_clean":"2024-04-25T16:30:15.391106+0000","last_became_active":"2024-04-25T16:30:09.991401+0000","last_became_peered":"2024-04-25T16:30:09.991401+0000","last_unstale":"2024-04-25T16:30:15.391106+0000","last_undegraded":"2024-04-25T16:30:15.391106+0000","last_fullsized":"2024-04-25T16:30:15.391106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:14.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390619+0000","last_change":"2024-04-25T16:30:09.999087+0000","last_active":"2024-04-25T16:30:15.390619+0000","last_peered":"2024-04-25T16:30:15.390619+0000","last_clean":"2024-04-25T16:30:15.390619+0000","last_became_active":"2024-04-25T16:30:09.998725+0000","last_became_peered":"2024-04-25T16:30:09.998725+0000","last_unstale":"2024-04-25T16:30:15.390619+0000","last_undegraded":"2024-04-25T16:30:15.390619+0000","last_fullsized":"2024-04-25T16:30:15.390619+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:19.155171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010744+0000","last_change":"2024-04-25T16:30:09.995880+0000","last_active":"2024-04-25T16:30:15.010744+0000","last_peered":"2024-04-25T16:30:15.010744+0000","last_clean":"2024-04-25T16:30:15.010744+0000","last_became_active":"2024-04-25T16:30:09.995514+0000","last_became_peered":"2024-04-25T16:30:09.995514+0000","last_unstale":"2024-04-25T16:30:15.010744+0000","last_undegraded":"2024-04-25T16:30:15.010744+0000","last_fullsized":"2024-04-25T16:30:15.010744+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011017+0000","last_change":"2024-04-25T16:30:09.998326+0000","last_active":"2024-04-25T16:30:15.011017+0000","last_peered":"2024-04-25T16:30:15.011017+0000","last_clean":"2024-04-25T16:30:15.011017+0000","last_became_active":"2024-04-25T16:30:09.995803+0000","last_became_peered":"2024-04-25T16:30:09.995803+0000","last_unstale":"2024-04-25T16:30:15.011017+0000","last_undegraded":"2024-04-25T16:30:15.011017+0000","last_fullsized":"2024-04-25T16:30:15.011017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:46.983947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5979,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":18799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645330,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":433354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841559,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36836,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708316,"statfs":{"total":95995035648,"available":95957315584,"internally_reserved":0,"allocated":1536000,"data_stored":1028074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037787,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":432744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234016,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":897024,"data_stored":432766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430245,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1486848,"data_stored":1023112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593771,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":888832,"data_stored":432162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":901120,"data_stored":436004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":901120,"data_stored":432856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":436038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":913408,"data_stored":432570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953526,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36224,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708928,"statfs":{"total":95995035648,"available":95957942272,"internally_reserved":0,"allocated":909312,"data_stored":433253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790000,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1495040,"data_stored":1022712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:30:38.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:30:38.978 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.978+0000 7f311524c640 1 -- 172.21.15.98:0/3018568046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31101098b0 msgr2=0x7f311010bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.978+0000 7f311524c640 1 --2- 172.21.15.98:0/3018568046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31101098b0 0x7f311010bcd0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3108009ca0 tx=0x7f310802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.978+0000 7f311524c640 1 -- 172.21.15.98:0/3018568046 shutdown_connections 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.978+0000 7f311524c640 1 --2- 172.21.15.98:0/3018568046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31101098b0 0x7f311010bcd0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.978+0000 7f311524c640 1 --2- 172.21.15.98:0/3018568046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3110106f50 0x7f3110109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.978+0000 7f311524c640 1 --2- 172.21.15.98:0/3018568046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31101045f0 0x7f3110106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.978+0000 7f311524c640 1 -- 172.21.15.98:0/3018568046 >> 172.21.15.98:0/3018568046 conn(0x7f31100fdbf0 msgr2=0x7f31101000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.978+0000 7f311524c640 1 -- 172.21.15.98:0/3018568046 shutdown_connections 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.979+0000 7f311524c640 1 -- 172.21.15.98:0/3018568046 wait complete. 2024-04-25T16:30:38.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.979+0000 7f311524c640 1 Processor -- start 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.979+0000 7f311524c640 1 -- start start 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f311524c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31101045f0 0x7f311019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f311524c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3110106f50 0x7f311019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f311524c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31101098b0 0x7f31101a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f311524c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f311010f530 con 0x7f3110106f50 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f311524c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f311010f3b0 con 0x7f31101045f0 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f311524c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f311010f6b0 con 0x7f31101098b0 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3110106f50 0x7f311019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310ed9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31101045f0 0x7f311019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3110106f50 0x7f311019a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37096/0 (socket says 172.21.15.98:37096) 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310e59c640 1 -- 172.21.15.98:0/3179129148 learned_addr learned my addr 172.21.15.98:0/3179129148 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310f59e640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31101098b0 0x7f31101a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310ed9d640 1 -- 172.21.15.98:0/3179129148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31101098b0 msgr2=0x7f31101a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310ed9d640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31101098b0 0x7f31101a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310ed9d640 1 -- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3110106f50 msgr2=0x7f311019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310ed9d640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3110106f50 0x7f311019a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310ed9d640 1 -- 172.21.15.98:0/3179129148 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31101a1c10 con 0x7f31101045f0 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310e59c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3110106f50 0x7f311019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310f59e640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31101098b0 0x7f31101a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.980+0000 7f310ed9d640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31101045f0 0x7f311019a0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f310000e9c0 tx=0x7f310000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.981+0000 7f30f7fff640 1 -- 172.21.15.98:0/3179129148 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310000cd80 con 0x7f31101045f0 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.981+0000 7f30f7fff640 1 -- 172.21.15.98:0/3179129148 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f31000035f0 con 0x7f31101045f0 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.981+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31101a1f00 con 0x7f31101045f0 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.981+0000 7f30f7fff640 1 -- 172.21.15.98:0/3179129148 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3100010690 con 0x7f31101045f0 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.981+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31101a2360 con 0x7f31101045f0 2024-04-25T16:30:38.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.982+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f311006a190 con 0x7f31101045f0 2024-04-25T16:30:38.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.983+0000 7f30f7fff640 1 -- 172.21.15.98:0/3179129148 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3100004000 con 0x7f31101045f0 2024-04-25T16:30:38.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.983+0000 7f30f7fff640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30f80afb80 0x7f30f80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:38.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.983+0000 7f310e59c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30f80afb80 0x7f30f80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:38.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.983+0000 7f30f7fff640 1 -- 172.21.15.98:0/3179129148 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f31000df650 con 0x7f31101045f0 2024-04-25T16:30:38.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.984+0000 7f310e59c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30f80afb80 0x7f30f80b2090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f311019b170 tx=0x7f310400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:38.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:38.987+0000 7f30f7fff640 1 -- 172.21.15.98:0/3179129148 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31000a7e90 con 0x7f31101045f0 2024-04-25T16:30:39.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.163+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3110062fe0 con 0x7f30f80afb80 2024-04-25T16:30:39.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.172+0000 7f30f7fff640 1 -- 172.21.15.98:0/3179129148 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340243 (secure 0 0 0) 0x7f3110062fe0 con 0x7f30f80afb80 2024-04-25T16:30:39.176 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:39.177 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T16:30:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30f80afb80 msgr2=0x7f30f80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30f80afb80 0x7f30f80b2090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f311019b170 tx=0x7f310400b410 comp rx=0 tx=0).stop 2024-04-25T16:30:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31101045f0 msgr2=0x7f311019a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31101045f0 0x7f311019a0d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f310000e9c0 tx=0x7f310000ee80 comp rx=0 tx=0).stop 2024-04-25T16:30:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 shutdown_connections 2024-04-25T16:30:39.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f30f80afb80 0x7f30f80b2090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31101098b0 0x7f31101a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3110106f50 0x7f311019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 --2- 172.21.15.98:0/3179129148 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31101045f0 0x7f311019a0d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:39.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 >> 172.21.15.98:0/3179129148 conn(0x7f31100fdbf0 msgr2=0x7f3110105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:39.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 shutdown_connections 2024-04-25T16:30:39.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:39.176+0000 7f311524c640 1 -- 172.21.15.98:0/3179129148 wait complete. 2024-04-25T16:30:39.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:39 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] up:active 2024-04-25T16:30:39.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:39 smithi098 ceph-mon[29162]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-25T16:30:39.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:39 smithi098 ceph-mon[29162]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T16:30:39.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:39 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:30:39.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:39 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1248976264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:39 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1018412835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:39 smithi098 ceph-mon[29162]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:39.187 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-04-25T16:30:37.928447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596464,"num_objects":46,"num_object_clones":0,"num_object_copies":139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":218,"num_write_kb":2184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":171,"ondisk_log_size":171,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":386468,"kb_used_data":12332,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124555356,"statfs":{"total":1151940427776,"available":1151544684544,"internally_reserved":0,"allocated":12627968,"data_stored":6975645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":382973242},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3686,"num_objects":22,"num_object_clones":0,"num_object_copies":67,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":20,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003930"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011452+0000","last_change":"2024-04-25T16:30:09.990700+0000","last_active":"2024-04-25T16:30:15.011452+0000","last_peered":"2024-04-25T16:30:15.011452+0000","last_clean":"2024-04-25T16:30:15.011452+0000","last_became_active":"2024-04-25T16:30:09.990225+0000","last_became_peered":"2024-04-25T16:30:09.990225+0000","last_unstale":"2024-04-25T16:30:15.011452+0000","last_undegraded":"2024-04-25T16:30:15.011452+0000","last_fullsized":"2024-04-25T16:30:15.011452+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:33.897277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391502+0000","last_change":"2024-04-25T16:30:09.989618+0000","last_active":"2024-04-25T16:30:15.391502+0000","last_peered":"2024-04-25T16:30:15.391502+0000","last_clean":"2024-04-25T16:30:15.391502+0000","last_became_active":"2024-04-25T16:30:09.989270+0000","last_became_peered":"2024-04-25T16:30:09.989270+0000","last_unstale":"2024-04-25T16:30:15.391502+0000","last_undegraded":"2024-04-25T16:30:15.391502+0000","last_fullsized":"2024-04-25T16:30:15.391502+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:03.952233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391028+0000","last_change":"2024-04-25T16:30:09.992442+0000","last_active":"2024-04-25T16:30:15.391028+0000","last_peered":"2024-04-25T16:30:15.391028+0000","last_clean":"2024-04-25T16:30:15.391028+0000","last_became_active":"2024-04-25T16:30:09.992115+0000","last_became_peered":"2024-04-25T16:30:09.992115+0000","last_unstale":"2024-04-25T16:30:15.391028+0000","last_undegraded":"2024-04-25T16:30:15.391028+0000","last_fullsized":"2024-04-25T16:30:15.391028+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:00.668427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391344+0000","last_change":"2024-04-25T16:30:10.001366+0000","last_active":"2024-04-25T16:30:15.391344+0000","last_peered":"2024-04-25T16:30:15.391344+0000","last_clean":"2024-04-25T16:30:15.391344+0000","last_became_active":"2024-04-25T16:30:10.001000+0000","last_became_peered":"2024-04-25T16:30:10.001000+0000","last_unstale":"2024-04-25T16:30:15.391344+0000","last_undegraded":"2024-04-25T16:30:15.391344+0000","last_fullsized":"2024-04-25T16:30:15.391344+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:32.269811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391245+0000","last_change":"2024-04-25T16:30:09.991372+0000","last_active":"2024-04-25T16:30:15.391245+0000","last_peered":"2024-04-25T16:30:15.391245+0000","last_clean":"2024-04-25T16:30:15.391245+0000","last_became_active":"2024-04-25T16:30:09.991053+0000","last_became_peered":"2024-04-25T16:30:09.991053+0000","last_unstale":"2024-04-25T16:30:15.391245+0000","last_undegraded":"2024-04-25T16:30:15.391245+0000","last_fullsized":"2024-04-25T16:30:15.391245+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:28.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.015408+0000","last_change":"2024-04-25T16:30:09.997563+0000","last_active":"2024-04-25T16:30:15.015408+0000","last_peered":"2024-04-25T16:30:15.015408+0000","last_clean":"2024-04-25T16:30:15.015408+0000","last_became_active":"2024-04-25T16:30:09.997158+0000","last_became_peered":"2024-04-25T16:30:09.997158+0000","last_unstale":"2024-04-25T16:30:15.015408+0000","last_undegraded":"2024-04-25T16:30:15.015408+0000","last_fullsized":"2024-04-25T16:30:15.015408+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:21.441957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173528+0000","last_change":"2024-04-25T16:30:09.989859+0000","last_active":"2024-04-25T16:30:15.173528+0000","last_peered":"2024-04-25T16:30:15.173528+0000","last_clean":"2024-04-25T16:30:15.173528+0000","last_became_active":"2024-04-25T16:30:09.989291+0000","last_became_peered":"2024-04-25T16:30:09.989291+0000","last_unstale":"2024-04-25T16:30:15.173528+0000","last_undegraded":"2024-04-25T16:30:15.173528+0000","last_fullsized":"2024-04-25T16:30:15.173528+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:07.833941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390141+0000","last_change":"2024-04-25T16:30:09.994820+0000","last_active":"2024-04-25T16:30:15.390141+0000","last_peered":"2024-04-25T16:30:15.390141+0000","last_clean":"2024-04-25T16:30:15.390141+0000","last_became_active":"2024-04-25T16:30:09.994494+0000","last_became_peered":"2024-04-25T16:30:09.994494+0000","last_unstale":"2024-04-25T16:30:15.390141+0000","last_undegraded":"2024-04-25T16:30:15.390141+0000","last_fullsized":"2024-04-25T16:30:15.390141+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:39.373266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390710+0000","last_change":"2024-04-25T16:30:09.994870+0000","last_active":"2024-04-25T16:30:15.390710+0000","last_peered":"2024-04-25T16:30:15.390710+0000","last_clean":"2024-04-25T16:30:15.390710+0000","last_became_active":"2024-04-25T16:30:09.994521+0000","last_became_peered":"2024-04-25T16:30:09.994521+0000","last_unstale":"2024-04-25T16:30:15.390710+0000","last_undegraded":"2024-04-25T16:30:15.390710+0000","last_fullsized":"2024-04-25T16:30:15.390710+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:18.093904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.015924+0000","last_change":"2024-04-25T16:30:09.999199+0000","last_active":"2024-04-25T16:30:15.015924+0000","last_peered":"2024-04-25T16:30:15.015924+0000","last_clean":"2024-04-25T16:30:15.015924+0000","last_became_active":"2024-04-25T16:30:09.998822+0000","last_became_peered":"2024-04-25T16:30:09.998822+0000","last_unstale":"2024-04-25T16:30:15.015924+0000","last_undegraded":"2024-04-25T16:30:15.015924+0000","last_fullsized":"2024-04-25T16:30:15.015924+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:07:28.955045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174178+0000","last_change":"2024-04-25T16:30:09.999519+0000","last_active":"2024-04-25T16:30:15.174178+0000","last_peered":"2024-04-25T16:30:15.174178+0000","last_clean":"2024-04-25T16:30:15.174178+0000","last_became_active":"2024-04-25T16:30:09.999158+0000","last_became_peered":"2024-04-25T16:30:09.999158+0000","last_unstale":"2024-04-25T16:30:15.174178+0000","last_undegraded":"2024-04-25T16:30:15.174178+0000","last_fullsized":"2024-04-25T16:30:15.174178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:38:06.686541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391442+0000","last_change":"2024-04-25T16:30:10.005455+0000","last_active":"2024-04-25T16:30:15.391442+0000","last_peered":"2024-04-25T16:30:15.391442+0000","last_clean":"2024-04-25T16:30:15.391442+0000","last_became_active":"2024-04-25T16:30:10.004891+0000","last_became_peered":"2024-04-25T16:30:10.004891+0000","last_unstale":"2024-04-25T16:30:15.391442+0000","last_undegraded":"2024-04-25T16:30:15.391442+0000","last_fullsized":"2024-04-25T16:30:15.391442+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.598321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010625+0000","last_change":"2024-04-25T16:30:09.992870+0000","last_active":"2024-04-25T16:30:15.010625+0000","last_peered":"2024-04-25T16:30:15.010625+0000","last_clean":"2024-04-25T16:30:15.010625+0000","last_became_active":"2024-04-25T16:30:09.992399+0000","last_became_peered":"2024-04-25T16:30:09.992399+0000","last_unstale":"2024-04-25T16:30:15.010625+0000","last_undegraded":"2024-04-25T16:30:15.010625+0000","last_fullsized":"2024-04-25T16:30:15.010625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.127287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390761+0000","last_change":"2024-04-25T16:30:09.988707+0000","last_active":"2024-04-25T16:30:15.390761+0000","last_peered":"2024-04-25T16:30:15.390761+0000","last_clean":"2024-04-25T16:30:15.390761+0000","last_became_active":"2024-04-25T16:30:09.988386+0000","last_became_peered":"2024-04-25T16:30:09.988386+0000","last_unstale":"2024-04-25T16:30:15.390761+0000","last_undegraded":"2024-04-25T16:30:15.390761+0000","last_fullsized":"2024-04-25T16:30:15.390761+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:03:15.973591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391413+0000","last_change":"2024-04-25T16:30:10.002780+0000","last_active":"2024-04-25T16:30:15.391413+0000","last_peered":"2024-04-25T16:30:15.391413+0000","last_clean":"2024-04-25T16:30:15.391413+0000","last_became_active":"2024-04-25T16:30:10.002055+0000","last_became_peered":"2024-04-25T16:30:10.002055+0000","last_unstale":"2024-04-25T16:30:15.391413+0000","last_undegraded":"2024-04-25T16:30:15.391413+0000","last_fullsized":"2024-04-25T16:30:15.391413+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:23.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390092+0000","last_change":"2024-04-25T16:30:09.998270+0000","last_active":"2024-04-25T16:30:15.390092+0000","last_peered":"2024-04-25T16:30:15.390092+0000","last_clean":"2024-04-25T16:30:15.390092+0000","last_became_active":"2024-04-25T16:30:09.997903+0000","last_became_peered":"2024-04-25T16:30:09.997903+0000","last_unstale":"2024-04-25T16:30:15.390092+0000","last_undegraded":"2024-04-25T16:30:15.390092+0000","last_fullsized":"2024-04-25T16:30:15.390092+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:14.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011302+0000","last_change":"2024-04-25T16:30:09.988894+0000","last_active":"2024-04-25T16:30:15.011302+0000","last_peered":"2024-04-25T16:30:15.011302+0000","last_clean":"2024-04-25T16:30:15.011302+0000","last_became_active":"2024-04-25T16:30:09.988508+0000","last_became_peered":"2024-04-25T16:30:09.988508+0000","last_unstale":"2024-04-25T16:30:15.011302+0000","last_undegraded":"2024-04-25T16:30:15.011302+0000","last_fullsized":"2024-04-25T16:30:15.011302+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:48.226193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391238+0000","last_change":"2024-04-25T16:30:09.989910+0000","last_active":"2024-04-25T16:30:15.391238+0000","last_peered":"2024-04-25T16:30:15.391238+0000","last_clean":"2024-04-25T16:30:15.391238+0000","last_became_active":"2024-04-25T16:30:09.989560+0000","last_became_peered":"2024-04-25T16:30:09.989560+0000","last_unstale":"2024-04-25T16:30:15.391238+0000","last_undegraded":"2024-04-25T16:30:15.391238+0000","last_fullsized":"2024-04-25T16:30:15.391238+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:12.834034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392709+0000","last_change":"2024-04-25T16:30:09.987930+0000","last_active":"2024-04-25T16:30:15.392709+0000","last_peered":"2024-04-25T16:30:15.392709+0000","last_clean":"2024-04-25T16:30:15.392709+0000","last_became_active":"2024-04-25T16:30:09.987595+0000","last_became_peered":"2024-04-25T16:30:09.987595+0000","last_unstale":"2024-04-25T16:30:15.392709+0000","last_undegraded":"2024-04-25T16:30:15.392709+0000","last_fullsized":"2024-04-25T16:30:15.392709+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:04.778172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391702+0000","last_change":"2024-04-25T16:30:14.006360+0000","last_active":"2024-04-25T16:30:15.391702+0000","last_peered":"2024-04-25T16:30:15.391702+0000","last_clean":"2024-04-25T16:30:15.391702+0000","last_became_active":"2024-04-25T16:30:14.005894+0000","last_became_peered":"2024-04-25T16:30:14.005894+0000","last_unstale":"2024-04-25T16:30:15.391702+0000","last_undegraded":"2024-04-25T16:30:15.391702+0000","last_fullsized":"2024-04-25T16:30:15.391702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:33.150381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"76'4","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391975+0000","last_change":"2024-04-25T16:30:08.992779+0000","last_active":"2024-04-25T16:30:15.391975+0000","last_peered":"2024-04-25T16:30:15.391975+0000","last_clean":"2024-04-25T16:30:15.391975+0000","last_became_active":"2024-04-25T16:30:08.992458+0000","last_became_peered":"2024-04-25T16:30:08.992458+0000","last_unstale":"2024-04-25T16:30:15.391975+0000","last_undegraded":"2024-04-25T16:30:15.391975+0000","last_fullsized":"2024-04-25T16:30:15.391975+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:21.430855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390554+0000","last_change":"2024-04-25T16:30:09.995353+0000","last_active":"2024-04-25T16:30:15.390554+0000","last_peered":"2024-04-25T16:30:15.390554+0000","last_clean":"2024-04-25T16:30:15.390554+0000","last_became_active":"2024-04-25T16:30:09.995011+0000","last_became_peered":"2024-04-25T16:30:09.995011+0000","last_unstale":"2024-04-25T16:30:15.390554+0000","last_undegraded":"2024-04-25T16:30:15.390554+0000","last_fullsized":"2024-04-25T16:30:15.390554+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:33.367268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390178+0000","last_change":"2024-04-25T16:30:14.005848+0000","last_active":"2024-04-25T16:30:15.390178+0000","last_peered":"2024-04-25T16:30:15.390178+0000","last_clean":"2024-04-25T16:30:15.390178+0000","last_became_active":"2024-04-25T16:30:14.005207+0000","last_became_peered":"2024-04-25T16:30:14.005207+0000","last_unstale":"2024-04-25T16:30:15.390178+0000","last_undegraded":"2024-04-25T16:30:15.390178+0000","last_fullsized":"2024-04-25T16:30:15.390178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:01.534652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390770+0000","last_change":"2024-04-25T16:30:08.997663+0000","last_active":"2024-04-25T16:30:15.390770+0000","last_peered":"2024-04-25T16:30:15.390770+0000","last_clean":"2024-04-25T16:30:15.390770+0000","last_became_active":"2024-04-25T16:30:08.997107+0000","last_became_peered":"2024-04-25T16:30:08.997107+0000","last_unstale":"2024-04-25T16:30:15.390770+0000","last_undegraded":"2024-04-25T16:30:15.390770+0000","last_fullsized":"2024-04-25T16:30:15.390770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:25.856556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391752+0000","last_change":"2024-04-25T16:30:09.987947+0000","last_active":"2024-04-25T16:30:15.391752+0000","last_peered":"2024-04-25T16:30:15.391752+0000","last_clean":"2024-04-25T16:30:15.391752+0000","last_became_active":"2024-04-25T16:30:09.987575+0000","last_became_peered":"2024-04-25T16:30:09.987575+0000","last_unstale":"2024-04-25T16:30:15.391752+0000","last_undegraded":"2024-04-25T16:30:15.391752+0000","last_fullsized":"2024-04-25T16:30:15.391752+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:28.443249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391491+0000","last_change":"2024-04-25T16:30:14.006711+0000","last_active":"2024-04-25T16:30:15.391491+0000","last_peered":"2024-04-25T16:30:15.391491+0000","last_clean":"2024-04-25T16:30:15.391491+0000","last_became_active":"2024-04-25T16:30:14.005500+0000","last_became_peered":"2024-04-25T16:30:14.005500+0000","last_unstale":"2024-04-25T16:30:15.391491+0000","last_undegraded":"2024-04-25T16:30:15.391491+0000","last_fullsized":"2024-04-25T16:30:15.391491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:58:26.288410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392394+0000","last_change":"2024-04-25T16:30:08.993302+0000","last_active":"2024-04-25T16:30:15.392394+0000","last_peered":"2024-04-25T16:30:15.392394+0000","last_clean":"2024-04-25T16:30:15.392394+0000","last_became_active":"2024-04-25T16:30:08.992918+0000","last_became_peered":"2024-04-25T16:30:08.992918+0000","last_unstale":"2024-04-25T16:30:15.392394+0000","last_undegraded":"2024-04-25T16:30:15.392394+0000","last_fullsized":"2024-04-25T16:30:15.392394+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:18.843803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391027+0000","last_change":"2024-04-25T16:30:09.988216+0000","last_active":"2024-04-25T16:30:15.391027+0000","last_peered":"2024-04-25T16:30:15.391027+0000","last_clean":"2024-04-25T16:30:15.391027+0000","last_became_active":"2024-04-25T16:30:09.987752+0000","last_became_peered":"2024-04-25T16:30:09.987752+0000","last_unstale":"2024-04-25T16:30:15.391027+0000","last_undegraded":"2024-04-25T16:30:15.391027+0000","last_fullsized":"2024-04-25T16:30:15.391027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:57:35.583000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390498+0000","last_change":"2024-04-25T16:30:14.001329+0000","last_active":"2024-04-25T16:30:15.390498+0000","last_peered":"2024-04-25T16:30:15.390498+0000","last_clean":"2024-04-25T16:30:15.390498+0000","last_became_active":"2024-04-25T16:30:14.000878+0000","last_became_peered":"2024-04-25T16:30:14.000878+0000","last_unstale":"2024-04-25T16:30:15.390498+0000","last_undegraded":"2024-04-25T16:30:15.390498+0000","last_fullsized":"2024-04-25T16:30:15.390498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:07.946611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391652+0000","last_change":"2024-04-25T16:30:08.989065+0000","last_active":"2024-04-25T16:30:15.391652+0000","last_peered":"2024-04-25T16:30:15.391652+0000","last_clean":"2024-04-25T16:30:15.391652+0000","last_became_active":"2024-04-25T16:30:08.988709+0000","last_became_peered":"2024-04-25T16:30:08.988709+0000","last_unstale":"2024-04-25T16:30:15.391652+0000","last_undegraded":"2024-04-25T16:30:15.391652+0000","last_fullsized":"2024-04-25T16:30:15.391652+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:34.306581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392480+0000","last_change":"2024-04-25T16:30:09.999641+0000","last_active":"2024-04-25T16:30:15.392480+0000","last_peered":"2024-04-25T16:30:15.392480+0000","last_clean":"2024-04-25T16:30:15.392480+0000","last_became_active":"2024-04-25T16:30:09.999319+0000","last_became_peered":"2024-04-25T16:30:09.999319+0000","last_unstale":"2024-04-25T16:30:15.392480+0000","last_undegraded":"2024-04-25T16:30:15.392480+0000","last_fullsized":"2024-04-25T16:30:15.392480+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:14.043201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392554+0000","last_change":"2024-04-25T16:30:13.999609+0000","last_active":"2024-04-25T16:30:15.392554+0000","last_peered":"2024-04-25T16:30:15.392554+0000","last_clean":"2024-04-25T16:30:15.392554+0000","last_became_active":"2024-04-25T16:30:13.999172+0000","last_became_peered":"2024-04-25T16:30:13.999172+0000","last_unstale":"2024-04-25T16:30:15.392554+0000","last_undegraded":"2024-04-25T16:30:15.392554+0000","last_fullsized":"2024-04-25T16:30:15.392554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:49.878153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"76'1","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391832+0000","last_change":"2024-04-25T16:30:08.988919+0000","last_active":"2024-04-25T16:30:15.391832+0000","last_peered":"2024-04-25T16:30:15.391832+0000","last_clean":"2024-04-25T16:30:15.391832+0000","last_became_active":"2024-04-25T16:30:08.988218+0000","last_became_peered":"2024-04-25T16:30:08.988218+0000","last_unstale":"2024-04-25T16:30:15.391832+0000","last_undegraded":"2024-04-25T16:30:15.391832+0000","last_fullsized":"2024-04-25T16:30:15.391832+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:57:57.046055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391080+0000","last_change":"2024-04-25T16:30:09.992498+0000","last_active":"2024-04-25T16:30:15.391080+0000","last_peered":"2024-04-25T16:30:15.391080+0000","last_clean":"2024-04-25T16:30:15.391080+0000","last_became_active":"2024-04-25T16:30:09.992190+0000","last_became_peered":"2024-04-25T16:30:09.992190+0000","last_unstale":"2024-04-25T16:30:15.391080+0000","last_undegraded":"2024-04-25T16:30:15.391080+0000","last_fullsized":"2024-04-25T16:30:15.391080+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:34.407963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391491+0000","last_change":"2024-04-25T16:30:14.007371+0000","last_active":"2024-04-25T16:30:15.391491+0000","last_peered":"2024-04-25T16:30:15.391491+0000","last_clean":"2024-04-25T16:30:15.391491+0000","last_became_active":"2024-04-25T16:30:14.006811+0000","last_became_peered":"2024-04-25T16:30:14.006811+0000","last_unstale":"2024-04-25T16:30:15.391491+0000","last_undegraded":"2024-04-25T16:30:15.391491+0000","last_fullsized":"2024-04-25T16:30:15.391491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:01.473142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:36.035768+0000","last_change":"2024-04-25T16:30:08.984080+0000","last_active":"2024-04-25T16:30:36.035768+0000","last_peered":"2024-04-25T16:30:36.035768+0000","last_clean":"2024-04-25T16:30:36.035768+0000","last_became_active":"2024-04-25T16:30:08.983702+0000","last_became_peered":"2024-04-25T16:30:08.983702+0000","last_unstale":"2024-04-25T16:30:36.035768+0000","last_undegraded":"2024-04-25T16:30:36.035768+0000","last_fullsized":"2024-04-25T16:30:36.035768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:22.311986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392067+0000","last_change":"2024-04-25T16:30:09.997913+0000","last_active":"2024-04-25T16:30:15.392067+0000","last_peered":"2024-04-25T16:30:15.392067+0000","last_clean":"2024-04-25T16:30:15.392067+0000","last_became_active":"2024-04-25T16:30:09.997567+0000","last_became_peered":"2024-04-25T16:30:09.997567+0000","last_unstale":"2024-04-25T16:30:15.392067+0000","last_undegraded":"2024-04-25T16:30:15.392067+0000","last_fullsized":"2024-04-25T16:30:15.392067+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:20.829798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011394+0000","last_change":"2024-04-25T16:30:14.003218+0000","last_active":"2024-04-25T16:30:15.011394+0000","last_peered":"2024-04-25T16:30:15.011394+0000","last_clean":"2024-04-25T16:30:15.011394+0000","last_became_active":"2024-04-25T16:30:14.002782+0000","last_became_peered":"2024-04-25T16:30:14.002782+0000","last_unstale":"2024-04-25T16:30:15.011394+0000","last_undegraded":"2024-04-25T16:30:15.011394+0000","last_fullsized":"2024-04-25T16:30:15.011394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010397+0000","last_change":"2024-04-25T16:30:08.993447+0000","last_active":"2024-04-25T16:30:15.010397+0000","last_peered":"2024-04-25T16:30:15.010397+0000","last_clean":"2024-04-25T16:30:15.010397+0000","last_became_active":"2024-04-25T16:30:08.993108+0000","last_became_peered":"2024-04-25T16:30:08.993108+0000","last_unstale":"2024-04-25T16:30:15.010397+0000","last_undegraded":"2024-04-25T16:30:15.010397+0000","last_fullsized":"2024-04-25T16:30:15.010397+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:00.511392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391022+0000","last_change":"2024-04-25T16:30:09.989863+0000","last_active":"2024-04-25T16:30:15.391022+0000","last_peered":"2024-04-25T16:30:15.391022+0000","last_clean":"2024-04-25T16:30:15.391022+0000","last_became_active":"2024-04-25T16:30:09.989520+0000","last_became_peered":"2024-04-25T16:30:09.989520+0000","last_unstale":"2024-04-25T16:30:15.391022+0000","last_undegraded":"2024-04-25T16:30:15.391022+0000","last_fullsized":"2024-04-25T16:30:15.391022+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:54.178298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391280+0000","last_change":"2024-04-25T16:30:13.999904+0000","last_active":"2024-04-25T16:30:15.391280+0000","last_peered":"2024-04-25T16:30:15.391280+0000","last_clean":"2024-04-25T16:30:15.391280+0000","last_became_active":"2024-04-25T16:30:13.999468+0000","last_became_peered":"2024-04-25T16:30:13.999468+0000","last_unstale":"2024-04-25T16:30:15.391280+0000","last_undegraded":"2024-04-25T16:30:15.391280+0000","last_fullsized":"2024-04-25T16:30:15.391280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:09.653365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390359+0000","last_change":"2024-04-25T16:30:08.987550+0000","last_active":"2024-04-25T16:30:15.390359+0000","last_peered":"2024-04-25T16:30:15.390359+0000","last_clean":"2024-04-25T16:30:15.390359+0000","last_became_active":"2024-04-25T16:30:08.987231+0000","last_became_peered":"2024-04-25T16:30:08.987231+0000","last_unstale":"2024-04-25T16:30:15.390359+0000","last_undegraded":"2024-04-25T16:30:15.390359+0000","last_fullsized":"2024-04-25T16:30:15.390359+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:01.110960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390870+0000","last_change":"2024-04-25T16:30:09.991985+0000","last_active":"2024-04-25T16:30:15.390870+0000","last_peered":"2024-04-25T16:30:15.390870+0000","last_clean":"2024-04-25T16:30:15.390870+0000","last_became_active":"2024-04-25T16:30:09.991645+0000","last_became_peered":"2024-04-25T16:30:09.991645+0000","last_unstale":"2024-04-25T16:30:15.390870+0000","last_undegraded":"2024-04-25T16:30:15.390870+0000","last_fullsized":"2024-04-25T16:30:15.390870+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:30.419416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390409+0000","last_change":"2024-04-25T16:30:13.999931+0000","last_active":"2024-04-25T16:30:15.390409+0000","last_peered":"2024-04-25T16:30:15.390409+0000","last_clean":"2024-04-25T16:30:15.390409+0000","last_became_active":"2024-04-25T16:30:13.997093+0000","last_became_peered":"2024-04-25T16:30:13.997093+0000","last_unstale":"2024-04-25T16:30:15.390409+0000","last_undegraded":"2024-04-25T16:30:15.390409+0000","last_fullsized":"2024-04-25T16:30:15.390409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:31.973370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:34.016840+0000","last_change":"2024-04-25T16:30:08.985721+0000","last_active":"2024-04-25T16:30:34.016840+0000","last_peered":"2024-04-25T16:30:34.016840+0000","last_clean":"2024-04-25T16:30:34.016840+0000","last_became_active":"2024-04-25T16:30:08.985097+0000","last_became_peered":"2024-04-25T16:30:08.985097+0000","last_unstale":"2024-04-25T16:30:34.016840+0000","last_undegraded":"2024-04-25T16:30:34.016840+0000","last_fullsized":"2024-04-25T16:30:34.016840+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:00.713027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391185+0000","last_change":"2024-04-25T16:30:09.992659+0000","last_active":"2024-04-25T16:30:15.391185+0000","last_peered":"2024-04-25T16:30:15.391185+0000","last_clean":"2024-04-25T16:30:15.391185+0000","last_became_active":"2024-04-25T16:30:09.985936+0000","last_became_peered":"2024-04-25T16:30:09.985936+0000","last_unstale":"2024-04-25T16:30:15.391185+0000","last_undegraded":"2024-04-25T16:30:15.391185+0000","last_fullsized":"2024-04-25T16:30:15.391185+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:10.808082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390316+0000","last_change":"2024-04-25T16:30:14.009352+0000","last_active":"2024-04-25T16:30:15.390316+0000","last_peered":"2024-04-25T16:30:15.390316+0000","last_clean":"2024-04-25T16:30:15.390316+0000","last_became_active":"2024-04-25T16:30:14.005250+0000","last_became_peered":"2024-04-25T16:30:14.005250+0000","last_unstale":"2024-04-25T16:30:15.390316+0000","last_undegraded":"2024-04-25T16:30:15.390316+0000","last_fullsized":"2024-04-25T16:30:15.390316+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:23.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174092+0000","last_change":"2024-04-25T16:30:08.990547+0000","last_active":"2024-04-25T16:30:15.174092+0000","last_peered":"2024-04-25T16:30:15.174092+0000","last_clean":"2024-04-25T16:30:15.174092+0000","last_became_active":"2024-04-25T16:30:08.989927+0000","last_became_peered":"2024-04-25T16:30:08.989927+0000","last_unstale":"2024-04-25T16:30:15.174092+0000","last_undegraded":"2024-04-25T16:30:15.174092+0000","last_fullsized":"2024-04-25T16:30:15.174092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:05.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391842+0000","last_change":"2024-04-25T16:30:10.002459+0000","last_active":"2024-04-25T16:30:15.391842+0000","last_peered":"2024-04-25T16:30:15.391842+0000","last_clean":"2024-04-25T16:30:15.391842+0000","last_became_active":"2024-04-25T16:30:10.002005+0000","last_became_peered":"2024-04-25T16:30:10.002005+0000","last_unstale":"2024-04-25T16:30:15.391842+0000","last_undegraded":"2024-04-25T16:30:15.391842+0000","last_fullsized":"2024-04-25T16:30:15.391842+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:01.220371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391637+0000","last_change":"2024-04-25T16:30:14.005800+0000","last_active":"2024-04-25T16:30:15.391637+0000","last_peered":"2024-04-25T16:30:15.391637+0000","last_clean":"2024-04-25T16:30:15.391637+0000","last_became_active":"2024-04-25T16:30:14.005153+0000","last_became_peered":"2024-04-25T16:30:14.005153+0000","last_unstale":"2024-04-25T16:30:15.391637+0000","last_undegraded":"2024-04-25T16:30:15.391637+0000","last_fullsized":"2024-04-25T16:30:15.391637+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:40.015615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"76'3","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392325+0000","last_change":"2024-04-25T16:30:08.985264+0000","last_active":"2024-04-25T16:30:15.392325+0000","last_peered":"2024-04-25T16:30:15.392325+0000","last_clean":"2024-04-25T16:30:15.392325+0000","last_became_active":"2024-04-25T16:30:08.984901+0000","last_became_peered":"2024-04-25T16:30:08.984901+0000","last_unstale":"2024-04-25T16:30:15.392325+0000","last_undegraded":"2024-04-25T16:30:15.392325+0000","last_fullsized":"2024-04-25T16:30:15.392325+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:15.271981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390914+0000","last_change":"2024-04-25T16:30:09.997002+0000","last_active":"2024-04-25T16:30:15.390914+0000","last_peered":"2024-04-25T16:30:15.390914+0000","last_clean":"2024-04-25T16:30:15.390914+0000","last_became_active":"2024-04-25T16:30:09.996636+0000","last_became_peered":"2024-04-25T16:30:09.996636+0000","last_unstale":"2024-04-25T16:30:15.390914+0000","last_undegraded":"2024-04-25T16:30:15.390914+0000","last_fullsized":"2024-04-25T16:30:15.390914+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:32.522050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390634+0000","last_change":"2024-04-25T16:30:14.006403+0000","last_active":"2024-04-25T16:30:15.390634+0000","last_peered":"2024-04-25T16:30:15.390634+0000","last_clean":"2024-04-25T16:30:15.390634+0000","last_became_active":"2024-04-25T16:30:14.004816+0000","last_became_peered":"2024-04-25T16:30:14.004816+0000","last_unstale":"2024-04-25T16:30:15.390634+0000","last_undegraded":"2024-04-25T16:30:15.390634+0000","last_fullsized":"2024-04-25T16:30:15.390634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:51:10.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"81'12","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:34.016640+0000","last_change":"2024-04-25T16:30:08.986119+0000","last_active":"2024-04-25T16:30:34.016640+0000","last_peered":"2024-04-25T16:30:34.016640+0000","last_clean":"2024-04-25T16:30:34.016640+0000","last_became_active":"2024-04-25T16:30:08.985806+0000","last_became_peered":"2024-04-25T16:30:08.985806+0000","last_unstale":"2024-04-25T16:30:34.016640+0000","last_undegraded":"2024-04-25T16:30:34.016640+0000","last_fullsized":"2024-04-25T16:30:34.016640+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:17.125886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392139+0000","last_change":"2024-04-25T16:30:09.996891+0000","last_active":"2024-04-25T16:30:15.392139+0000","last_peered":"2024-04-25T16:30:15.392139+0000","last_clean":"2024-04-25T16:30:15.392139+0000","last_became_active":"2024-04-25T16:30:09.995851+0000","last_became_peered":"2024-04-25T16:30:09.995851+0000","last_unstale":"2024-04-25T16:30:15.392139+0000","last_undegraded":"2024-04-25T16:30:15.392139+0000","last_fullsized":"2024-04-25T16:30:15.392139+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:42.602854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392769+0000","last_change":"2024-04-25T16:30:13.998277+0000","last_active":"2024-04-25T16:30:15.392769+0000","last_peered":"2024-04-25T16:30:15.392769+0000","last_clean":"2024-04-25T16:30:15.392769+0000","last_became_active":"2024-04-25T16:30:13.997313+0000","last_became_peered":"2024-04-25T16:30:13.997313+0000","last_unstale":"2024-04-25T16:30:15.392769+0000","last_undegraded":"2024-04-25T16:30:15.392769+0000","last_fullsized":"2024-04-25T16:30:15.392769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:56.379806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390803+0000","last_change":"2024-04-25T16:30:08.986377+0000","last_active":"2024-04-25T16:30:15.390803+0000","last_peered":"2024-04-25T16:30:15.390803+0000","last_clean":"2024-04-25T16:30:15.390803+0000","last_became_active":"2024-04-25T16:30:08.985892+0000","last_became_peered":"2024-04-25T16:30:08.985892+0000","last_unstale":"2024-04-25T16:30:15.390803+0000","last_undegraded":"2024-04-25T16:30:15.390803+0000","last_fullsized":"2024-04-25T16:30:15.390803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:53.651789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010479+0000","last_change":"2024-04-25T16:30:09.996724+0000","last_active":"2024-04-25T16:30:15.010479+0000","last_peered":"2024-04-25T16:30:15.010479+0000","last_clean":"2024-04-25T16:30:15.010479+0000","last_became_active":"2024-04-25T16:30:09.996413+0000","last_became_peered":"2024-04-25T16:30:09.996413+0000","last_unstale":"2024-04-25T16:30:15.010479+0000","last_undegraded":"2024-04-25T16:30:15.010479+0000","last_fullsized":"2024-04-25T16:30:15.010479+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:38.403476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011333+0000","last_change":"2024-04-25T16:30:14.006064+0000","last_active":"2024-04-25T16:30:15.011333+0000","last_peered":"2024-04-25T16:30:15.011333+0000","last_clean":"2024-04-25T16:30:15.011333+0000","last_became_active":"2024-04-25T16:30:14.005545+0000","last_became_peered":"2024-04-25T16:30:14.005545+0000","last_unstale":"2024-04-25T16:30:15.011333+0000","last_undegraded":"2024-04-25T16:30:15.011333+0000","last_fullsized":"2024-04-25T16:30:15.011333+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:36.611008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391384+0000","last_change":"2024-04-25T16:30:08.988413+0000","last_active":"2024-04-25T16:30:15.391384+0000","last_peered":"2024-04-25T16:30:15.391384+0000","last_clean":"2024-04-25T16:30:15.391384+0000","last_became_active":"2024-04-25T16:30:08.988075+0000","last_became_peered":"2024-04-25T16:30:08.988075+0000","last_unstale":"2024-04-25T16:30:15.391384+0000","last_undegraded":"2024-04-25T16:30:15.391384+0000","last_fullsized":"2024-04-25T16:30:15.391384+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:16.635876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173926+0000","last_change":"2024-04-25T16:30:09.997697+0000","last_active":"2024-04-25T16:30:15.173926+0000","last_peered":"2024-04-25T16:30:15.173926+0000","last_clean":"2024-04-25T16:30:15.173926+0000","last_became_active":"2024-04-25T16:30:09.997086+0000","last_became_peered":"2024-04-25T16:30:09.997086+0000","last_unstale":"2024-04-25T16:30:15.173926+0000","last_undegraded":"2024-04-25T16:30:15.173926+0000","last_fullsized":"2024-04-25T16:30:15.173926+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:35.345902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391124+0000","last_change":"2024-04-25T16:30:14.010486+0000","last_active":"2024-04-25T16:30:15.391124+0000","last_peered":"2024-04-25T16:30:15.391124+0000","last_clean":"2024-04-25T16:30:15.391124+0000","last_became_active":"2024-04-25T16:30:14.009273+0000","last_became_peered":"2024-04-25T16:30:14.009273+0000","last_unstale":"2024-04-25T16:30:15.391124+0000","last_undegraded":"2024-04-25T16:30:15.391124+0000","last_fullsized":"2024-04-25T16:30:15.391124+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:11.863575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:36.034025+0000","last_change":"2024-04-25T16:30:08.986661+0000","last_active":"2024-04-25T16:30:36.034025+0000","last_peered":"2024-04-25T16:30:36.034025+0000","last_clean":"2024-04-25T16:30:36.034025+0000","last_became_active":"2024-04-25T16:30:08.986306+0000","last_became_peered":"2024-04-25T16:30:08.986306+0000","last_unstale":"2024-04-25T16:30:36.034025+0000","last_undegraded":"2024-04-25T16:30:36.034025+0000","last_fullsized":"2024-04-25T16:30:36.034025+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:44.363693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173602+0000","last_change":"2024-04-25T16:30:09.993443+0000","last_active":"2024-04-25T16:30:15.173602+0000","last_peered":"2024-04-25T16:30:15.173602+0000","last_clean":"2024-04-25T16:30:15.173602+0000","last_became_active":"2024-04-25T16:30:09.993031+0000","last_became_peered":"2024-04-25T16:30:09.993031+0000","last_unstale":"2024-04-25T16:30:15.173602+0000","last_undegraded":"2024-04-25T16:30:15.173602+0000","last_fullsized":"2024-04-25T16:30:15.173602+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:22.908713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010976+0000","last_change":"2024-04-25T16:30:14.003183+0000","last_active":"2024-04-25T16:30:15.010976+0000","last_peered":"2024-04-25T16:30:15.010976+0000","last_clean":"2024-04-25T16:30:15.010976+0000","last_became_active":"2024-04-25T16:30:14.002182+0000","last_became_peered":"2024-04-25T16:30:14.002182+0000","last_unstale":"2024-04-25T16:30:15.010976+0000","last_undegraded":"2024-04-25T16:30:15.010976+0000","last_fullsized":"2024-04-25T16:30:15.010976+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:08.616556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"76'8","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390597+0000","last_change":"2024-04-25T16:30:08.995860+0000","last_active":"2024-04-25T16:30:15.390597+0000","last_peered":"2024-04-25T16:30:15.390597+0000","last_clean":"2024-04-25T16:30:15.390597+0000","last_became_active":"2024-04-25T16:30:08.995487+0000","last_became_peered":"2024-04-25T16:30:08.995487+0000","last_unstale":"2024-04-25T16:30:15.390597+0000","last_undegraded":"2024-04-25T16:30:15.390597+0000","last_fullsized":"2024-04-25T16:30:15.390597+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:48.206502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390893+0000","last_change":"2024-04-25T16:30:09.998820+0000","last_active":"2024-04-25T16:30:15.390893+0000","last_peered":"2024-04-25T16:30:15.390893+0000","last_clean":"2024-04-25T16:30:15.390893+0000","last_became_active":"2024-04-25T16:30:09.998448+0000","last_became_peered":"2024-04-25T16:30:09.998448+0000","last_unstale":"2024-04-25T16:30:15.390893+0000","last_undegraded":"2024-04-25T16:30:15.390893+0000","last_fullsized":"2024-04-25T16:30:15.390893+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:54.361250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390750+0000","last_change":"2024-04-25T16:30:13.992379+0000","last_active":"2024-04-25T16:30:15.390750+0000","last_peered":"2024-04-25T16:30:15.390750+0000","last_clean":"2024-04-25T16:30:15.390750+0000","last_became_active":"2024-04-25T16:30:13.991940+0000","last_became_peered":"2024-04-25T16:30:13.991940+0000","last_unstale":"2024-04-25T16:30:15.390750+0000","last_undegraded":"2024-04-25T16:30:15.390750+0000","last_fullsized":"2024-04-25T16:30:15.390750+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:59.152384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"76'1","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391883+0000","last_change":"2024-04-25T16:30:08.987701+0000","last_active":"2024-04-25T16:30:15.391883+0000","last_peered":"2024-04-25T16:30:15.391883+0000","last_clean":"2024-04-25T16:30:15.391883+0000","last_became_active":"2024-04-25T16:30:08.987376+0000","last_became_peered":"2024-04-25T16:30:08.987376+0000","last_unstale":"2024-04-25T16:30:15.391883+0000","last_undegraded":"2024-04-25T16:30:15.391883+0000","last_fullsized":"2024-04-25T16:30:15.391883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:21.810100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011317+0000","last_change":"2024-04-25T16:30:09.993005+0000","last_active":"2024-04-25T16:30:15.011317+0000","last_peered":"2024-04-25T16:30:15.011317+0000","last_clean":"2024-04-25T16:30:15.011317+0000","last_became_active":"2024-04-25T16:30:09.992619+0000","last_became_peered":"2024-04-25T16:30:09.992619+0000","last_unstale":"2024-04-25T16:30:15.011317+0000","last_undegraded":"2024-04-25T16:30:15.011317+0000","last_fullsized":"2024-04-25T16:30:15.011317+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:39:40.998241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391135+0000","last_change":"2024-04-25T16:30:14.147895+0000","last_active":"2024-04-25T16:30:15.391135+0000","last_peered":"2024-04-25T16:30:15.391135+0000","last_clean":"2024-04-25T16:30:15.391135+0000","last_became_active":"2024-04-25T16:30:14.147298+0000","last_became_peered":"2024-04-25T16:30:14.147298+0000","last_unstale":"2024-04-25T16:30:15.391135+0000","last_undegraded":"2024-04-25T16:30:15.391135+0000","last_fullsized":"2024-04-25T16:30:15.391135+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:06.269587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391402+0000","last_change":"2024-04-25T16:30:08.982683+0000","last_active":"2024-04-25T16:30:15.391402+0000","last_peered":"2024-04-25T16:30:15.391402+0000","last_clean":"2024-04-25T16:30:15.391402+0000","last_became_active":"2024-04-25T16:30:08.982236+0000","last_became_peered":"2024-04-25T16:30:08.982236+0000","last_unstale":"2024-04-25T16:30:15.391402+0000","last_undegraded":"2024-04-25T16:30:15.391402+0000","last_fullsized":"2024-04-25T16:30:15.391402+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:48.781791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.015701+0000","last_change":"2024-04-25T16:30:09.986734+0000","last_active":"2024-04-25T16:30:15.015701+0000","last_peered":"2024-04-25T16:30:15.015701+0000","last_clean":"2024-04-25T16:30:15.015701+0000","last_became_active":"2024-04-25T16:30:09.986337+0000","last_became_peered":"2024-04-25T16:30:09.986337+0000","last_unstale":"2024-04-25T16:30:15.015701+0000","last_undegraded":"2024-04-25T16:30:15.015701+0000","last_fullsized":"2024-04-25T16:30:15.015701+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.678009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010890+0000","last_change":"2024-04-25T16:30:14.003324+0000","last_active":"2024-04-25T16:30:15.010890+0000","last_peered":"2024-04-25T16:30:15.010890+0000","last_clean":"2024-04-25T16:30:15.010890+0000","last_became_active":"2024-04-25T16:30:14.002904+0000","last_became_peered":"2024-04-25T16:30:14.002904+0000","last_unstale":"2024-04-25T16:30:15.010890+0000","last_undegraded":"2024-04-25T16:30:15.010890+0000","last_fullsized":"2024-04-25T16:30:15.010890+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:28.750102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173660+0000","last_change":"2024-04-25T16:30:08.993469+0000","last_active":"2024-04-25T16:30:15.173660+0000","last_peered":"2024-04-25T16:30:15.173660+0000","last_clean":"2024-04-25T16:30:15.173660+0000","last_became_active":"2024-04-25T16:30:08.993126+0000","last_became_peered":"2024-04-25T16:30:08.993126+0000","last_unstale":"2024-04-25T16:30:15.173660+0000","last_undegraded":"2024-04-25T16:30:15.173660+0000","last_fullsized":"2024-04-25T16:30:15.173660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:10.274645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.015338+0000","last_change":"2024-04-25T16:30:10.000597+0000","last_active":"2024-04-25T16:30:15.015338+0000","last_peered":"2024-04-25T16:30:15.015338+0000","last_clean":"2024-04-25T16:30:15.015338+0000","last_became_active":"2024-04-25T16:30:09.998892+0000","last_became_peered":"2024-04-25T16:30:09.998892+0000","last_unstale":"2024-04-25T16:30:15.015338+0000","last_undegraded":"2024-04-25T16:30:15.015338+0000","last_fullsized":"2024-04-25T16:30:15.015338+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:09.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010682+0000","last_change":"2024-04-25T16:30:14.147240+0000","last_active":"2024-04-25T16:30:15.010682+0000","last_peered":"2024-04-25T16:30:15.010682+0000","last_clean":"2024-04-25T16:30:15.010682+0000","last_became_active":"2024-04-25T16:30:14.146793+0000","last_became_peered":"2024-04-25T16:30:14.146793+0000","last_unstale":"2024-04-25T16:30:15.010682+0000","last_undegraded":"2024-04-25T16:30:15.010682+0000","last_fullsized":"2024-04-25T16:30:15.010682+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:55.713386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390593+0000","last_change":"2024-04-25T16:30:09.994595+0000","last_active":"2024-04-25T16:30:15.390593+0000","last_peered":"2024-04-25T16:30:15.390593+0000","last_clean":"2024-04-25T16:30:15.390593+0000","last_became_active":"2024-04-25T16:30:09.994305+0000","last_became_peered":"2024-04-25T16:30:09.994305+0000","last_unstale":"2024-04-25T16:30:15.390593+0000","last_undegraded":"2024-04-25T16:30:15.390593+0000","last_fullsized":"2024-04-25T16:30:15.390593+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:27.839500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010206+0000","last_change":"2024-04-25T16:30:08.984203+0000","last_active":"2024-04-25T16:30:15.010206+0000","last_peered":"2024-04-25T16:30:15.010206+0000","last_clean":"2024-04-25T16:30:15.010206+0000","last_became_active":"2024-04-25T16:30:08.982883+0000","last_became_peered":"2024-04-25T16:30:08.982883+0000","last_unstale":"2024-04-25T16:30:15.010206+0000","last_undegraded":"2024-04-25T16:30:15.010206+0000","last_fullsized":"2024-04-25T16:30:15.010206+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:18.447624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174145+0000","last_change":"2024-04-25T16:30:14.007676+0000","last_active":"2024-04-25T16:30:15.174145+0000","last_peered":"2024-04-25T16:30:15.174145+0000","last_clean":"2024-04-25T16:30:15.174145+0000","last_became_active":"2024-04-25T16:30:14.007124+0000","last_became_peered":"2024-04-25T16:30:14.007124+0000","last_unstale":"2024-04-25T16:30:15.174145+0000","last_undegraded":"2024-04-25T16:30:15.174145+0000","last_fullsized":"2024-04-25T16:30:15.174145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:54.142339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392147+0000","last_change":"2024-04-25T16:30:09.990998+0000","last_active":"2024-04-25T16:30:15.392147+0000","last_peered":"2024-04-25T16:30:15.392147+0000","last_clean":"2024-04-25T16:30:15.392147+0000","last_became_active":"2024-04-25T16:30:09.990644+0000","last_became_peered":"2024-04-25T16:30:09.990644+0000","last_unstale":"2024-04-25T16:30:15.392147+0000","last_undegraded":"2024-04-25T16:30:15.392147+0000","last_fullsized":"2024-04-25T16:30:15.392147+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:42.175520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:36.034450+0000","last_change":"2024-04-25T16:30:08.988315+0000","last_active":"2024-04-25T16:30:36.034450+0000","last_peered":"2024-04-25T16:30:36.034450+0000","last_clean":"2024-04-25T16:30:36.034450+0000","last_became_active":"2024-04-25T16:30:08.987930+0000","last_became_peered":"2024-04-25T16:30:08.987930+0000","last_unstale":"2024-04-25T16:30:36.034450+0000","last_undegraded":"2024-04-25T16:30:36.034450+0000","last_fullsized":"2024-04-25T16:30:36.034450+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:44.884602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":97,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173409+0000","last_change":"2024-04-25T16:28:58.219510+0000","last_active":"2024-04-25T16:30:15.173409+0000","last_peered":"2024-04-25T16:30:15.173409+0000","last_clean":"2024-04-25T16:30:15.173409+0000","last_became_active":"2024-04-25T16:28:58.213888+0000","last_became_peered":"2024-04-25T16:28:58.213888+0000","last_unstale":"2024-04-25T16:30:15.173409+0000","last_undegraded":"2024-04-25T16:30:15.173409+0000","last_fullsized":"2024-04-25T16:30:15.173409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_clean_scrub_stamp":"2024-04-25T16:25:33.939186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:15.378406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392417+0000","last_change":"2024-04-25T16:30:14.149020+0000","last_active":"2024-04-25T16:30:15.392417+0000","last_peered":"2024-04-25T16:30:15.392417+0000","last_clean":"2024-04-25T16:30:15.392417+0000","last_became_active":"2024-04-25T16:30:14.148456+0000","last_became_peered":"2024-04-25T16:30:14.148456+0000","last_unstale":"2024-04-25T16:30:15.392417+0000","last_undegraded":"2024-04-25T16:30:15.392417+0000","last_fullsized":"2024-04-25T16:30:15.392417+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:28:41.011480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010200+0000","last_change":"2024-04-25T16:30:10.005210+0000","last_active":"2024-04-25T16:30:15.010200+0000","last_peered":"2024-04-25T16:30:15.010200+0000","last_clean":"2024-04-25T16:30:15.010200+0000","last_became_active":"2024-04-25T16:30:10.004936+0000","last_became_peered":"2024-04-25T16:30:10.004936+0000","last_unstale":"2024-04-25T16:30:15.010200+0000","last_undegraded":"2024-04-25T16:30:15.010200+0000","last_fullsized":"2024-04-25T16:30:15.010200+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:53.292675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173539+0000","last_change":"2024-04-25T16:30:08.990162+0000","last_active":"2024-04-25T16:30:15.173539+0000","last_peered":"2024-04-25T16:30:15.173539+0000","last_clean":"2024-04-25T16:30:15.173539+0000","last_became_active":"2024-04-25T16:30:08.989613+0000","last_became_peered":"2024-04-25T16:30:08.989613+0000","last_unstale":"2024-04-25T16:30:15.173539+0000","last_undegraded":"2024-04-25T16:30:15.173539+0000","last_fullsized":"2024-04-25T16:30:15.173539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:41.595381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011257+0000","last_change":"2024-04-25T16:30:14.001850+0000","last_active":"2024-04-25T16:30:15.011257+0000","last_peered":"2024-04-25T16:30:15.011257+0000","last_clean":"2024-04-25T16:30:15.011257+0000","last_became_active":"2024-04-25T16:30:14.001348+0000","last_became_peered":"2024-04-25T16:30:14.001348+0000","last_unstale":"2024-04-25T16:30:15.011257+0000","last_undegraded":"2024-04-25T16:30:15.011257+0000","last_fullsized":"2024-04-25T16:30:15.011257+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:56:41.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392209+0000","last_change":"2024-04-25T16:30:08.995833+0000","last_active":"2024-04-25T16:30:15.392209+0000","last_peered":"2024-04-25T16:30:15.392209+0000","last_clean":"2024-04-25T16:30:15.392209+0000","last_became_active":"2024-04-25T16:30:08.995545+0000","last_became_peered":"2024-04-25T16:30:08.995545+0000","last_unstale":"2024-04-25T16:30:15.392209+0000","last_undegraded":"2024-04-25T16:30:15.392209+0000","last_fullsized":"2024-04-25T16:30:15.392209+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:31.452219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011120+0000","last_change":"2024-04-25T16:30:09.990225+0000","last_active":"2024-04-25T16:30:15.011120+0000","last_peered":"2024-04-25T16:30:15.011120+0000","last_clean":"2024-04-25T16:30:15.011120+0000","last_became_active":"2024-04-25T16:30:09.989859+0000","last_became_peered":"2024-04-25T16:30:09.989859+0000","last_unstale":"2024-04-25T16:30:15.011120+0000","last_undegraded":"2024-04-25T16:30:15.011120+0000","last_fullsized":"2024-04-25T16:30:15.011120+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:40.589395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390379+0000","last_change":"2024-04-25T16:30:14.147223+0000","last_active":"2024-04-25T16:30:15.390379+0000","last_peered":"2024-04-25T16:30:15.390379+0000","last_clean":"2024-04-25T16:30:15.390379+0000","last_became_active":"2024-04-25T16:30:14.146821+0000","last_became_peered":"2024-04-25T16:30:14.146821+0000","last_unstale":"2024-04-25T16:30:15.390379+0000","last_undegraded":"2024-04-25T16:30:15.390379+0000","last_fullsized":"2024-04-25T16:30:15.390379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:01.289165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"77'1","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391245+0000","last_change":"2024-04-25T16:30:08.993175+0000","last_active":"2024-04-25T16:30:15.391245+0000","last_peered":"2024-04-25T16:30:15.391245+0000","last_clean":"2024-04-25T16:30:15.391245+0000","last_became_active":"2024-04-25T16:30:08.992145+0000","last_became_peered":"2024-04-25T16:30:08.992145+0000","last_unstale":"2024-04-25T16:30:15.391245+0000","last_undegraded":"2024-04-25T16:30:15.391245+0000","last_fullsized":"2024-04-25T16:30:15.391245+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:06.951435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390722+0000","last_change":"2024-04-25T16:30:09.986088+0000","last_active":"2024-04-25T16:30:15.390722+0000","last_peered":"2024-04-25T16:30:15.390722+0000","last_clean":"2024-04-25T16:30:15.390722+0000","last_became_active":"2024-04-25T16:30:09.985256+0000","last_became_peered":"2024-04-25T16:30:09.985256+0000","last_unstale":"2024-04-25T16:30:15.390722+0000","last_undegraded":"2024-04-25T16:30:15.390722+0000","last_fullsized":"2024-04-25T16:30:15.390722+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:29.197558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391394+0000","last_change":"2024-04-25T16:30:14.000699+0000","last_active":"2024-04-25T16:30:15.391394+0000","last_peered":"2024-04-25T16:30:15.391394+0000","last_clean":"2024-04-25T16:30:15.391394+0000","last_became_active":"2024-04-25T16:30:14.000255+0000","last_became_peered":"2024-04-25T16:30:14.000255+0000","last_unstale":"2024-04-25T16:30:15.391394+0000","last_undegraded":"2024-04-25T16:30:15.391394+0000","last_fullsized":"2024-04-25T16:30:15.391394+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:33.436693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:37.042310+0000","last_change":"2024-04-25T16:30:08.981926+0000","last_active":"2024-04-25T16:30:37.042310+0000","last_peered":"2024-04-25T16:30:37.042310+0000","last_clean":"2024-04-25T16:30:37.042310+0000","last_became_active":"2024-04-25T16:30:08.981590+0000","last_became_peered":"2024-04-25T16:30:08.981590+0000","last_unstale":"2024-04-25T16:30:37.042310+0000","last_undegraded":"2024-04-25T16:30:37.042310+0000","last_fullsized":"2024-04-25T16:30:37.042310+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:57.818774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391395+0000","last_change":"2024-04-25T16:30:10.003998+0000","last_active":"2024-04-25T16:30:15.391395+0000","last_peered":"2024-04-25T16:30:15.391395+0000","last_clean":"2024-04-25T16:30:15.391395+0000","last_became_active":"2024-04-25T16:30:10.003626+0000","last_became_peered":"2024-04-25T16:30:10.003626+0000","last_unstale":"2024-04-25T16:30:15.391395+0000","last_undegraded":"2024-04-25T16:30:15.391395+0000","last_fullsized":"2024-04-25T16:30:15.391395+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:10.686092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391014+0000","last_change":"2024-04-25T16:30:14.149278+0000","last_active":"2024-04-25T16:30:15.391014+0000","last_peered":"2024-04-25T16:30:15.391014+0000","last_clean":"2024-04-25T16:30:15.391014+0000","last_became_active":"2024-04-25T16:30:14.148654+0000","last_became_peered":"2024-04-25T16:30:14.148654+0000","last_unstale":"2024-04-25T16:30:15.391014+0000","last_undegraded":"2024-04-25T16:30:15.391014+0000","last_fullsized":"2024-04-25T16:30:15.391014+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:24.823222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.389940+0000","last_change":"2024-04-25T16:30:08.981612+0000","last_active":"2024-04-25T16:30:15.389940+0000","last_peered":"2024-04-25T16:30:15.389940+0000","last_clean":"2024-04-25T16:30:15.389940+0000","last_became_active":"2024-04-25T16:30:08.981222+0000","last_became_peered":"2024-04-25T16:30:08.981222+0000","last_unstale":"2024-04-25T16:30:15.389940+0000","last_undegraded":"2024-04-25T16:30:15.389940+0000","last_fullsized":"2024-04-25T16:30:15.389940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:14.914658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390765+0000","last_change":"2024-04-25T16:30:09.989592+0000","last_active":"2024-04-25T16:30:15.390765+0000","last_peered":"2024-04-25T16:30:15.390765+0000","last_clean":"2024-04-25T16:30:15.390765+0000","last_became_active":"2024-04-25T16:30:09.989222+0000","last_became_peered":"2024-04-25T16:30:09.989222+0000","last_unstale":"2024-04-25T16:30:15.390765+0000","last_undegraded":"2024-04-25T16:30:15.390765+0000","last_fullsized":"2024-04-25T16:30:15.390765+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:52.403239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391835+0000","last_change":"2024-04-25T16:30:14.010480+0000","last_active":"2024-04-25T16:30:15.391835+0000","last_peered":"2024-04-25T16:30:15.391835+0000","last_clean":"2024-04-25T16:30:15.391835+0000","last_became_active":"2024-04-25T16:30:14.009023+0000","last_became_peered":"2024-04-25T16:30:14.009023+0000","last_unstale":"2024-04-25T16:30:15.391835+0000","last_undegraded":"2024-04-25T16:30:15.391835+0000","last_fullsized":"2024-04-25T16:30:15.391835+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:10.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'5","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:36.034483+0000","last_change":"2024-04-25T16:30:08.986849+0000","last_active":"2024-04-25T16:30:36.034483+0000","last_peered":"2024-04-25T16:30:36.034483+0000","last_clean":"2024-04-25T16:30:36.034483+0000","last_became_active":"2024-04-25T16:30:08.986479+0000","last_became_peered":"2024-04-25T16:30:08.986479+0000","last_unstale":"2024-04-25T16:30:36.034483+0000","last_undegraded":"2024-04-25T16:30:36.034483+0000","last_fullsized":"2024-04-25T16:30:36.034483+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010791+0000","last_change":"2024-04-25T16:30:09.990927+0000","last_active":"2024-04-25T16:30:15.010791+0000","last_peered":"2024-04-25T16:30:15.010791+0000","last_clean":"2024-04-25T16:30:15.010791+0000","last_became_active":"2024-04-25T16:30:09.990482+0000","last_became_peered":"2024-04-25T16:30:09.990482+0000","last_unstale":"2024-04-25T16:30:15.010791+0000","last_undegraded":"2024-04-25T16:30:15.010791+0000","last_fullsized":"2024-04-25T16:30:15.010791+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:18.649721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:17.915026+0000","last_change":"2024-04-25T16:30:14.006159+0000","last_active":"2024-04-25T16:30:17.915026+0000","last_peered":"2024-04-25T16:30:17.915026+0000","last_clean":"2024-04-25T16:30:17.915026+0000","last_became_active":"2024-04-25T16:30:14.005413+0000","last_became_peered":"2024-04-25T16:30:14.005413+0000","last_unstale":"2024-04-25T16:30:17.915026+0000","last_undegraded":"2024-04-25T16:30:17.915026+0000","last_fullsized":"2024-04-25T16:30:17.915026+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:52.934385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390457+0000","last_change":"2024-04-25T16:30:08.986207+0000","last_active":"2024-04-25T16:30:15.390457+0000","last_peered":"2024-04-25T16:30:15.390457+0000","last_clean":"2024-04-25T16:30:15.390457+0000","last_became_active":"2024-04-25T16:30:08.985857+0000","last_became_peered":"2024-04-25T16:30:08.985857+0000","last_unstale":"2024-04-25T16:30:15.390457+0000","last_undegraded":"2024-04-25T16:30:15.390457+0000","last_fullsized":"2024-04-25T16:30:15.390457+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:45.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390450+0000","last_change":"2024-04-25T16:30:09.993566+0000","last_active":"2024-04-25T16:30:15.390450+0000","last_peered":"2024-04-25T16:30:15.390450+0000","last_clean":"2024-04-25T16:30:15.390450+0000","last_became_active":"2024-04-25T16:30:09.992250+0000","last_became_peered":"2024-04-25T16:30:09.992250+0000","last_unstale":"2024-04-25T16:30:15.390450+0000","last_undegraded":"2024-04-25T16:30:15.390450+0000","last_fullsized":"2024-04-25T16:30:15.390450+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:33.279300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391315+0000","last_change":"2024-04-25T16:30:14.147443+0000","last_active":"2024-04-25T16:30:15.391315+0000","last_peered":"2024-04-25T16:30:15.391315+0000","last_clean":"2024-04-25T16:30:15.391315+0000","last_became_active":"2024-04-25T16:30:14.146838+0000","last_became_peered":"2024-04-25T16:30:14.146838+0000","last_unstale":"2024-04-25T16:30:15.391315+0000","last_undegraded":"2024-04-25T16:30:15.391315+0000","last_fullsized":"2024-04-25T16:30:15.391315+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:51.532685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391369+0000","last_change":"2024-04-25T16:30:08.990123+0000","last_active":"2024-04-25T16:30:15.391369+0000","last_peered":"2024-04-25T16:30:15.391369+0000","last_clean":"2024-04-25T16:30:15.391369+0000","last_became_active":"2024-04-25T16:30:08.989769+0000","last_became_peered":"2024-04-25T16:30:08.989769+0000","last_unstale":"2024-04-25T16:30:15.391369+0000","last_undegraded":"2024-04-25T16:30:15.391369+0000","last_fullsized":"2024-04-25T16:30:15.391369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:39.970292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173729+0000","last_change":"2024-04-25T16:30:09.998486+0000","last_active":"2024-04-25T16:30:15.173729+0000","last_peered":"2024-04-25T16:30:15.173729+0000","last_clean":"2024-04-25T16:30:15.173729+0000","last_became_active":"2024-04-25T16:30:09.998154+0000","last_became_peered":"2024-04-25T16:30:09.998154+0000","last_unstale":"2024-04-25T16:30:15.173729+0000","last_undegraded":"2024-04-25T16:30:15.173729+0000","last_fullsized":"2024-04-25T16:30:15.173729+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:55.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390297+0000","last_change":"2024-04-25T16:30:13.995864+0000","last_active":"2024-04-25T16:30:15.390297+0000","last_peered":"2024-04-25T16:30:15.390297+0000","last_clean":"2024-04-25T16:30:15.390297+0000","last_became_active":"2024-04-25T16:30:13.995326+0000","last_became_peered":"2024-04-25T16:30:13.995326+0000","last_unstale":"2024-04-25T16:30:15.390297+0000","last_undegraded":"2024-04-25T16:30:15.390297+0000","last_fullsized":"2024-04-25T16:30:15.390297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:11.718826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391723+0000","last_change":"2024-04-25T16:30:08.984189+0000","last_active":"2024-04-25T16:30:15.391723+0000","last_peered":"2024-04-25T16:30:15.391723+0000","last_clean":"2024-04-25T16:30:15.391723+0000","last_became_active":"2024-04-25T16:30:08.983821+0000","last_became_peered":"2024-04-25T16:30:08.983821+0000","last_unstale":"2024-04-25T16:30:15.391723+0000","last_undegraded":"2024-04-25T16:30:15.391723+0000","last_fullsized":"2024-04-25T16:30:15.391723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:51.315397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174203+0000","last_change":"2024-04-25T16:30:09.998763+0000","last_active":"2024-04-25T16:30:15.174203+0000","last_peered":"2024-04-25T16:30:15.174203+0000","last_clean":"2024-04-25T16:30:15.174203+0000","last_became_active":"2024-04-25T16:30:09.998439+0000","last_became_peered":"2024-04-25T16:30:09.998439+0000","last_unstale":"2024-04-25T16:30:15.174203+0000","last_undegraded":"2024-04-25T16:30:15.174203+0000","last_fullsized":"2024-04-25T16:30:15.174203+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:14.279883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391766+0000","last_change":"2024-04-25T16:30:14.005427+0000","last_active":"2024-04-25T16:30:15.391766+0000","last_peered":"2024-04-25T16:30:15.391766+0000","last_clean":"2024-04-25T16:30:15.391766+0000","last_became_active":"2024-04-25T16:30:14.005006+0000","last_became_peered":"2024-04-25T16:30:14.005006+0000","last_unstale":"2024-04-25T16:30:15.391766+0000","last_undegraded":"2024-04-25T16:30:15.391766+0000","last_fullsized":"2024-04-25T16:30:15.391766+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:49.563365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"76'1","reported_seq":24,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.392462+0000","last_change":"2024-04-25T16:30:08.985557+0000","last_active":"2024-04-25T16:30:15.392462+0000","last_peered":"2024-04-25T16:30:15.392462+0000","last_clean":"2024-04-25T16:30:15.392462+0000","last_became_active":"2024-04-25T16:30:08.985215+0000","last_became_peered":"2024-04-25T16:30:08.985215+0000","last_unstale":"2024-04-25T16:30:15.392462+0000","last_undegraded":"2024-04-25T16:30:15.392462+0000","last_fullsized":"2024-04-25T16:30:15.392462+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:51.504089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.173893+0000","last_change":"2024-04-25T16:30:09.991336+0000","last_active":"2024-04-25T16:30:15.173893+0000","last_peered":"2024-04-25T16:30:15.173893+0000","last_clean":"2024-04-25T16:30:15.173893+0000","last_became_active":"2024-04-25T16:30:09.990819+0000","last_became_peered":"2024-04-25T16:30:09.990819+0000","last_unstale":"2024-04-25T16:30:15.173893+0000","last_undegraded":"2024-04-25T16:30:15.173893+0000","last_fullsized":"2024-04-25T16:30:15.173893+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:45.988061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390125+0000","last_change":"2024-04-25T16:30:13.998463+0000","last_active":"2024-04-25T16:30:15.390125+0000","last_peered":"2024-04-25T16:30:15.390125+0000","last_clean":"2024-04-25T16:30:15.390125+0000","last_became_active":"2024-04-25T16:30:13.997952+0000","last_became_peered":"2024-04-25T16:30:13.997952+0000","last_unstale":"2024-04-25T16:30:15.390125+0000","last_undegraded":"2024-04-25T16:30:15.390125+0000","last_fullsized":"2024-04-25T16:30:15.390125+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:58.507942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"76'2","reported_seq":25,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010889+0000","last_change":"2024-04-25T16:30:08.987232+0000","last_active":"2024-04-25T16:30:15.010889+0000","last_peered":"2024-04-25T16:30:15.010889+0000","last_clean":"2024-04-25T16:30:15.010889+0000","last_became_active":"2024-04-25T16:30:08.986873+0000","last_became_peered":"2024-04-25T16:30:08.986873+0000","last_unstale":"2024-04-25T16:30:15.010889+0000","last_undegraded":"2024-04-25T16:30:15.010889+0000","last_fullsized":"2024-04-25T16:30:15.010889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:40.746479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.174197+0000","last_change":"2024-04-25T16:30:09.988566+0000","last_active":"2024-04-25T16:30:15.174197+0000","last_peered":"2024-04-25T16:30:15.174197+0000","last_clean":"2024-04-25T16:30:15.174197+0000","last_became_active":"2024-04-25T16:30:09.988164+0000","last_became_peered":"2024-04-25T16:30:09.988164+0000","last_unstale":"2024-04-25T16:30:15.174197+0000","last_undegraded":"2024-04-25T16:30:15.174197+0000","last_fullsized":"2024-04-25T16:30:15.174197+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:40.476766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011052+0000","last_change":"2024-04-25T16:30:09.996349+0000","last_active":"2024-04-25T16:30:15.011052+0000","last_peered":"2024-04-25T16:30:15.011052+0000","last_clean":"2024-04-25T16:30:15.011052+0000","last_became_active":"2024-04-25T16:30:09.994974+0000","last_became_peered":"2024-04-25T16:30:09.994974+0000","last_unstale":"2024-04-25T16:30:15.011052+0000","last_undegraded":"2024-04-25T16:30:15.011052+0000","last_fullsized":"2024-04-25T16:30:15.011052+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:46.929057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391959+0000","last_change":"2024-04-25T16:30:09.997837+0000","last_active":"2024-04-25T16:30:15.391959+0000","last_peered":"2024-04-25T16:30:15.391959+0000","last_clean":"2024-04-25T16:30:15.391959+0000","last_became_active":"2024-04-25T16:30:09.997361+0000","last_became_peered":"2024-04-25T16:30:09.997361+0000","last_unstale":"2024-04-25T16:30:15.391959+0000","last_undegraded":"2024-04-25T16:30:15.391959+0000","last_fullsized":"2024-04-25T16:30:15.391959+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:35.698528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390589+0000","last_change":"2024-04-25T16:30:09.991794+0000","last_active":"2024-04-25T16:30:15.390589+0000","last_peered":"2024-04-25T16:30:15.390589+0000","last_clean":"2024-04-25T16:30:15.390589+0000","last_became_active":"2024-04-25T16:30:09.991440+0000","last_became_peered":"2024-04-25T16:30:09.991440+0000","last_unstale":"2024-04-25T16:30:15.390589+0000","last_undegraded":"2024-04-25T16:30:15.390589+0000","last_fullsized":"2024-04-25T16:30:15.390589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:11.322254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391942+0000","last_change":"2024-04-25T16:30:10.002364+0000","last_active":"2024-04-25T16:30:15.391942+0000","last_peered":"2024-04-25T16:30:15.391942+0000","last_clean":"2024-04-25T16:30:15.391942+0000","last_became_active":"2024-04-25T16:30:10.001957+0000","last_became_peered":"2024-04-25T16:30:10.001957+0000","last_unstale":"2024-04-25T16:30:15.391942+0000","last_undegraded":"2024-04-25T16:30:15.391942+0000","last_fullsized":"2024-04-25T16:30:15.391942+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:19.150175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011117+0000","last_change":"2024-04-25T16:30:10.004150+0000","last_active":"2024-04-25T16:30:15.011117+0000","last_peered":"2024-04-25T16:30:15.011117+0000","last_clean":"2024-04-25T16:30:15.011117+0000","last_became_active":"2024-04-25T16:30:10.003733+0000","last_became_peered":"2024-04-25T16:30:10.003733+0000","last_unstale":"2024-04-25T16:30:15.011117+0000","last_undegraded":"2024-04-25T16:30:15.011117+0000","last_fullsized":"2024-04-25T16:30:15.011117+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:11.850366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011362+0000","last_change":"2024-04-25T16:30:09.991056+0000","last_active":"2024-04-25T16:30:15.011362+0000","last_peered":"2024-04-25T16:30:15.011362+0000","last_clean":"2024-04-25T16:30:15.011362+0000","last_became_active":"2024-04-25T16:30:09.988708+0000","last_became_peered":"2024-04-25T16:30:09.988708+0000","last_unstale":"2024-04-25T16:30:15.011362+0000","last_undegraded":"2024-04-25T16:30:15.011362+0000","last_fullsized":"2024-04-25T16:30:15.011362+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:13.062269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390938+0000","last_change":"2024-04-25T16:30:09.996571+0000","last_active":"2024-04-25T16:30:15.390938+0000","last_peered":"2024-04-25T16:30:15.390938+0000","last_clean":"2024-04-25T16:30:15.390938+0000","last_became_active":"2024-04-25T16:30:09.996206+0000","last_became_peered":"2024-04-25T16:30:09.996206+0000","last_unstale":"2024-04-25T16:30:15.390938+0000","last_undegraded":"2024-04-25T16:30:15.390938+0000","last_fullsized":"2024-04-25T16:30:15.390938+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:07.107949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390912+0000","last_change":"2024-04-25T16:30:09.995067+0000","last_active":"2024-04-25T16:30:15.390912+0000","last_peered":"2024-04-25T16:30:15.390912+0000","last_clean":"2024-04-25T16:30:15.390912+0000","last_became_active":"2024-04-25T16:30:09.994678+0000","last_became_peered":"2024-04-25T16:30:09.994678+0000","last_unstale":"2024-04-25T16:30:15.390912+0000","last_undegraded":"2024-04-25T16:30:15.390912+0000","last_fullsized":"2024-04-25T16:30:15.390912+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:09.000869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390478+0000","last_change":"2024-04-25T16:30:09.996821+0000","last_active":"2024-04-25T16:30:15.390478+0000","last_peered":"2024-04-25T16:30:15.390478+0000","last_clean":"2024-04-25T16:30:15.390478+0000","last_became_active":"2024-04-25T16:30:09.996464+0000","last_became_peered":"2024-04-25T16:30:09.996464+0000","last_unstale":"2024-04-25T16:30:15.390478+0000","last_undegraded":"2024-04-25T16:30:15.390478+0000","last_fullsized":"2024-04-25T16:30:15.390478+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:54.961544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.391106+0000","last_change":"2024-04-25T16:30:09.991733+0000","last_active":"2024-04-25T16:30:15.391106+0000","last_peered":"2024-04-25T16:30:15.391106+0000","last_clean":"2024-04-25T16:30:15.391106+0000","last_became_active":"2024-04-25T16:30:09.991401+0000","last_became_peered":"2024-04-25T16:30:09.991401+0000","last_unstale":"2024-04-25T16:30:15.391106+0000","last_undegraded":"2024-04-25T16:30:15.391106+0000","last_fullsized":"2024-04-25T16:30:15.391106+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:14.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.390619+0000","last_change":"2024-04-25T16:30:09.999087+0000","last_active":"2024-04-25T16:30:15.390619+0000","last_peered":"2024-04-25T16:30:15.390619+0000","last_clean":"2024-04-25T16:30:15.390619+0000","last_became_active":"2024-04-25T16:30:09.998725+0000","last_became_peered":"2024-04-25T16:30:09.998725+0000","last_unstale":"2024-04-25T16:30:15.390619+0000","last_undegraded":"2024-04-25T16:30:15.390619+0000","last_fullsized":"2024-04-25T16:30:15.390619+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:19.155171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.010744+0000","last_change":"2024-04-25T16:30:09.995880+0000","last_active":"2024-04-25T16:30:15.010744+0000","last_peered":"2024-04-25T16:30:15.010744+0000","last_clean":"2024-04-25T16:30:15.010744+0000","last_became_active":"2024-04-25T16:30:09.995514+0000","last_became_peered":"2024-04-25T16:30:09.995514+0000","last_unstale":"2024-04-25T16:30:15.010744+0000","last_undegraded":"2024-04-25T16:30:15.010744+0000","last_fullsized":"2024-04-25T16:30:15.010744+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-25T16:30:15.011017+0000","last_change":"2024-04-25T16:30:09.998326+0000","last_active":"2024-04-25T16:30:15.011017+0000","last_peered":"2024-04-25T16:30:15.011017+0000","last_clean":"2024-04-25T16:30:15.011017+0000","last_became_active":"2024-04-25T16:30:09.995803+0000","last_became_peered":"2024-04-25T16:30:09.995803+0000","last_unstale":"2024-04-25T16:30:15.011017+0000","last_undegraded":"2024-04-25T16:30:15.011017+0000","last_fullsized":"2024-04-25T16:30:15.011017+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:46.983947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5979,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":94,"num_write_kb":33,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":18799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645330,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":433354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841559,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36836,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708316,"statfs":{"total":95995035648,"available":95957315584,"internally_reserved":0,"allocated":1536000,"data_stored":1028074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037787,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":892928,"data_stored":432744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234016,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":897024,"data_stored":432766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430245,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28472,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716680,"statfs":{"total":95995035648,"available":95965880320,"internally_reserved":0,"allocated":1486848,"data_stored":1023112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593771,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":888832,"data_stored":432162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":901120,"data_stored":436004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36216,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708936,"statfs":{"total":95995035648,"available":95957950464,"internally_reserved":0,"allocated":901120,"data_stored":432856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":436038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":913408,"data_stored":432570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953526,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36224,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708928,"statfs":{"total":95995035648,"available":95957942272,"internally_reserved":0,"allocated":909312,"data_stored":433253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790000,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36800,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708352,"statfs":{"total":95995035648,"available":95957352448,"internally_reserved":0,"allocated":1495040,"data_stored":1022712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:30:39.192 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T16:30:39.192 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T16:30:39.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T16:30:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:39 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] up:active 2024-04-25T16:30:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:39 smithi203 ceph-mon[33115]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-25T16:30:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:39 smithi203 ceph-mon[33115]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T16:30:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:39 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:30:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:39 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1248976264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:39 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1018412835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:39 smithi203 ceph-mon[33115]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:39.466 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:39.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] up:active 2024-04-25T16:30:39.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-mon[32556]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-25T16:30:39.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-mon[32556]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T16:30:39.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:30:39.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1248976264' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1018412835' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:39.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-mon[32556]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:40.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:40 smithi098 ceph-mon[29162]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:40.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:40 smithi098 ceph-mon[29162]: pgmap v297: 129 pgs: 129 active+clean; 588 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4.5 KiB/s wr, 14 op/s 2024-04-25T16:30:40.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:40 smithi173 ceph-mon[32556]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:40.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:40 smithi173 ceph-mon[32556]: pgmap v297: 129 pgs: 129 active+clean; 588 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4.5 KiB/s wr, 14 op/s 2024-04-25T16:30:40.265 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:30:39.955+0000 7f8d10d3c700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T16:30:40.265 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:30:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:30:39.955+0000 7f8d10d3c700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:30:40.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:40 smithi203 ceph-mon[33115]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:30:40.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:40 smithi203 ceph-mon[33115]: pgmap v297: 129 pgs: 129 active+clean; 588 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 4.5 KiB/s wr, 14 op/s 2024-04-25T16:30:41.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 -- 172.21.15.98:0/4273742138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c109fa0 msgr2=0x7fe94c10c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 --2- 172.21.15.98:0/4273742138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c109fa0 0x7fe94c10c3c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe94400b0a0 tx=0x7fe94400b3b0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 -- 172.21.15.98:0/4273742138 shutdown_connections 2024-04-25T16:30:41.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 --2- 172.21.15.98:0/4273742138 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe94c104ce0 0x7fe94c107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 --2- 172.21.15.98:0/4273742138 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c109fa0 0x7fe94c10c3c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 --2- 172.21.15.98:0/4273742138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94c107640 0x7fe94c109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 -- 172.21.15.98:0/4273742138 >> 172.21.15.98:0/4273742138 conn(0x7fe94c0fe740 msgr2=0x7fe94c100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 -- 172.21.15.98:0/4273742138 shutdown_connections 2024-04-25T16:30:41.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.194+0000 7fe951b4e700 1 -- 172.21.15.98:0/4273742138 wait complete. 2024-04-25T16:30:41.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.195+0000 7fe951b4e700 1 Processor -- start 2024-04-25T16:30:41.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.195+0000 7fe951b4e700 1 -- start start 2024-04-25T16:30:41.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.195+0000 7fe951b4e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe94c104ce0 0x7fe94c19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.195+0000 7fe951b4e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c107640 0x7fe94c19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.195+0000 7fe951b4e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94c109fa0 0x7fe94c1a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.195+0000 7fe951b4e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe94c10fb20 con 0x7fe94c107640 2024-04-25T16:30:41.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.195+0000 7fe951b4e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe94c10f9a0 con 0x7fe94c109fa0 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.195+0000 7fe951b4e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe94c10fca0 con 0x7fe94c104ce0 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c107640 0x7fe94c19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c107640 0x7fe94c19aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37128/0 (socket says 172.21.15.98:37128) 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 -- 172.21.15.98:0/3973535422 learned_addr learned my addr 172.21.15.98:0/3973535422 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94b7fe700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe94c104ce0 0x7fe94c19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94bfff700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94c109fa0 0x7fe94c1a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 -- 172.21.15.98:0/3973535422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe94c104ce0 msgr2=0x7fe94c19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe94c104ce0 0x7fe94c19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 -- 172.21.15.98:0/3973535422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94c109fa0 msgr2=0x7fe94c1a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94c109fa0 0x7fe94c1a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 -- 172.21.15.98:0/3973535422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe94c102ad0 con 0x7fe94c107640 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe94affd700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c107640 0x7fe94c19aba0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe94000d950 tx=0x7fe94000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe948ff9700 1 -- 172.21.15.98:0/3973535422 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe940016070 con 0x7fe94c107640 2024-04-25T16:30:41.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe948ff9700 1 -- 172.21.15.98:0/3973535422 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe94000fd20 con 0x7fe94c107640 2024-04-25T16:30:41.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe94c102dc0 con 0x7fe94c107640 2024-04-25T16:30:41.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe948ff9700 1 -- 172.21.15.98:0/3973535422 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe94000e3b0 con 0x7fe94c107640 2024-04-25T16:30:41.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.196+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe94c103300 con 0x7fe94c107640 2024-04-25T16:30:41.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.197+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe94c051190 con 0x7fe94c107640 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.199+0000 7fe948ff9700 1 -- 172.21.15.98:0/3973535422 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fe940019070 con 0x7fe94c107640 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.200+0000 7fe948ff9700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9340afc10 0x7fe9340b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.200+0000 7fe94b7fe700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9340afc10 0x7fe9340b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.200+0000 7fe948ff9700 1 -- 172.21.15.98:0/3973535422 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fe9400deac0 con 0x7fe94c107640 2024-04-25T16:30:41.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.200+0000 7fe94b7fe700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9340afc10 0x7fe9340b2120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe94c06bf00 tx=0x7fe93c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:41.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.204+0000 7fe948ff9700 1 -- 172.21.15.98:0/3973535422 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9400a7300 con 0x7fe94c107640 2024-04-25T16:30:41.489 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.489+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fe94c1046e0 con 0x7fe94c107640 2024-04-25T16:30:41.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.492+0000 7fe948ff9700 1 -- 172.21.15.98:0/3973535422 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fe9400a6c00 con 0x7fe94c107640 2024-04-25T16:30:41.492 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-04-25T16:30:41.492 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQAxhSpmmpdKHRAAD5H3HWhSFcVMVeCYNcpG/Q== 2024-04-25T16:30:41.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.494+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9340afc10 msgr2=0x7fe9340b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.494+0000 7fe951b4e700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9340afc10 0x7fe9340b2120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe94c06bf00 tx=0x7fe93c00b3c0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.494+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c107640 msgr2=0x7fe94c19aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.494+0000 7fe951b4e700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c107640 0x7fe94c19aba0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe94000d950 tx=0x7fe94000dc60 comp rx=0 tx=0).stop 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.495+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 shutdown_connections 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.495+0000 7fe951b4e700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9340afc10 0x7fe9340b2120 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.495+0000 7fe951b4e700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe94c104ce0 0x7fe94c19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.495+0000 7fe951b4e700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe94c109fa0 0x7fe94c1a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.495+0000 7fe951b4e700 1 --2- 172.21.15.98:0/3973535422 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe94c107640 0x7fe94c19aba0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.495+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 >> 172.21.15.98:0/3973535422 conn(0x7fe94c0fe740 msgr2=0x7fe94c1057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.495+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 shutdown_connections 2024-04-25T16:30:41.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:41.495+0000 7fe951b4e700 1 -- 172.21.15.98:0/3973535422 wait complete. 2024-04-25T16:30:41.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:41 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3973535422' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T16:30:41.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:41 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3973535422' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T16:30:41.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:41 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3973535422' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T16:30:41.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:41 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3973535422' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T16:30:42.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:41 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3973535422' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T16:30:42.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:41 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3973535422' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T16:30:42.293 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:30:42.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T16:30:42.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T16:30:42.331 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T16:30:42.332 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T16:30:42.332 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T16:30:42.608 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:42.854 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:42 smithi098 ceph-mon[29162]: pgmap v298: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2024-04-25T16:30:42.938 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:42 smithi203 ceph-mon[33115]: pgmap v298: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2024-04-25T16:30:43.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:42 smithi173 ceph-mon[32556]: pgmap v298: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 8.3 KiB/s wr, 17 op/s 2024-04-25T16:30:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.313+0000 7fa4fafa0700 1 -- 172.21.15.98:0/2660242476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4107cf0 msgr2=0x7fa4f4108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.313+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/2660242476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4107cf0 0x7fa4f4108100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc0099c0 tx=0x7fa4dc009cd0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.313+0000 7fa4fafa0700 1 -- 172.21.15.98:0/2660242476 shutdown_connections 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.313+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/2660242476 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4f4103410 0x7fa4f4103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.313+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/2660242476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4108640 0x7fa4f410ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.313+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/2660242476 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4107cf0 0x7fa4f4108100 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.313+0000 7fa4fafa0700 1 -- 172.21.15.98:0/2660242476 >> 172.21.15.98:0/2660242476 conn(0x7fa4f4078d20 msgr2=0x7fa4f4079160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.313+0000 7fa4fafa0700 1 -- 172.21.15.98:0/2660242476 shutdown_connections 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 -- 172.21.15.98:0/2660242476 wait complete. 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 Processor -- start 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 -- start start 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4103410 0x7fa4f419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4f4107cf0 0x7fa4f419fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:44.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4108640 0x7fa4f4199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f41131f0 con 0x7fa4f4108640 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f4113070 con 0x7fa4f4103410 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.314+0000 7fa4fafa0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa4f4113370 con 0x7fa4f4107cf0 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4108640 0x7fa4f4199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4108640 0x7fa4f4199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37140/0 (socket says 172.21.15.98:37140) 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 -- 172.21.15.98:0/370380068 learned_addr learned my addr 172.21.15.98:0/370380068 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f3fff700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4f4107cf0 0x7fa4f419fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f8d3c700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4103410 0x7fa4f419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 -- 172.21.15.98:0/370380068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4f4107cf0 msgr2=0x7fa4f419fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4f4107cf0 0x7fa4f419fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 -- 172.21.15.98:0/370380068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4103410 msgr2=0x7fa4f419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4103410 0x7fa4f419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 -- 172.21.15.98:0/370380068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4f419a1b0 con 0x7fa4f4108640 2024-04-25T16:30:44.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f8d3c700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4103410 0x7fa4f419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:44.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f953d700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4108640 0x7fa4f4199ad0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d960 tx=0x7fa4e400dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:44.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f1ffb700 1 -- 172.21.15.98:0/370380068 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e401b070 con 0x7fa4f4108640 2024-04-25T16:30:44.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f1ffb700 1 -- 172.21.15.98:0/370380068 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa4e400ec50 con 0x7fa4f4108640 2024-04-25T16:30:44.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4f1ffb700 1 -- 172.21.15.98:0/370380068 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4e40145a0 con 0x7fa4f4108640 2024-04-25T16:30:44.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.315+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4f419a4a0 con 0x7fa4f4108640 2024-04-25T16:30:44.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.316+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4f40758c0 con 0x7fa4f4108640 2024-04-25T16:30:44.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.317+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4f4104870 con 0x7fa4f4108640 2024-04-25T16:30:44.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.318+0000 7fa4f1ffb700 1 -- 172.21.15.98:0/370380068 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa4e401e070 con 0x7fa4f4108640 2024-04-25T16:30:44.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.319+0000 7fa4f1ffb700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa4e00afc90 0x7fa4e00b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:44.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.319+0000 7fa4f8d3c700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa4e00afc90 0x7fa4e00b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:44.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.319+0000 7fa4f1ffb700 1 -- 172.21.15.98:0/370380068 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fa4e40e0cf0 con 0x7fa4f4108640 2024-04-25T16:30:44.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.319+0000 7fa4f8d3c700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa4e00afc90 0x7fa4e00b21a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa4f419ac10 tx=0x7fa4ec009570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:44.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.322+0000 7fa4f1ffb700 1 -- 172.21.15.98:0/370380068 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa4e4025070 con 0x7fa4f4108640 2024-04-25T16:30:44.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.580+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa4f4051190 con 0x7fa4f4108640 2024-04-25T16:30:44.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.586+0000 7fa4f1ffb700 1 -- 172.21.15.98:0/370380068 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+284478 (secure 0 0 0) 0x7fa4e40a9670 con 0x7fa4f4108640 2024-04-25T16:30:44.590 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:44.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa4e00afc90 msgr2=0x7fa4e00b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa4e00afc90 0x7fa4e00b21a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa4f419ac10 tx=0x7fa4ec009570 comp rx=0 tx=0).stop 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4108640 msgr2=0x7fa4f4199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4108640 0x7fa4f4199ad0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d960 tx=0x7fa4e400dd20 comp rx=0 tx=0).stop 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 shutdown_connections 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa4e00afc90 0x7fa4e00b21a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa4f4108640 0x7fa4f4199ad0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4f4103410 0x7fa4f419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 --2- 172.21.15.98:0/370380068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4f4107cf0 0x7fa4f419fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 >> 172.21.15.98:0/370380068 conn(0x7fa4f4078d20 msgr2=0x7fa4f41008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.589+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 shutdown_connections 2024-04-25T16:30:44.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:44.590+0000 7fa4fafa0700 1 -- 172.21.15.98:0/370380068 wait complete. 2024-04-25T16:30:45.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:44 smithi098 ceph-mon[29162]: pgmap v299: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.3 KiB/s wr, 21 op/s 2024-04-25T16:30:45.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:44 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/370380068' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T16:30:45.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:44 smithi173 ceph-mon[32556]: pgmap v299: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.3 KiB/s wr, 21 op/s 2024-04-25T16:30:45.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:44 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/370380068' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T16:30:45.343 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":4283487583},{"type":"v1","addr":"172.21.15.98:6801","nonce":4283487583}]},"active_addr":"172.21.15.98:6801/4283487583","active_change":"2024-04-25T16:23:07.830395+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1611661772}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3811231551}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":79813565}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2544586759}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1089426072}]}]} 2024-04-25T16:30:45.349 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T16:30:45.349 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T16:30:45.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T16:30:45.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:44 smithi203 ceph-mon[33115]: pgmap v299: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 8.3 KiB/s wr, 21 op/s 2024-04-25T16:30:45.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:44 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/370380068' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T16:30:45.587 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:47.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:46 smithi098 ceph-mon[29162]: pgmap v300: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-04-25T16:30:47.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:46 smithi173 ceph-mon[32556]: pgmap v300: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-04-25T16:30:47.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.273+0000 7ff7d84c9700 1 -- 172.21.15.98:0/1792527218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0103360 msgr2=0x7ff7d0103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.273+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/1792527218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0103360 0x7ff7d0103770 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00099c0 tx=0x7ff7c0009cd0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 -- 172.21.15.98:0/1792527218 shutdown_connections 2024-04-25T16:30:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/1792527218 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7d01085e0 0x7ff7d010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/1792527218 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0103360 0x7ff7d0103770 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/1792527218 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7d0107c90 0x7ff7d01080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 -- 172.21.15.98:0/1792527218 >> 172.21.15.98:0/1792527218 conn(0x7ff7d00fec90 msgr2=0x7ff7d01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 -- 172.21.15.98:0/1792527218 shutdown_connections 2024-04-25T16:30:47.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 -- 172.21.15.98:0/1792527218 wait complete. 2024-04-25T16:30:47.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 Processor -- start 2024-04-25T16:30:47.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.274+0000 7ff7d84c9700 1 -- start start 2024-04-25T16:30:47.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d84c9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7d0103360 0x7ff7d019ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d84c9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0107c90 0x7ff7d019bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d84c9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7d01085e0 0x7ff7d019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.276 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d84c9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d01130c0 con 0x7ff7d0107c90 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d84c9700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d0112f40 con 0x7ff7d0103360 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d84c9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d0113240 con 0x7ff7d01085e0 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0107c90 0x7ff7d019bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0107c90 0x7ff7d019bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32980/0 (socket says 172.21.15.98:32980) 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 -- 172.21.15.98:0/990505133 learned_addr learned my addr 172.21.15.98:0/990505133 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d6a66700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7d01085e0 0x7ff7d019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d6265700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7d0103360 0x7ff7d019ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 -- 172.21.15.98:0/990505133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7d01085e0 msgr2=0x7ff7d019c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7d01085e0 0x7ff7d019c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 -- 172.21.15.98:0/990505133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7d0103360 msgr2=0x7ff7d019ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7d0103360 0x7ff7d019ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 -- 172.21.15.98:0/990505133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d01997f0 con 0x7ff7d0107c90 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d6265700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7d0103360 0x7ff7d019ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:47.277 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.275+0000 7ff7d5a64700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0107c90 0x7ff7d019bf40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00c9e0 tx=0x7ff7cc00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:47.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.276+0000 7ff7c77fe700 1 -- 172.21.15.98:0/990505133 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc007aa0 con 0x7ff7d0107c90 2024-04-25T16:30:47.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.276+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d0199ae0 con 0x7ff7d0107c90 2024-04-25T16:30:47.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.276+0000 7ff7c77fe700 1 -- 172.21.15.98:0/990505133 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7cc007c40 con 0x7ff7d0107c90 2024-04-25T16:30:47.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.276+0000 7ff7c77fe700 1 -- 172.21.15.98:0/990505133 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc018660 con 0x7ff7d0107c90 2024-04-25T16:30:47.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.276+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d019a020 con 0x7ff7d0107c90 2024-04-25T16:30:47.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.277+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d0051190 con 0x7ff7d0107c90 2024-04-25T16:30:47.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.278+0000 7ff7c77fe700 1 -- 172.21.15.98:0/990505133 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7ff7cc00f450 con 0x7ff7d0107c90 2024-04-25T16:30:47.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.279+0000 7ff7c77fe700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7bc0afc10 0x7ff7bc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:47.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.279+0000 7ff7d6265700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7bc0afc10 0x7ff7bc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:47.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.279+0000 7ff7c77fe700 1 -- 172.21.15.98:0/990505133 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7ff7cc0df240 con 0x7ff7d0107c90 2024-04-25T16:30:47.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.279+0000 7ff7d6265700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7bc0afc10 0x7ff7bc0b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0000c00 tx=0x7ff7c0005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:47.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.282+0000 7ff7c77fe700 1 -- 172.21.15.98:0/990505133 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7cc014030 con 0x7ff7d0107c90 2024-04-25T16:30:47.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:46 smithi203 ceph-mon[33115]: pgmap v300: 129 pgs: 129 active+clean; 628 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-04-25T16:30:47.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.481+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7d019b4c0 con 0x7ff7d0107c90 2024-04-25T16:30:47.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.482+0000 7ff7c77fe700 1 -- 172.21.15.98:0/990505133 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7ff7cc0a7b10 con 0x7ff7d0107c90 2024-04-25T16:30:47.482 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:47.483 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:47.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7bc0afc10 msgr2=0x7ff7bc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7bc0afc10 0x7ff7bc0b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0000c00 tx=0x7ff7c0005940 comp rx=0 tx=0).stop 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0107c90 msgr2=0x7ff7d019bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0107c90 0x7ff7d019bf40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc00c9e0 tx=0x7ff7cc00ccf0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 shutdown_connections 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7bc0afc10 0x7ff7bc0b2120 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff7d01085e0 0x7ff7d019c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7d0103360 0x7ff7d019ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 --2- 172.21.15.98:0/990505133 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff7d0107c90 0x7ff7d019bf40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 >> 172.21.15.98:0/990505133 conn(0x7ff7d00fec90 msgr2=0x7ff7d01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 shutdown_connections 2024-04-25T16:30:47.485 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:47.485+0000 7ff7d84c9700 1 -- 172.21.15.98:0/990505133 wait complete. 2024-04-25T16:30:48.194 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:47 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/990505133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:48.196 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T16:30:48.196 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T16:30:48.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:47 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/990505133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:48.431 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:48.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:47 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/990505133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:49.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:48 smithi098 ceph-mon[29162]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 24 op/s 2024-04-25T16:30:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:48 smithi173 ceph-mon[32556]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 24 op/s 2024-04-25T16:30:49.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:48 smithi203 ceph-mon[33115]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.7 KiB/s wr, 24 op/s 2024-04-25T16:30:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 -- 172.21.15.98:0/2920226119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4115220 msgr2=0x7ff0d4117640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/2920226119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4115220 0x7ff0d4117640 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00b0a0 tx=0x7ff0cc00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 -- 172.21.15.98:0/2920226119 shutdown_connections 2024-04-25T16:30:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/2920226119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0d4076b40 0x7ff0d4076f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/2920226119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4115220 0x7ff0d4117640 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/2920226119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0d4077490 0x7ff0d4114ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 -- 172.21.15.98:0/2920226119 >> 172.21.15.98:0/2920226119 conn(0x7ff0d40ffd70 msgr2=0x7ff0d41021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 -- 172.21.15.98:0/2920226119 shutdown_connections 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.237+0000 7ff0d93f2700 1 -- 172.21.15.98:0/2920226119 wait complete. 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d93f2700 1 Processor -- start 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d93f2700 1 -- start start 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d93f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4076b40 0x7ff0d41a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d93f2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0d4077490 0x7ff0d41a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d93f2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0d4115220 0x7ff0d41a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d93f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d411a7d0 con 0x7ff0d4076b40 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d93f2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d411a650 con 0x7ff0d4077490 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d93f2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d411a950 con 0x7ff0d4115220 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4076b40 0x7ff0d41a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d2ffd700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4076b40 0x7ff0d41a1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:32994/0 (socket says 172.21.15.98:32994) 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d2ffd700 1 -- 172.21.15.98:0/563779909 learned_addr learned my addr 172.21.15.98:0/563779909 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:50.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d37fe700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0d4115220 0x7ff0d41a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.238+0000 7ff0d27fc700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0d4077490 0x7ff0d41a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d2ffd700 1 -- 172.21.15.98:0/563779909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0d4115220 msgr2=0x7ff0d41a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d2ffd700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0d4115220 0x7ff0d41a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d2ffd700 1 -- 172.21.15.98:0/563779909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0d4077490 msgr2=0x7ff0d41a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d2ffd700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0d4077490 0x7ff0d41a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d2ffd700 1 -- 172.21.15.98:0/563779909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d41a5f80 con 0x7ff0d4076b40 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d27fc700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0d4077490 0x7ff0d41a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d2ffd700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4076b40 0x7ff0d41a1160 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400d950 tx=0x7ff0c400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0bbfff700 1 -- 172.21.15.98:0/563779909 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4016070 con 0x7ff0d4076b40 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0bbfff700 1 -- 172.21.15.98:0/563779909 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff0c4009dd0 con 0x7ff0d4076b40 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d41a6210 con 0x7ff0d4076b40 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0bbfff700 1 -- 172.21.15.98:0/563779909 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c400f3c0 con 0x7ff0d4076b40 2024-04-25T16:30:50.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.239+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d41a6750 con 0x7ff0d4076b40 2024-04-25T16:30:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.240+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d4051190 con 0x7ff0d4076b40 2024-04-25T16:30:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.243+0000 7ff0bbfff700 1 -- 172.21.15.98:0/563779909 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7ff0c4019070 con 0x7ff0d4076b40 2024-04-25T16:30:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.243+0000 7ff0bbfff700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff0bc0afc10 0x7ff0bc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.243+0000 7ff0d27fc700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff0bc0afc10 0x7ff0bc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.243+0000 7ff0bbfff700 1 -- 172.21.15.98:0/563779909 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7ff0c40dec30 con 0x7ff0d4076b40 2024-04-25T16:30:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.244+0000 7ff0d27fc700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff0bc0afc10 0x7ff0bc0b2120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8005fd0 tx=0x7ff0c800a5e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.247+0000 7ff0bbfff700 1 -- 172.21.15.98:0/563779909 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff0c40a7470 con 0x7ff0d4076b40 2024-04-25T16:30:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.445+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0d402e510 con 0x7ff0d4076b40 2024-04-25T16:30:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.447+0000 7ff0bbfff700 1 -- 172.21.15.98:0/563779909 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24139 (secure 0 0 0) 0x7ff0c40a6d70 con 0x7ff0d4076b40 2024-04-25T16:30:50.447 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:30:50.447 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":81,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:30:14.347619+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:30:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff0bc0afc10 msgr2=0x7ff0bc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff0bc0afc10 0x7ff0bc0b2120 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8005fd0 tx=0x7ff0c800a5e0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4076b40 msgr2=0x7ff0d41a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4076b40 0x7ff0d41a1160 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400d950 tx=0x7ff0c400dc60 comp rx=0 tx=0).stop 2024-04-25T16:30:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 shutdown_connections 2024-04-25T16:30:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff0bc0afc10 0x7ff0bc0b2120 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0d4115220 0x7ff0d41a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff0d4076b40 0x7ff0d41a1160 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 --2- 172.21.15.98:0/563779909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0d4077490 0x7ff0d41a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 >> 172.21.15.98:0/563779909 conn(0x7ff0d40ffd70 msgr2=0x7ff0d4100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 shutdown_connections 2024-04-25T16:30:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:50.449+0000 7ff0d93f2700 1 -- 172.21.15.98:0/563779909 wait complete. 2024-04-25T16:30:51.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:50 smithi098 ceph-mon[29162]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-04-25T16:30:51.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:50 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/563779909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:51.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T16:30:51.260 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T16:30:51.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T16:30:51.261 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T16:30:51.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T16:30:51.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T16:30:51.262 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T16:30:51.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T16:30:51.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T16:30:51.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T16:30:51.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T16:30:51.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T16:30:51.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:50 smithi173 ceph-mon[32556]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-04-25T16:30:51.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:50 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/563779909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:51.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:50 smithi203 ceph-mon[33115]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-04-25T16:30:51.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:50 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/563779909' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:30:51.935 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:51.939 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:51.945 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:51.955 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:51.997 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:52.368 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:52 smithi098 ceph-mon[29162]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-04-25T16:30:52.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:52 smithi203 ceph-mon[33115]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-04-25T16:30:52.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:52 smithi173 ceph-mon[32556]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.2 KiB/s wr, 14 op/s 2024-04-25T16:30:53.097 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:30:55.110 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:54 smithi098 ceph-mon[29162]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-04-25T16:30:55.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:54 smithi173 ceph-mon[32556]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-04-25T16:30:55.418 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:54 smithi203 ceph-mon[33115]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-04-25T16:30:57.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:56 smithi098 ceph-mon[29162]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-25T16:30:57.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:56 smithi173 ceph-mon[32556]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-25T16:30:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:56 smithi203 ceph-mon[33115]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-04-25T16:30:59.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:30:58 smithi098 ceph-mon[29162]: pgmap v306: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-25T16:30:59.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:30:58 smithi173 ceph-mon[32556]: pgmap v306: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-25T16:30:59.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:30:58 smithi203 ceph-mon[33115]: pgmap v306: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 -- 172.21.15.98:0/3683679879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0108640 msgr2=0x7f4aa010ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 --2- 172.21.15.98:0/3683679879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0108640 0x7f4aa010ffe0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800b0d0 tx=0x7f4a9800b3e0 comp rx=0 tx=0).stop 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 -- 172.21.15.98:0/3683679879 shutdown_connections 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 --2- 172.21.15.98:0/3683679879 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa0103af0 0x7f4aa0103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 --2- 172.21.15.98:0/3683679879 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0108640 0x7f4aa010ffe0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 --2- 172.21.15.98:0/3683679879 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4aa0107cf0 0x7f4aa0108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 -- 172.21.15.98:0/3683679879 >> 172.21.15.98:0/3683679879 conn(0x7f4aa00fe3f0 msgr2=0x7f4aa0100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 -- 172.21.15.98:0/3683679879 shutdown_connections 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.823+0000 7f4aa68df700 1 -- 172.21.15.98:0/3683679879 wait complete. 2024-04-25T16:30:59.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.824+0000 7f4aa68df700 1 Processor -- start 2024-04-25T16:30:59.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.824+0000 7f4aa68df700 1 -- start start 2024-04-25T16:30:59.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.824+0000 7f4aa68df700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0103af0 0x7f4aa019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.824+0000 7f4aa68df700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4aa0107cf0 0x7f4aa019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.824+0000 7f4aa68df700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa0108640 0x7f4aa01a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.824+0000 7f4aa68df700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa01131d0 con 0x7f4aa0103af0 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.824+0000 7f4aa68df700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa0113050 con 0x7f4aa0107cf0 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.824+0000 7f4aa68df700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa0113350 con 0x7f4aa0108640 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0103af0 0x7f4aa019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0103af0 0x7f4aa019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40124/0 (socket says 172.21.15.98:40124) 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 -- 172.21.15.98:0/1220500965 learned_addr learned my addr 172.21.15.98:0/1220500965 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9f7fe700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4aa0107cf0 0x7f4aa019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4aa4e7c700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa0108640 0x7f4aa01a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 -- 172.21.15.98:0/1220500965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa0108640 msgr2=0x7f4aa01a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:59.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa0108640 0x7f4aa01a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 -- 172.21.15.98:0/1220500965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4aa0107cf0 msgr2=0x7f4aa019d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4aa0107cf0 0x7f4aa019d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 -- 172.21.15.98:0/1220500965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa01a1d70 con 0x7f4aa0103af0 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4aa4e7c700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4aa0108640 0x7f4aa01a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9ffff700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0103af0 0x7f4aa019cea0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000d950 tx=0x7f4a9000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90016070 con 0x7f4aa0103af0 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4a90009dd0 con 0x7f4aa0103af0 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa01a2060 con 0x7f4aa0103af0 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a9000f3c0 con 0x7f4aa0103af0 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.825+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa01a9980 con 0x7f4aa0103af0 2024-04-25T16:30:59.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.826+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4aa0063da0 con 0x7f4aa0103af0 2024-04-25T16:30:59.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.828+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f4a90019070 con 0x7f4aa0103af0 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.828+0000 7f4a9d7fa700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4a880afc10 0x7f4a880b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.828+0000 7f4a9f7fe700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4a880afc10 0x7f4a880b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.829+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f4a900df1f0 con 0x7f4aa0103af0 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.829+0000 7f4a9d7fa700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] conn(0x7f4a880ba920 0x7f4a880bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.829+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 --> [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a880bd480 con 0x7f4a880ba920 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.829+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f4a900df640 con 0x7f4aa0103af0 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.829+0000 7f4aa4e7c700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] conn(0x7f4a880ba920 0x7f4a880bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.829+0000 7f4a9f7fe700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4a880afc10 0x7f4a880b2120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0107500 tx=0x7f4a94009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.829+0000 7f4aa4e7c700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] conn(0x7f4a880ba920 0x7f4a880bcdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:30:59.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.830+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 <== osd.9 v2:172.21.15.98:6826/3292339123 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a880bd480 con 0x7f4a880ba920 2024-04-25T16:30:59.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.846+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 --> [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4aa00641a0 con 0x7f4a880ba920 2024-04-25T16:30:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.846+0000 7f4a9d7fa700 1 -- 172.21.15.98:0/1220500965 <== osd.9 v2:172.21.15.98:6826/3292339123 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4aa00641a0 con 0x7f4a880ba920 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.846+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] conn(0x7f4a880ba920 msgr2=0x7f4a880bcdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.846+0000 7f4aa68df700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] conn(0x7f4a880ba920 0x7f4a880bcdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4a880afc10 msgr2=0x7f4a880b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa68df700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4a880afc10 0x7f4a880b2120 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4aa0107500 tx=0x7f4a94009190 comp rx=0 tx=0).stop 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0103af0 msgr2=0x7f4aa019cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa68df700 1 --2- 172.21.15.98:0/1220500965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4aa0103af0 0x7f4aa019cea0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4a9000d950 tx=0x7f4a9000dc60 comp rx=0 tx=0).stop 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa4e7c700 1 -- 172.21.15.98:0/1220500965 reap_dead start 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 shutdown_connections 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 >> 172.21.15.98:0/1220500965 conn(0x7f4aa00fe3f0 msgr2=0x7f4aa01096b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 shutdown_connections 2024-04-25T16:30:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:30:59.847+0000 7f4aa68df700 1 -- 172.21.15.98:0/1220500965 wait complete. 2024-04-25T16:31:01.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:00 smithi098 ceph-mon[29162]: pgmap v307: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:01.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:00 smithi173 ceph-mon[32556]: pgmap v307: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:01.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:00 smithi203 ceph-mon[33115]: pgmap v307: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:03.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:02 smithi098 ceph-mon[29162]: pgmap v308: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:03.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:02 smithi173 ceph-mon[32556]: pgmap v308: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:03.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:02 smithi203 ceph-mon[33115]: pgmap v308: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:03.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 -- 172.21.15.98:0/1250558313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 msgr2=0x7f6ce410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 --2- 172.21.15.98:0/1250558313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 0x7f6ce410e1d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009990 tx=0x7f6cd8009ca0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 -- 172.21.15.98:0/1250558313 shutdown_connections 2024-04-25T16:31:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 --2- 172.21.15.98:0/1250558313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ce4105330 0x7f6ce4105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 --2- 172.21.15.98:0/1250558313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce410e7a0 0x7f6ce4110c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 --2- 172.21.15.98:0/1250558313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 0x7f6ce410e1d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 -- 172.21.15.98:0/1250558313 >> 172.21.15.98:0/1250558313 conn(0x7f6ce40fec90 msgr2=0x7f6ce41010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 -- 172.21.15.98:0/1250558313 shutdown_connections 2024-04-25T16:31:03.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 -- 172.21.15.98:0/1250558313 wait complete. 2024-04-25T16:31:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.605+0000 7f6ce9368700 1 Processor -- start 2024-04-25T16:31:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce9368700 1 -- start start 2024-04-25T16:31:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce9368700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce4105330 0x7f6ce41a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce9368700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 0x7f6ce41a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce9368700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ce410e7a0 0x7f6ce419a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce9368700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce41141b0 con 0x7f6ce4105c80 2024-04-25T16:31:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce9368700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce4114030 con 0x7f6ce4105330 2024-04-25T16:31:03.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce9368700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce4114330 con 0x7f6ce410e7a0 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 0x7f6ce41a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 0x7f6ce41a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40160/0 (socket says 172.21.15.98:40160) 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce2ffd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce4105330 0x7f6ce41a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 -- 172.21.15.98:0/992113798 learned_addr learned my addr 172.21.15.98:0/992113798 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 -- 172.21.15.98:0/992113798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ce410e7a0 msgr2=0x7f6ce419a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ce410e7a0 0x7f6ce419a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 -- 172.21.15.98:0/992113798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce4105330 msgr2=0x7f6ce41a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce4105330 0x7f6ce41a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 -- 172.21.15.98:0/992113798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce419b0a0 con 0x7f6ce4105c80 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce2ffd700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce4105330 0x7f6ce41a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.606+0000 7f6ce27fc700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 0x7f6ce41a0d10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8006010 tx=0x7f6cd800bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.607+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8027070 con 0x7f6ce4105c80 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.607+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce419b330 con 0x7f6ce4105c80 2024-04-25T16:31:03.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.607+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6cd8022470 con 0x7f6ce4105c80 2024-04-25T16:31:03.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.607+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8017650 con 0x7f6ce4105c80 2024-04-25T16:31:03.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.607+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce41a7660 con 0x7f6ce4105c80 2024-04-25T16:31:03.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.607+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ce4063e20 con 0x7f6ce4105c80 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.609+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6cd8009d10 con 0x7f6ce4105c80 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.610+0000 7f6ccbfff700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6ccc0afc10 0x7f6ccc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.610+0000 7f6ce2ffd700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6ccc0afc10 0x7f6ccc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.610+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f6cd80e03c0 con 0x7f6ce4105c80 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.610+0000 7f6ccbfff700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] conn(0x7f6ccc0ba920 0x7f6ccc0bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.610+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 --> [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6cd80095d0 con 0x7f6ccc0ba920 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.610+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f6cd80e0810 con 0x7f6ce4105c80 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.610+0000 7f6ce2ffd700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6ccc0afc10 0x7f6ccc0b2120 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6ce419bb30 tx=0x7f6cd400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.610+0000 7f6ce37fe700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] conn(0x7f6ccc0ba920 0x7f6ccc0bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.611+0000 7f6ce37fe700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] conn(0x7f6ccc0ba920 0x7f6ccc0bcdb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:03.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.612+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 <== osd.10 v2:172.21.15.173:6824/3946840322 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6cd80095d0 con 0x7f6ccc0ba920 2024-04-25T16:31:03.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.627+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 --> [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6ce4064220 con 0x7f6ccc0ba920 2024-04-25T16:31:03.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.628+0000 7f6ccbfff700 1 -- 172.21.15.98:0/992113798 <== osd.10 v2:172.21.15.173:6824/3946840322 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6ce4064220 con 0x7f6ccc0ba920 2024-04-25T16:31:03.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.628+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 >> [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] conn(0x7f6ccc0ba920 msgr2=0x7f6ccc0bcdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.628+0000 7f6ce9368700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] conn(0x7f6ccc0ba920 0x7f6ccc0bcdb0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.628+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6ccc0afc10 msgr2=0x7f6ccc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.628+0000 7f6ce9368700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6ccc0afc10 0x7f6ccc0b2120 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6ce419bb30 tx=0x7f6cd400b410 comp rx=0 tx=0).stop 2024-04-25T16:31:03.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.628+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 msgr2=0x7f6ce41a0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.628+0000 7f6ce9368700 1 --2- 172.21.15.98:0/992113798 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ce4105c80 0x7f6ce41a0d10 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8006010 tx=0x7f6cd800bfc0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.628+0000 7f6ce37fe700 1 -- 172.21.15.98:0/992113798 reap_dead start 2024-04-25T16:31:03.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.629+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 shutdown_connections 2024-04-25T16:31:03.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.629+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 >> 172.21.15.98:0/992113798 conn(0x7f6ce40fec90 msgr2=0x7f6ce40ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:03.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.629+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 shutdown_connections 2024-04-25T16:31:03.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.629+0000 7f6ce9368700 1 -- 172.21.15.98:0/992113798 wait complete. 2024-04-25T16:31:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 -- 172.21.15.98:0/251074108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1de0102630 msgr2=0x7f1de0102a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 --2- 172.21.15.98:0/251074108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1de0102630 0x7f1de0102a40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009990 tx=0x7f1dd4009ca0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 -- 172.21.15.98:0/251074108 shutdown_connections 2024-04-25T16:31:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 --2- 172.21.15.98:0/251074108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1de0102f80 0x7f1de0111010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 --2- 172.21.15.98:0/251074108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1de01091a0 0x7f1de01095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 --2- 172.21.15.98:0/251074108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1de0102630 0x7f1de0102a40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 -- 172.21.15.98:0/251074108 >> 172.21.15.98:0/251074108 conn(0x7f1de00fe3f0 msgr2=0x7f1de0100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 -- 172.21.15.98:0/251074108 shutdown_connections 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.509+0000 7f1de6221700 1 -- 172.21.15.98:0/251074108 wait complete. 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1de6221700 1 Processor -- start 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1de6221700 1 -- start start 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1de6221700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1de0102630 0x7f1de019ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1de6221700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1de0102f80 0x7f1de019b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1de6221700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1de01091a0 0x7f1de019f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1de6221700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1de01142f0 con 0x7f1de0102630 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1de6221700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1de0114170 con 0x7f1de01091a0 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1de6221700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1de0114470 con 0x7f1de0102f80 2024-04-25T16:31:03.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1ddf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1de0102630 0x7f1de019ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1ddffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1de01091a0 0x7f1de019f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1ddffff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1de01091a0 0x7f1de019f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:45446/0 (socket says 172.21.15.98:45446) 2024-04-25T16:31:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1ddffff700 1 -- 172.21.15.98:0/1105020464 learned_addr learned my addr 172.21.15.98:0/1105020464 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.510+0000 7f1ddeffd700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1de0102f80 0x7f1de019b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddffff700 1 -- 172.21.15.98:0/1105020464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1de0102f80 msgr2=0x7f1de019b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddffff700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1de0102f80 0x7f1de019b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddffff700 1 -- 172.21.15.98:0/1105020464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1de0102630 msgr2=0x7f1de019ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddffff700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1de0102630 0x7f1de019ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddffff700 1 -- 172.21.15.98:0/1105020464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de019fc00 con 0x7f1de01091a0 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddf7fe700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1de0102630 0x7f1de019ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddffff700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1de01091a0 0x7f1de019f560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007ee0 tx=0x7f1dd800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8004bb0 con 0x7f1de01091a0 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de019fef0 con 0x7f1de01091a0 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1dd8004d50 con 0x7f1de01091a0 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd80058b0 con 0x7f1de01091a0 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.511+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de01a7810 con 0x7f1de01091a0 2024-04-25T16:31:03.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.512+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1de0063da0 con 0x7f1de01091a0 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.515+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f1dd801a020 con 0x7f1de01091a0 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.516+0000 7f1ddcff9700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1dc80afc10 0x7f1dc80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.516+0000 7f1ddf7fe700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1dc80afc10 0x7f1dc80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.516+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f1dd80df100 con 0x7f1de01091a0 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.516+0000 7f1ddcff9700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] conn(0x7f1dc80ba920 0x7f1dc80bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.516+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 --> [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1dd8023e70 con 0x7f1dc80ba920 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.516+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f1dd80df550 con 0x7f1de01091a0 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.516+0000 7f1ddeffd700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] conn(0x7f1dc80ba920 0x7f1dc80bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.516+0000 7f1ddf7fe700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1dc80afc10 0x7f1dc80b2120 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1de0076150 tx=0x7f1dd000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.517+0000 7f1ddeffd700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] conn(0x7f1dc80ba920 0x7f1dc80bcdb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.518+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 <== osd.4 v2:172.21.15.173:6808/1101244004 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1dd8023e70 con 0x7f1dc80ba920 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.534+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 --> [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1de00641a0 con 0x7f1dc80ba920 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.534+0000 7f1ddcff9700 1 -- 172.21.15.98:0/1105020464 <== osd.4 v2:172.21.15.173:6808/1101244004 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1de00641a0 con 0x7f1dc80ba920 2024-04-25T16:31:03.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 >> [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] conn(0x7f1dc80ba920 msgr2=0x7f1dc80bcdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] conn(0x7f1dc80ba920 0x7f1dc80bcdb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1dc80afc10 msgr2=0x7f1dc80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1dc80afc10 0x7f1dc80b2120 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1de0076150 tx=0x7f1dd000a380 comp rx=0 tx=0).stop 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1de01091a0 msgr2=0x7f1de019f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 --2- 172.21.15.98:0/1105020464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1de01091a0 0x7f1de019f560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8007ee0 tx=0x7f1dd800d3b0 comp rx=0 tx=0).stop 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1ddffff700 1 -- 172.21.15.98:0/1105020464 reap_dead start 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 shutdown_connections 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 >> 172.21.15.98:0/1105020464 conn(0x7f1de00fe3f0 msgr2=0x7f1de0100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 shutdown_connections 2024-04-25T16:31:03.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:03.535+0000 7f1de6221700 1 -- 172.21.15.98:0/1105020464 wait complete. 2024-04-25T16:31:04.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:04 smithi098 ceph-mon[29162]: pgmap v309: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T16:31:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 -- 172.21.15.98:0/1797451024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0107ce0 msgr2=0x7f0ac01080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 --2- 172.21.15.98:0/1797451024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0107ce0 0x7f0ac01080f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80099c0 tx=0x7f0aa8009cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 -- 172.21.15.98:0/1797451024 shutdown_connections 2024-04-25T16:31:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 --2- 172.21.15.98:0/1797451024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac0108630 0x7f0ac010ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 --2- 172.21.15.98:0/1797451024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac01033b0 0x7f0ac01037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 --2- 172.21.15.98:0/1797451024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0107ce0 0x7f0ac01080f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 -- 172.21.15.98:0/1797451024 >> 172.21.15.98:0/1797451024 conn(0x7f0ac0078cb0 msgr2=0x7f0ac00790f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:04.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 -- 172.21.15.98:0/1797451024 shutdown_connections 2024-04-25T16:31:04.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.339+0000 7f0ac5e45700 1 -- 172.21.15.98:0/1797451024 wait complete. 2024-04-25T16:31:04.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0ac5e45700 1 Processor -- start 2024-04-25T16:31:04.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0ac5e45700 1 -- start start 2024-04-25T16:31:04.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0ac5e45700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac01033b0 0x7f0ac019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0ac5e45700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac0107ce0 0x7f0ac019b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0ac5e45700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0108630 0x7f0ac019f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0ac5e45700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac01130f0 con 0x7f0ac0108630 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0ac5e45700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac0112f70 con 0x7f0ac0107ce0 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0ac5e45700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac0113270 con 0x7f0ac01033b0 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0abffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0108630 0x7f0ac019f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.340+0000 7f0abffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0108630 0x7f0ac019f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40180/0 (socket says 172.21.15.98:40180) 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abffff700 1 -- 172.21.15.98:0/684756310 learned_addr learned my addr 172.21.15.98:0/684756310 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0ab6ffd700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac0107ce0 0x7f0ac019b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abf7fe700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac01033b0 0x7f0ac019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abffff700 1 -- 172.21.15.98:0/684756310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac01033b0 msgr2=0x7f0ac019ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abffff700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0ac01033b0 0x7f0ac019ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abffff700 1 -- 172.21.15.98:0/684756310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac0107ce0 msgr2=0x7f0ac019b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abffff700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ac0107ce0 0x7f0ac019b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abffff700 1 -- 172.21.15.98:0/684756310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac019fc20 con 0x7f0ac0108630 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abffff700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0108630 0x7f0ac019f580 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8007ee0 tx=0x7f0ab800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8004bb0 con 0x7f0ac0108630 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac019ff10 con 0x7f0ac0108630 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0ab8004d50 con 0x7f0ac0108630 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab80058b0 con 0x7f0ac0108630 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.341+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac01a7830 con 0x7f0ac0108630 2024-04-25T16:31:04.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.342+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ac0063d60 con 0x7f0ac0108630 2024-04-25T16:31:04.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.344+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f0ab801a020 con 0x7f0ac0108630 2024-04-25T16:31:04.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.344+0000 7f0abd7fa700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0aac0afc10 0x7f0aac0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.344+0000 7f0abf7fe700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0aac0afc10 0x7f0aac0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.344+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f0ab80df0e0 con 0x7f0ac0108630 2024-04-25T16:31:04.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.344+0000 7f0abd7fa700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] conn(0x7f0aac0ba920 0x7f0aac0bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.344+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 --> [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ab8005a50 con 0x7f0aac0ba920 2024-04-25T16:31:04.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.344+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f0ab80df530 con 0x7f0ac0108630 2024-04-25T16:31:04.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.345+0000 7f0ab6ffd700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] conn(0x7f0aac0ba920 0x7f0aac0bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.345+0000 7f0abf7fe700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0aac0afc10 0x7f0aac0b2120 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0005950 tx=0x7f0ab000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:04.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.345+0000 7f0ab6ffd700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] conn(0x7f0aac0ba920 0x7f0aac0bcdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:04.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.346+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 <== osd.1 v2:172.21.15.173:6800/2144972305 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0ab8005a50 con 0x7f0aac0ba920 2024-04-25T16:31:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.362+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 --> [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ac0064160 con 0x7f0aac0ba920 2024-04-25T16:31:04.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0abd7fa700 1 -- 172.21.15.98:0/684756310 <== osd.1 v2:172.21.15.173:6800/2144972305 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0ac0064160 con 0x7f0aac0ba920 2024-04-25T16:31:04.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 >> [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] conn(0x7f0aac0ba920 msgr2=0x7f0aac0bcdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0ac5e45700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] conn(0x7f0aac0ba920 0x7f0aac0bcdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0aac0afc10 msgr2=0x7f0aac0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0ac5e45700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0aac0afc10 0x7f0aac0b2120 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0005950 tx=0x7f0ab000b410 comp rx=0 tx=0).stop 2024-04-25T16:31:04.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0108630 msgr2=0x7f0ac019f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0ac5e45700 1 --2- 172.21.15.98:0/684756310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0ac0108630 0x7f0ac019f580 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8007ee0 tx=0x7f0ab800d3b0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0abffff700 1 -- 172.21.15.98:0/684756310 reap_dead start 2024-04-25T16:31:04.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 shutdown_connections 2024-04-25T16:31:04.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.363+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 >> 172.21.15.98:0/684756310 conn(0x7f0ac0078cb0 msgr2=0x7f0ac0075da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:04.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.364+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 shutdown_connections 2024-04-25T16:31:04.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.364+0000 7f0ac5e45700 1 -- 172.21.15.98:0/684756310 wait complete. 2024-04-25T16:31:04.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:04 smithi203 ceph-mon[33115]: pgmap v309: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T16:31:04.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:04 smithi173 ceph-mon[32556]: pgmap v309: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-25T16:31:04.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 -- 172.21.15.98:0/1384273265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0109710 msgr2=0x7f09b0111070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 --2- 172.21.15.98:0/1384273265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0109710 0x7f09b0111070 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f09a800b0d0 tx=0x7f09a800b3e0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 -- 172.21.15.98:0/1384273265 shutdown_connections 2024-04-25T16:31:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 --2- 172.21.15.98:0/1384273265 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0109710 0x7f09b0111070 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 --2- 172.21.15.98:0/1384273265 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09b0104490 0x7f09b01048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 --2- 172.21.15.98:0/1384273265 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b0108dc0 0x7f09b01091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 -- 172.21.15.98:0/1384273265 >> 172.21.15.98:0/1384273265 conn(0x7f09b00ffda0 msgr2=0x7f09b01021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 -- 172.21.15.98:0/1384273265 shutdown_connections 2024-04-25T16:31:04.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 -- 172.21.15.98:0/1384273265 wait complete. 2024-04-25T16:31:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.903+0000 7f09b5fd0700 1 Processor -- start 2024-04-25T16:31:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09b5fd0700 1 -- start start 2024-04-25T16:31:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09b5fd0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0104490 0x7f09b019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09b5fd0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09b0108dc0 0x7f09b019d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09b5fd0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b0109710 0x7f09b01a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09b5fd0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f09b01141f0 con 0x7f09b0104490 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09b5fd0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f09b0114070 con 0x7f09b0108dc0 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09b5fd0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f09b0114370 con 0x7f09b0109710 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0104490 0x7f09b019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09af7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0104490 0x7f09b019cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40210/0 (socket says 172.21.15.98:40210) 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09af7fe700 1 -- 172.21.15.98:0/2302565855 learned_addr learned my addr 172.21.15.98:0/2302565855 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09aeffd700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09b0108dc0 0x7f09b019d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.904+0000 7f09affff700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b0109710 0x7f09b01a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09af7fe700 1 -- 172.21.15.98:0/2302565855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b0109710 msgr2=0x7f09b01a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09af7fe700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09b0109710 0x7f09b01a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09af7fe700 1 -- 172.21.15.98:0/2302565855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09b0108dc0 msgr2=0x7f09b019d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09af7fe700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09b0108dc0 0x7f09b019d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09af7fe700 1 -- 172.21.15.98:0/2302565855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b01a1c50 con 0x7f09b0104490 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09aeffd700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09b0108dc0 0x7f09b019d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09af7fe700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0104490 0x7f09b019cd80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f09a400d950 tx=0x7f09a400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4016070 con 0x7f09b0104490 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f09a400fd20 con 0x7f09b0104490 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a400e3b0 con 0x7f09b0104490 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b01a1f40 con 0x7f09b0104490 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.905+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b01a9860 con 0x7f09b0104490 2024-04-25T16:31:04.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.906+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09b0107f80 con 0x7f09b0104490 2024-04-25T16:31:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.907+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f09a4019070 con 0x7f09b0104490 2024-04-25T16:31:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.908+0000 7f09acff9700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09980afc90 0x7f09980b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.908+0000 7f09aeffd700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09980afc90 0x7f09980b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.908+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f09a40def50 con 0x7f09b0104490 2024-04-25T16:31:04.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.908+0000 7f09acff9700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] conn(0x7f09980ba990 0x7f09980bce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.908+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 --> [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09980bd4f0 con 0x7f09980ba990 2024-04-25T16:31:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.908+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f09a40df3a0 con 0x7f09b0104490 2024-04-25T16:31:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.909+0000 7f09aeffd700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09980afc90 0x7f09980b21a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f09b01103d0 tx=0x7f09a000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.909+0000 7f09affff700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] conn(0x7f09980ba990 0x7f09980bce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.909+0000 7f09affff700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] conn(0x7f09980ba990 0x7f09980bce20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:04.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.910+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 <== osd.11 v2:172.21.15.203:6824/1060778865 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f09980bd4f0 con 0x7f09980ba990 2024-04-25T16:31:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.926+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 --> [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f09b0105940 con 0x7f09980ba990 2024-04-25T16:31:04.926 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09acff9700 1 -- 172.21.15.98:0/2302565855 <== osd.11 v2:172.21.15.203:6824/1060778865 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f09b0105940 con 0x7f09980ba990 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 >> [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] conn(0x7f09980ba990 msgr2=0x7f09980bce20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09b5fd0700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] conn(0x7f09980ba990 0x7f09980bce20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09980afc90 msgr2=0x7f09980b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09b5fd0700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09980afc90 0x7f09980b21a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f09b01103d0 tx=0x7f09a000b410 comp rx=0 tx=0).stop 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0104490 msgr2=0x7f09b019cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09b5fd0700 1 --2- 172.21.15.98:0/2302565855 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f09b0104490 0x7f09b019cd80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f09a400d950 tx=0x7f09a400dc60 comp rx=0 tx=0).stop 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09affff700 1 -- 172.21.15.98:0/2302565855 reap_dead start 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 shutdown_connections 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.927+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 >> 172.21.15.98:0/2302565855 conn(0x7f09b00ffda0 msgr2=0x7f09b0110db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.928+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 shutdown_connections 2024-04-25T16:31:04.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.928+0000 7f09b5fd0700 1 -- 172.21.15.98:0/2302565855 wait complete. 2024-04-25T16:31:05.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 -- 172.21.15.98:0/1004475876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600109720 msgr2=0x7fb600111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 --2- 172.21.15.98:0/1004475876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600109720 0x7fb600111080 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800b0d0 tx=0x7fb5f800b3e0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 -- 172.21.15.98:0/1004475876 shutdown_connections 2024-04-25T16:31:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 --2- 172.21.15.98:0/1004475876 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600109720 0x7fb600111080 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 --2- 172.21.15.98:0/1004475876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6001044a0 0x7fb6001048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 --2- 172.21.15.98:0/1004475876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb600108dd0 0x7fb6001091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 -- 172.21.15.98:0/1004475876 >> 172.21.15.98:0/1004475876 conn(0x7fb6000ffd70 msgr2=0x7fb6001021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 -- 172.21.15.98:0/1004475876 shutdown_connections 2024-04-25T16:31:05.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.561+0000 7fb604aa0700 1 -- 172.21.15.98:0/1004475876 wait complete. 2024-04-25T16:31:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb604aa0700 1 Processor -- start 2024-04-25T16:31:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb604aa0700 1 -- start start 2024-04-25T16:31:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb604aa0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6001044a0 0x7fb6001a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb604aa0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600108dd0 0x7fb6001a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb604aa0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb600109720 0x7fb60019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb604aa0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb600114200 con 0x7fb600108dd0 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb604aa0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb600114080 con 0x7fb600109720 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb604aa0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb600114380 con 0x7fb6001044a0 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb5fdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600108dd0 0x7fb6001a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb5fdd9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600108dd0 0x7fb6001a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47902/0 (socket says 172.21.15.98:47902) 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb5fdd9b700 1 -- 172.21.15.98:0/786118654 learned_addr learned my addr 172.21.15.98:0/786118654 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.562+0000 7fb5fe59c700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6001044a0 0x7fb6001a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5fed9d700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb600109720 0x7fb60019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5fdd9b700 1 -- 172.21.15.98:0/786118654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6001044a0 msgr2=0x7fb6001a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5fdd9b700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6001044a0 0x7fb6001a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5fdd9b700 1 -- 172.21.15.98:0/786118654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb600109720 msgr2=0x7fb60019a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5fdd9b700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb600109720 0x7fb60019a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5fdd9b700 1 -- 172.21.15.98:0/786118654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb60019afe0 con 0x7fb600108dd0 2024-04-25T16:31:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5fdd9b700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600108dd0 0x7fb6001a0cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400d980 tx=0x7fb5f400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4017070 con 0x7fb600108dd0 2024-04-25T16:31:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb60019b2d0 con 0x7fb600108dd0 2024-04-25T16:31:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb5f400fd20 con 0x7fb600108dd0 2024-04-25T16:31:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f400e3b0 con 0x7fb600108dd0 2024-04-25T16:31:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.563+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6001a7610 con 0x7fb600108dd0 2024-04-25T16:31:05.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.564+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb600063e20 con 0x7fb600108dd0 2024-04-25T16:31:05.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.566+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fb5f400f890 con 0x7fb600108dd0 2024-04-25T16:31:05.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.566+0000 7fb5ef7fe700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5e80afc10 0x7fb5e80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.566+0000 7fb5fe59c700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5e80afc10 0x7fb5e80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.566+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fb5f40dee00 con 0x7fb600108dd0 2024-04-25T16:31:05.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.566+0000 7fb5ef7fe700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] conn(0x7fb5e80ba920 0x7fb5e80bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:05.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.566+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 --> [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5f4016ec0 con 0x7fb5e80ba920 2024-04-25T16:31:05.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.566+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fb5f40df250 con 0x7fb600108dd0 2024-04-25T16:31:05.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.567+0000 7fb5fe59c700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5e80afc10 0x7fb5e80b2120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb60019bae0 tx=0x7fb5f000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.567+0000 7fb5fed9d700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] conn(0x7fb5e80ba920 0x7fb5e80bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:05.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.567+0000 7fb5fed9d700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] conn(0x7fb5e80ba920 0x7fb5e80bcdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:05.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.568+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 <== osd.2 v2:172.21.15.203:6800/3173578158 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5f4016ec0 con 0x7fb5e80ba920 2024-04-25T16:31:05.583 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.584+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 --> [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb600064220 con 0x7fb5e80ba920 2024-04-25T16:31:05.584 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb5ef7fe700 1 -- 172.21.15.98:0/786118654 <== osd.2 v2:172.21.15.203:6800/3173578158 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb600064220 con 0x7fb5e80ba920 2024-04-25T16:31:05.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 >> [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] conn(0x7fb5e80ba920 msgr2=0x7fb5e80bcdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] conn(0x7fb5e80ba920 0x7fb5e80bcdb0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:05.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5e80afc10 msgr2=0x7fb5e80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5e80afc10 0x7fb5e80b2120 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb60019bae0 tx=0x7fb5f000a380 comp rx=0 tx=0).stop 2024-04-25T16:31:05.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600108dd0 msgr2=0x7fb6001a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:05.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 --2- 172.21.15.98:0/786118654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb600108dd0 0x7fb6001a0cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400d980 tx=0x7fb5f400dc90 comp rx=0 tx=0).stop 2024-04-25T16:31:05.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb5fed9d700 1 -- 172.21.15.98:0/786118654 reap_dead start 2024-04-25T16:31:05.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 shutdown_connections 2024-04-25T16:31:05.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 >> 172.21.15.98:0/786118654 conn(0x7fb6000ffd70 msgr2=0x7fb600108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:05.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.585+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 shutdown_connections 2024-04-25T16:31:05.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:05.586+0000 7fb604aa0700 1 -- 172.21.15.98:0/786118654 wait complete. 2024-04-25T16:31:07.059 INFO:teuthology.orchestra.run.smithi098.stdout:309237645336 2024-04-25T16:31:07.060 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T16:31:07.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:06 smithi098 ceph-mon[29162]: pgmap v310: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:07.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:06 smithi173 ceph-mon[32556]: pgmap v310: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:07.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:06 smithi203 ceph-mon[33115]: pgmap v310: 129 pgs: 129 active+clean; 644 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:07.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 -- 172.21.15.98:0/3148694061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01054d0 msgr2=0x7f15f0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:07.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 --2- 172.21.15.98:0/3148694061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01054d0 0x7f15f0102b80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f15e4009990 tx=0x7f15e4009ca0 comp rx=0 tx=0).stop 2024-04-25T16:31:07.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 -- 172.21.15.98:0/3148694061 shutdown_connections 2024-04-25T16:31:07.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 --2- 172.21.15.98:0/3148694061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15f01050c0 0x7f15f0102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:07.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 --2- 172.21.15.98:0/3148694061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15f0103210 0x7f15f0112e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:07.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 --2- 172.21.15.98:0/3148694061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01054d0 0x7f15f0102b80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:07.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 -- 172.21.15.98:0/3148694061 >> 172.21.15.98:0/3148694061 conn(0x7f15f00fe390 msgr2=0x7f15f01007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:07.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 -- 172.21.15.98:0/3148694061 shutdown_connections 2024-04-25T16:31:07.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.517+0000 7f15f52a6700 1 -- 172.21.15.98:0/3148694061 wait complete. 2024-04-25T16:31:07.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15f52a6700 1 Processor -- start 2024-04-25T16:31:07.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15f52a6700 1 -- start start 2024-04-25T16:31:07.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15f52a6700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15f0103210 0x7f15f019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:07.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15f52a6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01050c0 0x7f15f019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:07.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15f52a6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15f01054d0 0x7f15f01a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:07.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15f52a6700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f15f01163c0 con 0x7f15f01050c0 2024-04-25T16:31:07.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15f52a6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f15f0116240 con 0x7f15f01054d0 2024-04-25T16:31:07.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15f52a6700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f15f0116540 con 0x7f15f0103210 2024-04-25T16:31:07.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15ee7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01050c0 0x7f15f019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:07.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15ee7fc700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01050c0 0x7f15f019d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47914/0 (socket says 172.21.15.98:47914) 2024-04-25T16:31:07.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.518+0000 7f15ee7fc700 1 -- 172.21.15.98:0/881864236 learned_addr learned my addr 172.21.15.98:0/881864236 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:07.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15eeffd700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15f0103210 0x7f15f019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:07.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15ef7fe700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15f01054d0 0x7f15f01a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:07.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15ee7fc700 1 -- 172.21.15.98:0/881864236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15f0103210 msgr2=0x7f15f019cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:07.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15ee7fc700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f15f0103210 0x7f15f019cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:07.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15ee7fc700 1 -- 172.21.15.98:0/881864236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15f01054d0 msgr2=0x7f15f01a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:07.525 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15ee7fc700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15f01054d0 0x7f15f01a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:07.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15ee7fc700 1 -- 172.21.15.98:0/881864236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f01a1bf0 con 0x7f15f01050c0 2024-04-25T16:31:07.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15ef7fe700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15f01054d0 0x7f15f01a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:07.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15ee7fc700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01050c0 0x7f15f019d370 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f15e40057c0 tx=0x7f15e4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:07.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4027070 con 0x7f15f01050c0 2024-04-25T16:31:07.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f01a1e80 con 0x7f15f01050c0 2024-04-25T16:31:07.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f15e400fda0 con 0x7f15f01050c0 2024-04-25T16:31:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e4022c70 con 0x7f15f01050c0 2024-04-25T16:31:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.519+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f01a2360 con 0x7f15f01050c0 2024-04-25T16:31:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.520+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f15f0112940 con 0x7f15f01050c0 2024-04-25T16:31:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.521+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f15e402a070 con 0x7f15f01050c0 2024-04-25T16:31:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.522+0000 7f15d7fff700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f15d80afc10 0x7f15d80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.522+0000 7f15eeffd700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f15d80afc10 0x7f15d80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:07.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.522+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f15e40e05a0 con 0x7f15f01050c0 2024-04-25T16:31:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.522+0000 7f15d7fff700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] conn(0x7f15d80ba920 0x7f15d80bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.522+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 --> [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15e402ee50 con 0x7f15d80ba920 2024-04-25T16:31:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.522+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f15e40e09f0 con 0x7f15f01050c0 2024-04-25T16:31:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.523+0000 7f15ef7fe700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] conn(0x7f15d80ba920 0x7f15d80bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.523+0000 7f15eeffd700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f15d80afc10 0x7f15d80b2120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f15e0005fd0 tx=0x7f15e0005e50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.523+0000 7f15ef7fe700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] conn(0x7f15d80ba920 0x7f15d80bcdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:07.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.524+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 <== osd.3 v2:172.21.15.98:6810/214861667 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f15e402ee50 con 0x7f15d80ba920 2024-04-25T16:31:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.540+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 --> [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f15f0103860 con 0x7f15d80ba920 2024-04-25T16:31:07.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.540+0000 7f15d7fff700 1 -- 172.21.15.98:0/881864236 <== osd.3 v2:172.21.15.98:6810/214861667 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f15f0103860 con 0x7f15d80ba920 2024-04-25T16:31:07.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.540+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] conn(0x7f15d80ba920 msgr2=0x7f15d80bcdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.540+0000 7f15f52a6700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] conn(0x7f15d80ba920 0x7f15d80bcdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f15d80afc10 msgr2=0x7f15d80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15f52a6700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f15d80afc10 0x7f15d80b2120 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f15e0005fd0 tx=0x7f15e0005e50 comp rx=0 tx=0).stop 2024-04-25T16:31:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01050c0 msgr2=0x7f15f019d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15f52a6700 1 --2- 172.21.15.98:0/881864236 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f15f01050c0 0x7f15f019d370 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f15e40057c0 tx=0x7f15e4009f90 comp rx=0 tx=0).stop 2024-04-25T16:31:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15ef7fe700 1 -- 172.21.15.98:0/881864236 reap_dead start 2024-04-25T16:31:07.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 shutdown_connections 2024-04-25T16:31:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 >> 172.21.15.98:0/881864236 conn(0x7f15f00fe390 msgr2=0x7f15f0111ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 shutdown_connections 2024-04-25T16:31:07.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:07.541+0000 7f15f52a6700 1 -- 172.21.15.98:0/881864236 wait complete. 2024-04-25T16:31:07.923 INFO:teuthology.orchestra.run.smithi098.stdout:55834574923 2024-04-25T16:31:07.923 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T16:31:08.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.583+0000 7f59ebc05700 1 -- 172.21.15.98:0/3998611801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4102740 msgr2=0x7f59e4102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:08.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.583+0000 7f59ebc05700 1 --2- 172.21.15.98:0/3998611801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4102740 0x7f59e4102b50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009990 tx=0x7f59d4009ca0 comp rx=0 tx=0).stop 2024-04-25T16:31:08.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.583+0000 7f59ebc05700 1 -- 172.21.15.98:0/3998611801 shutdown_connections 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.583+0000 7f59ebc05700 1 --2- 172.21.15.98:0/3998611801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59e41091a0 0x7f59e41095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.583+0000 7f59ebc05700 1 --2- 172.21.15.98:0/3998611801 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e4103090 0x7f59e4111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.583+0000 7f59ebc05700 1 --2- 172.21.15.98:0/3998611801 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4102740 0x7f59e4102b50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.583+0000 7f59ebc05700 1 -- 172.21.15.98:0/3998611801 >> 172.21.15.98:0/3998611801 conn(0x7f59e40fe3b0 msgr2=0x7f59e4100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.584+0000 7f59ebc05700 1 -- 172.21.15.98:0/3998611801 shutdown_connections 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.584+0000 7f59ebc05700 1 -- 172.21.15.98:0/3998611801 wait complete. 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.584+0000 7f59ebc05700 1 Processor -- start 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.584+0000 7f59ebc05700 1 -- start start 2024-04-25T16:31:08.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.584+0000 7f59ebc05700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e4102740 0x7f59e419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.584+0000 7f59ebc05700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4103090 0x7f59e419b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.584+0000 7f59ebc05700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59e41091a0 0x7f59e419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59ebc05700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f59e4114370 con 0x7f59e4103090 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59ebc05700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41141f0 con 0x7f59e4102740 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59ebc05700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f59e41144f0 con 0x7f59e41091a0 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4103090 0x7f59e419b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4103090 0x7f59e419b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40200/0 (socket says 172.21.15.98:40200) 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 -- 172.21.15.98:0/2443368104 learned_addr learned my addr 172.21.15.98:0/2443368104 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59ea1a2700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59e41091a0 0x7f59e419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e99a1700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e4102740 0x7f59e419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:08.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 -- 172.21.15.98:0/2443368104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59e41091a0 msgr2=0x7f59e419f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59e41091a0 0x7f59e419f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 -- 172.21.15.98:0/2443368104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e4102740 msgr2=0x7f59e419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59e4102740 0x7f59e419ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 -- 172.21.15.98:0/2443368104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e419fc30 con 0x7f59e4103090 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59e91a0700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4103090 0x7f59e419b300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f59d4000c00 tx=0x7f59d40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4027070 con 0x7f59e4103090 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e419fec0 con 0x7f59e4103090 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f59d400fd00 con 0x7f59e4103090 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4022e40 con 0x7f59e4103090 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.585+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e41a7840 con 0x7f59e4103090 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.586+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f59e41a03f0 con 0x7f59e4103090 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.588+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f59d402a070 con 0x7f59e4103090 2024-04-25T16:31:08.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.588+0000 7f59daffd700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f59d00afc10 0x7f59d00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.589+0000 7f59e99a1700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f59d00afc10 0x7f59d00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.589+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f59d40e0570 con 0x7f59e4103090 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.589+0000 7f59daffd700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] conn(0x7f59d00ba920 0x7f59d00bcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.589+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 --> [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59d00bd480 con 0x7f59d00ba920 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.589+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f59d40e09c0 con 0x7f59e4103090 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.589+0000 7f59ea1a2700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] conn(0x7f59d00ba920 0x7f59d00bcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.589+0000 7f59e99a1700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f59d00afc10 0x7f59d00b2120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f59e0006fd0 tx=0x7f59e0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.589+0000 7f59ea1a2700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] conn(0x7f59d00ba920 0x7f59d00bcdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.590+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 <== osd.6 v2:172.21.15.98:6818/3921819389 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f59d00bd480 con 0x7f59d00ba920 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.608+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 --> [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f59e4063fb0 con 0x7f59d00ba920 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.608+0000 7f59daffd700 1 -- 172.21.15.98:0/2443368104 <== osd.6 v2:172.21.15.98:6818/3921819389 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f59e4063fb0 con 0x7f59d00ba920 2024-04-25T16:31:08.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.608+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] conn(0x7f59d00ba920 msgr2=0x7f59d00bcdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.608+0000 7f59ebc05700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] conn(0x7f59d00ba920 0x7f59d00bcdb0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f59d00afc10 msgr2=0x7f59d00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ebc05700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f59d00afc10 0x7f59d00b2120 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f59e0006fd0 tx=0x7f59e0008040 comp rx=0 tx=0).stop 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4103090 msgr2=0x7f59e419b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ebc05700 1 --2- 172.21.15.98:0/2443368104 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f59e4103090 0x7f59e419b300 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f59d4000c00 tx=0x7f59d40047b0 comp rx=0 tx=0).stop 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ea1a2700 1 -- 172.21.15.98:0/2443368104 reap_dead start 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 shutdown_connections 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 >> 172.21.15.98:0/2443368104 conn(0x7f59e40fe3b0 msgr2=0x7f59e41007d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:08.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 shutdown_connections 2024-04-25T16:31:08.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:04.609+0000 7f59ebc05700 1 -- 172.21.15.98:0/2443368104 wait complete. 2024-04-25T16:31:08.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-25T16:31:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-25T16:31:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-04-25T16:31:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-25T16:31:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-25T16:31:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-25T16:31:08.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-25T16:31:08.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T16:31:08.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-04-25T16:31:08.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]: dispatch 2024-04-25T16:31:08.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-25T16:31:08.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-25T16:31:08.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-04-25T16:31:08.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]: dispatch 2024-04-25T16:31:08.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]: dispatch 2024-04-25T16:31:08.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-25T16:31:08.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]: dispatch 2024-04-25T16:31:08.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]: dispatch 2024-04-25T16:31:08.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]: dispatch 2024-04-25T16:31:08.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]: dispatch 2024-04-25T16:31:08.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-25T16:31:08.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2024-04-25T16:31:09.105 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: pgmap v311: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:31:09.105 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-04-25T16:31:09.105 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-25T16:31:09.105 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-25T16:31:09.105 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-25T16:31:09.105 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-25T16:31:09.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T16:31:09.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-04-25T16:31:09.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]': finished 2024-04-25T16:31:09.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-25T16:31:09.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-25T16:31:09.106 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:09 smithi098 ceph-mon[29162]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T16:31:09.136 INFO:teuthology.orchestra.run.smithi098.stdout:77309411398 2024-04-25T16:31:09.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T16:31:09.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: pgmap v311: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-25T16:31:09.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:09 smithi173 ceph-mon[32556]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: pgmap v311: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.12", "id": [3, 9]}]': finished 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 8]}]': finished 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 6]}]': finished 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3c", "id": [4, 9]}]': finished 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [5, 10]}]': finished 2024-04-25T16:31:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 8]}]': finished 2024-04-25T16:31:09.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-25T16:31:09.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2024-04-25T16:31:09.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:09 smithi203 ceph-mon[33115]: osdmap e82: 12 total, 12 up, 12 in 2024-04-25T16:31:09.536 INFO:teuthology.orchestra.run.smithi098.stdout:137438953532 2024-04-25T16:31:09.536 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T16:31:09.810 INFO:teuthology.orchestra.run.smithi098.stdout:184683593778 2024-04-25T16:31:09.811 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T16:31:10.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:10 smithi098 ceph-mon[29162]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T16:31:10.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:31:10.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:10 smithi098 ceph-mon[29162]: pgmap v314: 129 pgs: 3 peering, 126 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-25T16:31:10.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:10 smithi173 ceph-mon[32556]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T16:31:10.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:31:10.244 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:10 smithi173 ceph-mon[32556]: pgmap v314: 129 pgs: 3 peering, 126 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-25T16:31:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:10 smithi203 ceph-mon[33115]: osdmap e83: 12 total, 12 up, 12 in 2024-04-25T16:31:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:31:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:10 smithi203 ceph-mon[33115]: pgmap v314: 129 pgs: 3 peering, 126 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 383 B/s wr, 10 op/s 2024-04-25T16:31:10.851 INFO:teuthology.orchestra.run.smithi098.stdout:257698037793 2024-04-25T16:31:10.852 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T16:31:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:11.051 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.057+0000 7f5b37255700 1 -- 172.21.15.98:0/1447413766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b30107ce0 msgr2=0x7f5b301080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.057+0000 7f5b37255700 1 --2- 172.21.15.98:0/1447413766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b30107ce0 0x7f5b301080f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5b240099c0 tx=0x7f5b24009cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.057+0000 7f5b37255700 1 -- 172.21.15.98:0/1447413766 shutdown_connections 2024-04-25T16:31:11.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.057+0000 7f5b37255700 1 --2- 172.21.15.98:0/1447413766 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b30108630 0x7f5b3010ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.057+0000 7f5b37255700 1 --2- 172.21.15.98:0/1447413766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b301033b0 0x7f5b301037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.057+0000 7f5b37255700 1 --2- 172.21.15.98:0/1447413766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b30107ce0 0x7f5b301080f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.057+0000 7f5b37255700 1 -- 172.21.15.98:0/1447413766 >> 172.21.15.98:0/1447413766 conn(0x7f5b30078cb0 msgr2=0x7f5b300790f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.058+0000 7f5b37255700 1 -- 172.21.15.98:0/1447413766 shutdown_connections 2024-04-25T16:31:11.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.058+0000 7f5b37255700 1 -- 172.21.15.98:0/1447413766 wait complete. 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b37255700 1 Processor -- start 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b37255700 1 -- start start 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b37255700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b301033b0 0x7f5b3019baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b37255700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b30107ce0 0x7f5b3019c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b37255700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30108630 0x7f5b301a6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b37255700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301130f0 con 0x7f5b301033b0 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b37255700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5b30112f70 con 0x7f5b30108630 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b37255700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5b30113270 con 0x7f5b30107ce0 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.059+0000 7f5b34ff1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b301033b0 0x7f5b3019baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b34ff1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b301033b0 0x7f5b3019baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47950/0 (socket says 172.21.15.98:47950) 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b34ff1700 1 -- 172.21.15.98:0/1447138974 learned_addr learned my addr 172.21.15.98:0/1447138974 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:11.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b2ffff700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b30107ce0 0x7f5b3019c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b357f2700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30108630 0x7f5b301a6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b2ffff700 1 -- 172.21.15.98:0/1447138974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30108630 msgr2=0x7f5b301a6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b2ffff700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b30108630 0x7f5b301a6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b2ffff700 1 -- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b301033b0 msgr2=0x7f5b3019baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b2ffff700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b301033b0 0x7f5b3019baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b2ffff700 1 -- 172.21.15.98:0/1447138974 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b3019c8e0 con 0x7f5b30107ce0 2024-04-25T16:31:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b34ff1700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5b301033b0 0x7f5b3019baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:11.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b2ffff700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b30107ce0 0x7f5b3019c030 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b240098e0 tx=0x7f5b24004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.060+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b24026070 con 0x7f5b30107ce0 2024-04-25T16:31:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.061+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5b24003c50 con 0x7f5b30107ce0 2024-04-25T16:31:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.061+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b2400f950 con 0x7f5b30107ce0 2024-04-25T16:31:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.061+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b301a7170 con 0x7f5b30107ce0 2024-04-25T16:31:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.061+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b301a75c0 con 0x7f5b30107ce0 2024-04-25T16:31:11.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.062+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5b30063d60 con 0x7f5b30107ce0 2024-04-25T16:31:11.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.063+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5b24029070 con 0x7f5b30107ce0 2024-04-25T16:31:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.065+0000 7f5b2dffb700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5b180afc90 0x7f5b180b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:11.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.065+0000 7f5b34ff1700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5b180afc90 0x7f5b180b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.065+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f5b240dfcb0 con 0x7f5b30107ce0 2024-04-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.065+0000 7f5b2dffb700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] conn(0x7f5b180bae40 0x7f5b180bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.065+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 --> [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b24027e50 con 0x7f5b180bae40 2024-04-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.065+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f5b240e0060 con 0x7f5b30107ce0 2024-04-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.065+0000 7f5b34ff1700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5b180afc90 0x7f5b180b21a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5b20011550 tx=0x7f5b20009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:11.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.066+0000 7f5b357f2700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] conn(0x7f5b180bae40 0x7f5b180bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:11.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.066+0000 7f5b357f2700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] conn(0x7f5b180bae40 0x7f5b180bd330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:11.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.067+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 <== osd.0 v2:172.21.15.98:6802/2011959581 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5b24027e50 con 0x7f5b180bae40 2024-04-25T16:31:11.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.091+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 --> [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5b30064160 con 0x7f5b180bae40 2024-04-25T16:31:11.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b2dffb700 1 -- 172.21.15.98:0/1447138974 <== osd.0 v2:172.21.15.98:6802/2011959581 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5b30064160 con 0x7f5b180bae40 2024-04-25T16:31:11.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] conn(0x7f5b180bae40 msgr2=0x7f5b180bd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:11.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] conn(0x7f5b180bae40 0x7f5b180bd330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:11.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5b180afc90 msgr2=0x7f5b180b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:11.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5b180afc90 0x7f5b180b21a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5b20011550 tx=0x7f5b20009260 comp rx=0 tx=0).stop 2024-04-25T16:31:11.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b30107ce0 msgr2=0x7f5b3019c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:11.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 --2- 172.21.15.98:0/1447138974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5b30107ce0 0x7f5b3019c030 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b240098e0 tx=0x7f5b24004270 comp rx=0 tx=0).stop 2024-04-25T16:31:11.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b357f2700 1 -- 172.21.15.98:0/1447138974 reap_dead start 2024-04-25T16:31:11.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 shutdown_connections 2024-04-25T16:31:11.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 >> 172.21.15.98:0/1447138974 conn(0x7f5b30078cb0 msgr2=0x7f5b301008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:11.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 shutdown_connections 2024-04-25T16:31:11.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:11.092+0000 7f5b37255700 1 -- 172.21.15.98:0/1447138974 wait complete. 2024-04-25T16:31:11.771 INFO:teuthology.orchestra.run.smithi098.stdout:111669149761 2024-04-25T16:31:11.771 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T16:31:11.990 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:11.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:11.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:11.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:12.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:12.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:12.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:12.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:12.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:12.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:12.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:12.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:13.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:12 smithi203 ceph-mon[33115]: pgmap v315: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-25T16:31:13.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:12 smithi098 ceph-mon[29162]: pgmap v315: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-25T16:31:13.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:12 smithi173 ceph-mon[32556]: pgmap v315: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 383 B/s wr, 8 op/s 2024-04-25T16:31:13.565 INFO:teuthology.orchestra.run.smithi098.stdout:34359738449 2024-04-25T16:31:13.566 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T16:31:13.675 INFO:teuthology.orchestra.run.smithi098.stdout:283467841565 2024-04-25T16:31:13.675 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T16:31:15.195 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:14 smithi098 ceph-mon[29162]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 383 B/s wr, 10 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T16:31:15.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:14 smithi173 ceph-mon[32556]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 383 B/s wr, 10 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T16:31:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1111254742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b201052f0 msgr2=0x7f2b20105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 --2- 172.21.15.98:0/1111254742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b201052f0 0x7f2b20105700 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2b10009990 tx=0x7f2b10009ca0 comp rx=0 tx=0).stop 2024-04-25T16:31:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1111254742 shutdown_connections 2024-04-25T16:31:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 --2- 172.21.15.98:0/1111254742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b2010e7b0 0x7f2b20110c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 --2- 172.21.15.98:0/1111254742 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b201052f0 0x7f2b20105700 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:15.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 --2- 172.21.15.98:0/1111254742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b20105cd0 0x7f2b2010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1111254742 >> 172.21.15.98:0/1111254742 conn(0x7f2b200fec50 msgr2=0x7f2b20101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1111254742 shutdown_connections 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.284+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1111254742 wait complete. 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.285+0000 7f2b27f6f700 1 Processor -- start 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.285+0000 7f2b27f6f700 1 -- start start 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.285+0000 7f2b27f6f700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b201052f0 0x7f2b20102fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.285+0000 7f2b27f6f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20105cd0 0x7f2b20103520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.285+0000 7f2b27f6f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b2010e7b0 0x7f2b20103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.285+0000 7f2b27f6f700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2b201141c0 con 0x7f2b20105cd0 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.285+0000 7f2b27f6f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2b20114040 con 0x7f2b2010e7b0 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.285+0000 7f2b27f6f700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2b20114340 con 0x7f2b201052f0 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2550a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20105cd0 0x7f2b20103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b25d0b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b201052f0 0x7f2b20102fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:15.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b25d0b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b201052f0 0x7f2b20102fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:60832/0 (socket says 172.21.15.98:60832) 2024-04-25T16:31:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b25d0b700 1 -- 172.21.15.98:0/1843524270 learned_addr learned my addr 172.21.15.98:0/1843524270 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2550a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20105cd0 0x7f2b20103520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:47938/0 (socket says 172.21.15.98:47938) 2024-04-25T16:31:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2650c700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b2010e7b0 0x7f2b20103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:15.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2550a700 1 -- 172.21.15.98:0/1843524270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b201052f0 msgr2=0x7f2b20102fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2550a700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b201052f0 0x7f2b20102fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2550a700 1 -- 172.21.15.98:0/1843524270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b2010e7b0 msgr2=0x7f2b20103a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2550a700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b2010e7b0 0x7f2b20103a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2550a700 1 -- 172.21.15.98:0/1843524270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b201a8f90 con 0x7f2b20105cd0 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b25d0b700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2b201052f0 0x7f2b20102fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b2550a700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20105cd0 0x7f2b20103520 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2b20076a10 tx=0x7f2b1c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c016070 con 0x7f2b20105cd0 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2b1c00fd20 con 0x7f2b20105cd0 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b201a9280 con 0x7f2b20105cd0 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c00e3b0 con 0x7f2b20105cd0 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.286+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b201a9810 con 0x7f2b20105cd0 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.287+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2b20063d60 con 0x7f2b20105cd0 2024-04-25T16:31:15.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.289+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2b1c019070 con 0x7f2b20105cd0 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.289+0000 7f2b16ffd700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2b0c0afc90 0x7f2b0c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.290+0000 7f2b25d0b700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2b0c0afc90 0x7f2b0c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.290+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f2b1c0def90 con 0x7f2b20105cd0 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.290+0000 7f2b16ffd700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] conn(0x7f2b0c0bae40 0x7f2b0c0bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.290+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 --> [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b1c015ec0 con 0x7f2b0c0bae40 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.290+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f2b1c0df3e0 con 0x7f2b20105cd0 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.290+0000 7f2b25d0b700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2b0c0afc90 0x7f2b0c0b21a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2b10003760 tx=0x7f2b10009230 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.290+0000 7f2b2650c700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] conn(0x7f2b0c0bae40 0x7f2b0c0bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.291+0000 7f2b2650c700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] conn(0x7f2b0c0bae40 0x7f2b0c0bd330 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.291+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 <== osd.5 v2:172.21.15.203:6808/4007044064 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2b1c015ec0 con 0x7f2b0c0bae40 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 --> [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2b20064160 con 0x7f2b0c0bae40 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b16ffd700 1 -- 172.21.15.98:0/1843524270 <== osd.5 v2:172.21.15.203:6808/4007044064 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2b20064160 con 0x7f2b0c0bae40 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 >> [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] conn(0x7f2b0c0bae40 msgr2=0x7f2b0c0bd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:15.299 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] conn(0x7f2b0c0bae40 0x7f2b0c0bd330 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2b0c0afc90 msgr2=0x7f2b0c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2b0c0afc90 0x7f2b0c0b21a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2b10003760 tx=0x7f2b10009230 comp rx=0 tx=0).stop 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20105cd0 msgr2=0x7f2b20103520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 --2- 172.21.15.98:0/1843524270 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2b20105cd0 0x7f2b20103520 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2b20076a10 tx=0x7f2b1c00dc30 comp rx=0 tx=0).stop 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b2650c700 1 -- 172.21.15.98:0/1843524270 reap_dead start 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 shutdown_connections 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.308+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 >> 172.21.15.98:0/1843524270 conn(0x7f2b200fec50 msgr2=0x7f2b200ff360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.309+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 shutdown_connections 2024-04-25T16:31:15.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:10.309+0000 7f2b27f6f700 1 -- 172.21.15.98:0/1843524270 wait complete. 2024-04-25T16:31:15.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:14 smithi203 ceph-mon[33115]: pgmap v316: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 383 B/s wr, 10 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T16:31:15.918 INFO:teuthology.orchestra.run.smithi098.stdout:158913790008 2024-04-25T16:31:15.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T16:31:16.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3640240322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc110ad0 msgr2=0x7f24cc112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:16.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 --2- 172.21.15.98:0/3640240322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc110ad0 0x7f24cc112ef0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f24c400b0a0 tx=0x7f24c400b3b0 comp rx=0 tx=0).stop 2024-04-25T16:31:16.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3640240322 shutdown_connections 2024-04-25T16:31:16.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 --2- 172.21.15.98:0/3640240322 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc110ad0 0x7f24cc112ef0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 --2- 172.21.15.98:0/3640240322 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24cc102f20 0x7f24cc103330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 --2- 172.21.15.98:0/3640240322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24cc103900 0x7f24cc110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3640240322 >> 172.21.15.98:0/3640240322 conn(0x7f24cc0fec90 msgr2=0x7f24cc1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3640240322 shutdown_connections 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.421+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3640240322 wait complete. 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.422+0000 7f24d2fb0700 1 Processor -- start 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.422+0000 7f24d2fb0700 1 -- start start 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.422+0000 7f24d2fb0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24cc102f20 0x7f24cc19cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24d2fb0700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc103900 0x7f24cc19d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24d2fb0700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24cc110ad0 0x7f24cc1a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24d2fb0700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1163e0 con 0x7f24cc103900 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24d2fb0700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc116260 con 0x7f24cc102f20 2024-04-25T16:31:16.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24d2fb0700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc116560 con 0x7f24cc110ad0 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc103900 0x7f24cc19d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc103900 0x7f24cc19d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58608/0 (socket says 172.21.15.98:58608) 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 -- 172.21.15.98:0/3167792827 learned_addr learned my addr 172.21.15.98:0/3167792827 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24d0d4c700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24cc102f20 0x7f24cc19cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24d154d700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24cc110ad0 0x7f24cc1a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 -- 172.21.15.98:0/3167792827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24cc110ad0 msgr2=0x7f24cc1a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24cc110ad0 0x7f24cc1a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 -- 172.21.15.98:0/3167792827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24cc102f20 msgr2=0x7f24cc19cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24cc102f20 0x7f24cc19cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 -- 172.21.15.98:0/3167792827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc1a1b90 con 0x7f24cc103900 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24cbfff700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc103900 0x7f24cc19d310 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f24b800c970 tx=0x7f24b800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.423+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8007910 con 0x7f24cc103900 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.424+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc1a1e20 con 0x7f24cc103900 2024-04-25T16:31:16.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.424+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24b800f450 con 0x7f24cc103900 2024-04-25T16:31:16.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.424+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8018660 con 0x7f24cc103900 2024-04-25T16:31:16.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.424+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc1a2360 con 0x7f24cc103900 2024-04-25T16:31:16.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.425+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f24cc063e20 con 0x7f24cc103900 2024-04-25T16:31:16.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.426+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f24b8007ab0 con 0x7f24cc103900 2024-04-25T16:31:16.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.427+0000 7f24c9ffb700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24bc0afc10 0x7f24bc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:16.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.427+0000 7f24d0d4c700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24bc0afc10 0x7f24bc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:16.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.427+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f24b80df380 con 0x7f24cc103900 2024-04-25T16:31:16.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.427+0000 7f24c9ffb700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] conn(0x7f24bc0badc0 0x7f24bc0bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:16.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.427+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 --> [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24b8019ec0 con 0x7f24bc0badc0 2024-04-25T16:31:16.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.427+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f24b80df7d0 con 0x7f24cc103900 2024-04-25T16:31:16.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.428+0000 7f24d154d700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] conn(0x7f24bc0badc0 0x7f24bc0bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:16.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.428+0000 7f24d0d4c700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24bc0afc10 0x7f24bc0b2120 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f24c0005950 tx=0x7f24c000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:16.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.428+0000 7f24d154d700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] conn(0x7f24bc0badc0 0x7f24bc0bd2b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:16.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.429+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 <== osd.7 v2:172.21.15.173:6816/442659395 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f24b8019ec0 con 0x7f24bc0badc0 2024-04-25T16:31:16.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.447+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 --> [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f24cc064220 con 0x7f24bc0badc0 2024-04-25T16:31:16.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.448+0000 7f24c9ffb700 1 -- 172.21.15.98:0/3167792827 <== osd.7 v2:172.21.15.173:6816/442659395 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f24cc064220 con 0x7f24bc0badc0 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 >> [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] conn(0x7f24bc0badc0 msgr2=0x7f24bc0bd2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] conn(0x7f24bc0badc0 0x7f24bc0bd2b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24bc0afc10 msgr2=0x7f24bc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24bc0afc10 0x7f24bc0b2120 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f24c0005950 tx=0x7f24c000b410 comp rx=0 tx=0).stop 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc103900 msgr2=0x7f24cc19d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 --2- 172.21.15.98:0/3167792827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24cc103900 0x7f24cc19d310 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f24b800c970 tx=0x7f24b800cc80 comp rx=0 tx=0).stop 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d154d700 1 -- 172.21.15.98:0/3167792827 reap_dead start 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 shutdown_connections 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 >> 172.21.15.98:0/3167792827 conn(0x7f24cc0fec90 msgr2=0x7f24cc112720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 shutdown_connections 2024-04-25T16:31:16.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.449+0000 7f24d2fb0700 1 -- 172.21.15.98:0/3167792827 wait complete. 2024-04-25T16:31:17.228 INFO:teuthology.orchestra.run.smithi098.stdout:206158430254 2024-04-25T16:31:17.228 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T16:31:17.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:16 smithi098 ceph-mon[29162]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T16:31:17.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:16 smithi173 ceph-mon[32556]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T16:31:17.370 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 -- 172.21.15.98:0/3639801869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 msgr2=0x7fac30104880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 --2- 172.21.15.98:0/3639801869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 0x7fac30104880 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fac240099c0 tx=0x7fac24009cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 -- 172.21.15.98:0/3639801869 shutdown_connections 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 --2- 172.21.15.98:0/3639801869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac301096f0 0x7fac30111050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 --2- 172.21.15.98:0/3639801869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 0x7fac30104880 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 --2- 172.21.15.98:0/3639801869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac30108da0 0x7fac301091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 -- 172.21.15.98:0/3639801869 >> 172.21.15.98:0/3639801869 conn(0x7fac300ffd00 msgr2=0x7fac30102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 -- 172.21.15.98:0/3639801869 shutdown_connections 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 -- 172.21.15.98:0/3639801869 wait complete. 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 Processor -- start 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.653+0000 7fac36b55700 1 -- start start 2024-04-25T16:31:17.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac36b55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 0x7fac301a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac36b55700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac30108da0 0x7fac301a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac36b55700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac301096f0 0x7fac3019aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac36b55700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fac301141d0 con 0x7fac30104470 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac36b55700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fac30114050 con 0x7fac30108da0 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac36b55700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fac30114350 con 0x7fac301096f0 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 0x7fac301a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 0x7fac301a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58620/0 (socket says 172.21.15.98:58620) 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 -- 172.21.15.98:0/4095222424 learned_addr learned my addr 172.21.15.98:0/4095222424 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac2ffff700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac30108da0 0x7fac301a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac350f2700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac301096f0 0x7fac3019aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 -- 172.21.15.98:0/4095222424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac301096f0 msgr2=0x7fac3019aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac301096f0 0x7fac3019aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 -- 172.21.15.98:0/4095222424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac30108da0 msgr2=0x7fac301a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:17.372 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac30108da0 0x7fac301a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 -- 172.21.15.98:0/4095222424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3019b070 con 0x7fac30104470 2024-04-25T16:31:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac348f1700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 0x7fac301a0810 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fac24005f50 tx=0x7fac240057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac24027070 con 0x7fac30104470 2024-04-25T16:31:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fac2400fc80 con 0x7fac30104470 2024-04-25T16:31:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.654+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac3019b300 con 0x7fac30104470 2024-04-25T16:31:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.655+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac24022d70 con 0x7fac30104470 2024-04-25T16:31:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.655+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac3019b830 con 0x7fac30104470 2024-04-25T16:31:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.655+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fac30063e20 con 0x7fac30104470 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.657+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fac2402a070 con 0x7fac30104470 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.658+0000 7fac2dffb700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fac200afc90 0x7fac200b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.658+0000 7fac2ffff700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fac200afc90 0x7fac200b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.658+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fac240e0520 con 0x7fac30104470 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.658+0000 7fac2dffb700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] conn(0x7fac200bae40 0x7fac200bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.658+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 --> [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac24004e50 con 0x7fac200bae40 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.658+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fac240e0970 con 0x7fac30104470 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.658+0000 7fac2ffff700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fac200afc90 0x7fac200b21a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fac30107470 tx=0x7fac1c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.658+0000 7fac350f2700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] conn(0x7fac200bae40 0x7fac200bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.659+0000 7fac350f2700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] conn(0x7fac200bae40 0x7fac200bd330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.660+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 <== osd.8 v2:172.21.15.203:6816/3847547758 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fac24004e50 con 0x7fac200bae40 2024-04-25T16:31:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.676+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 --> [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fac30064250 con 0x7fac200bae40 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac2dffb700 1 -- 172.21.15.98:0/4095222424 <== osd.8 v2:172.21.15.203:6816/3847547758 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fac30064250 con 0x7fac200bae40 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 >> [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] conn(0x7fac200bae40 msgr2=0x7fac200bd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] conn(0x7fac200bae40 0x7fac200bd330 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fac200afc90 msgr2=0x7fac200b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fac200afc90 0x7fac200b21a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fac30107470 tx=0x7fac1c00a380 comp rx=0 tx=0).stop 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 msgr2=0x7fac301a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 --2- 172.21.15.98:0/4095222424 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fac30104470 0x7fac301a0810 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fac24005f50 tx=0x7fac240057c0 comp rx=0 tx=0).stop 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac350f2700 1 -- 172.21.15.98:0/4095222424 reap_dead start 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 shutdown_connections 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 >> 172.21.15.98:0/4095222424 conn(0x7fac300ffd00 msgr2=0x7fac30102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 shutdown_connections 2024-04-25T16:31:17.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:16.677+0000 7fac36b55700 1 -- 172.21.15.98:0/4095222424 wait complete. 2024-04-25T16:31:17.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:16 smithi203 ceph-mon[33115]: pgmap v317: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-04-25T16:31:17.987 INFO:teuthology.orchestra.run.smithi098.stdout:231928234025 2024-04-25T16:31:17.987 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T16:31:18.017 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:18.018 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:18.022 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:18.022 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:18.026 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:18.818 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:18.819 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:19.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:18 smithi098 ceph-mon[29162]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-04-25T16:31:19.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:18 smithi173 ceph-mon[32556]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-04-25T16:31:19.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:18 smithi203 ceph-mon[33115]: pgmap v318: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-04-25T16:31:20.070 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:20.070 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:21.107 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:21 smithi098 ceph-mon[29162]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-04-25T16:31:21.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:21 smithi173 ceph-mon[32556]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-04-25T16:31:21.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:21 smithi203 ceph-mon[33115]: pgmap v319: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-04-25T16:31:22.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:22 smithi173 ceph-mon[32556]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T16:31:22.322 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:22 smithi098 ceph-mon[29162]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T16:31:22.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:22 smithi203 ceph-mon[33115]: pgmap v320: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T16:31:22.957 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:23.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.877+0000 7fa94650e700 1 -- 172.21.15.98:0/830624338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940109710 msgr2=0x7fa940111070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.877+0000 7fa94650e700 1 --2- 172.21.15.98:0/830624338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940109710 0x7fa940111070 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa93800b0d0 tx=0x7fa93800b3e0 comp rx=0 tx=0).stop 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.877+0000 7fa94650e700 1 -- 172.21.15.98:0/830624338 shutdown_connections 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.877+0000 7fa94650e700 1 --2- 172.21.15.98:0/830624338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa940104490 0x7fa9401048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.877+0000 7fa94650e700 1 --2- 172.21.15.98:0/830624338 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940109710 0x7fa940111070 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.877+0000 7fa94650e700 1 --2- 172.21.15.98:0/830624338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa940108dc0 0x7fa9401091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.877+0000 7fa94650e700 1 -- 172.21.15.98:0/830624338 >> 172.21.15.98:0/830624338 conn(0x7fa9400ffd40 msgr2=0x7fa940102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.878+0000 7fa94650e700 1 -- 172.21.15.98:0/830624338 shutdown_connections 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.878+0000 7fa94650e700 1 -- 172.21.15.98:0/830624338 wait complete. 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.878+0000 7fa94650e700 1 Processor -- start 2024-04-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.878+0000 7fa94650e700 1 -- start start 2024-04-25T16:31:23.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa94650e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940104490 0x7fa9401a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:23.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa94650e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa940108dc0 0x7fa9401a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa94650e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa940109710 0x7fa94019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa94650e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa940114260 con 0x7fa940104490 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa94650e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa9401140e0 con 0x7fa940108dc0 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa94650e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9401143e0 con 0x7fa940109710 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940104490 0x7fa9401a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940104490 0x7fa9401a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:58644/0 (socket says 172.21.15.98:58644) 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 -- 172.21.15.98:0/190410420 learned_addr learned my addr 172.21.15.98:0/190410420 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93f7fe700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa940108dc0 0x7fa9401a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa944aab700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa940109710 0x7fa94019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 -- 172.21.15.98:0/190410420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa940109710 msgr2=0x7fa94019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:23.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa940109710 0x7fa94019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 -- 172.21.15.98:0/190410420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa940108dc0 msgr2=0x7fa9401a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa940108dc0 0x7fa9401a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 -- 172.21.15.98:0/190410420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94019b040 con 0x7fa940104490 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.879+0000 7fa93ffff700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940104490 0x7fa9401a07e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa93000c9b0 tx=0x7fa93000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.880+0000 7fa93d7fa700 1 -- 172.21.15.98:0/190410420 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930007b00 con 0x7fa940104490 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.880+0000 7fa93d7fa700 1 -- 172.21.15.98:0/190410420 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa930007ca0 con 0x7fa940104490 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.880+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94019b330 con 0x7fa940104490 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.880+0000 7fa93d7fa700 1 -- 172.21.15.98:0/190410420 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930018660 con 0x7fa940104490 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.880+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9401a7670 con 0x7fa940104490 2024-04-25T16:31:23.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.881+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa940051190 con 0x7fa940104490 2024-04-25T16:31:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.882+0000 7fa93d7fa700 1 -- 172.21.15.98:0/190410420 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa93000f450 con 0x7fa940104490 2024-04-25T16:31:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.883+0000 7fa93d7fa700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa9280afc10 0x7fa9280b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.883+0000 7fa93f7fe700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa9280afc10 0x7fa9280b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:23.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.883+0000 7fa93d7fa700 1 -- 172.21.15.98:0/190410420 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fa9300df310 con 0x7fa940104490 2024-04-25T16:31:23.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.883+0000 7fa93f7fe700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa9280afc10 0x7fa9280b2120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa940103350 tx=0x7fa934009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:23.886 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:23.886+0000 7fa93d7fa700 1 -- 172.21.15.98:0/190410420 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa930014030 con 0x7fa940104490 2024-04-25T16:31:24.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.093+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa94002e510 con 0x7fa940104490 2024-04-25T16:31:24.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.093+0000 7fa93d7fa700 1 -- 172.21.15.98:0/190410420 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa9300a7b60 con 0x7fa940104490 2024-04-25T16:31:24.093 INFO:teuthology.orchestra.run.smithi098.stdout:34359738451 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa9280afc10 msgr2=0x7fa9280b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa9280afc10 0x7fa9280b2120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa940103350 tx=0x7fa934009190 comp rx=0 tx=0).stop 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940104490 msgr2=0x7fa9401a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940104490 0x7fa9401a07e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa93000c9b0 tx=0x7fa93000ccc0 comp rx=0 tx=0).stop 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 shutdown_connections 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa9280afc10 0x7fa9280b2120 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa940109710 0x7fa94019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa940104490 0x7fa9401a07e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 --2- 172.21.15.98:0/190410420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa940108dc0 0x7fa9401a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 >> 172.21.15.98:0/190410420 conn(0x7fa9400ffd40 msgr2=0x7fa94010a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:24.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 shutdown_connections 2024-04-25T16:31:24.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:24.095+0000 7fa94650e700 1 -- 172.21.15.98:0/190410420 wait complete. 2024-04-25T16:31:24.765 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738449 got 34359738451 for osd.0 2024-04-25T16:31:24.765 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:25.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:24 smithi098 ceph-mon[29162]: pgmap v321: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T16:31:25.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:24 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/190410420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:31:25.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:24 smithi173 ceph-mon[32556]: pgmap v321: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T16:31:25.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:24 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/190410420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:31:25.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:24 smithi203 ceph-mon[33115]: pgmap v321: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-04-25T16:31:25.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:24 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/190410420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:31:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.984+0000 7fef3980b700 1 -- 172.21.15.98:0/1433650858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef34105d10 msgr2=0x7fef3410e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.984+0000 7fef3980b700 1 --2- 172.21.15.98:0/1433650858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef34105d10 0x7fef3410e260 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fef28009990 tx=0x7fef28009ca0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.985+0000 7fef3980b700 1 -- 172.21.15.98:0/1433650858 shutdown_connections 2024-04-25T16:31:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.985+0000 7fef3980b700 1 --2- 172.21.15.98:0/1433650858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef341053c0 0x7fef341057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.985+0000 7fef3980b700 1 --2- 172.21.15.98:0/1433650858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef3410e7f0 0x7fef34110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.985+0000 7fef3980b700 1 --2- 172.21.15.98:0/1433650858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef34105d10 0x7fef3410e260 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.985+0000 7fef3980b700 1 -- 172.21.15.98:0/1433650858 >> 172.21.15.98:0/1433650858 conn(0x7fef34078cd0 msgr2=0x7fef34079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:26.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.985+0000 7fef3980b700 1 -- 172.21.15.98:0/1433650858 shutdown_connections 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.985+0000 7fef3980b700 1 -- 172.21.15.98:0/1433650858 wait complete. 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.985+0000 7fef3980b700 1 Processor -- start 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef3980b700 1 -- start start 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef3980b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef341053c0 0x7fef341030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef3980b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef34105d10 0x7fef341035f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef3980b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef3410e7f0 0x7fef34103b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef3980b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fef34114200 con 0x7fef3410e7f0 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef3980b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fef34114080 con 0x7fef34105d10 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef3980b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fef34114380 con 0x7fef341053c0 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef337fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef3410e7f0 0x7fef34103b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef327fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef34105d10 0x7fef341035f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef327fc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef34105d10 0x7fef341035f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:50592/0 (socket says 172.21.15.98:50592) 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef337fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef3410e7f0 0x7fef34103b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57780/0 (socket says 172.21.15.98:57780) 2024-04-25T16:31:26.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef327fc700 1 -- 172.21.15.98:0/301068623 learned_addr learned my addr 172.21.15.98:0/301068623 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef32ffd700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef341053c0 0x7fef341030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef327fc700 1 -- 172.21.15.98:0/301068623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef341053c0 msgr2=0x7fef341030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef327fc700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef341053c0 0x7fef341030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef327fc700 1 -- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef3410e7f0 msgr2=0x7fef34103b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef327fc700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef3410e7f0 0x7fef34103b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef327fc700 1 -- 172.21.15.98:0/301068623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef341a8f90 con 0x7fef34105d10 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.986+0000 7fef337fe700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef3410e7f0 0x7fef34103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.987+0000 7fef327fc700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef34105d10 0x7fef341035f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef28006010 tx=0x7fef2800bfc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.987+0000 7fef38809700 1 -- 172.21.15.98:0/301068623 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28027070 con 0x7fef34105d10 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.987+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef341a9220 con 0x7fef34105d10 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.987+0000 7fef38809700 1 -- 172.21.15.98:0/301068623 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fef28022470 con 0x7fef34105d10 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.987+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef341a9750 con 0x7fef34105d10 2024-04-25T16:31:26.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.987+0000 7fef38809700 1 -- 172.21.15.98:0/301068623 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef28004a30 con 0x7fef34105d10 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.988+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef34051190 con 0x7fef34105d10 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.989+0000 7fef38809700 1 -- 172.21.15.98:0/301068623 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fef28009d10 con 0x7fef34105d10 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.990+0000 7fef38809700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef1c0afc10 0x7fef1c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.990+0000 7fef32ffd700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef1c0afc10 0x7fef1c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.990+0000 7fef38809700 1 -- 172.21.15.98:0/301068623 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fef280dfff0 con 0x7fef34105d10 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.991+0000 7fef32ffd700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef1c0afc10 0x7fef1c0b2120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fef24005950 tx=0x7fef2400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:25.994+0000 7fef38809700 1 -- 172.21.15.98:0/301068623 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef28030080 con 0x7fef34105d10 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.202+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fef3402e510 con 0x7fef34105d10 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.203+0000 7fef38809700 1 -- 172.21.15.98:0/301068623 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fef28013070 con 0x7fef34105d10 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef1c0afc10 msgr2=0x7fef1c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef1c0afc10 0x7fef1c0b2120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fef24005950 tx=0x7fef2400a300 comp rx=0 tx=0).stop 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef34105d10 msgr2=0x7fef341035f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef34105d10 0x7fef341035f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fef28006010 tx=0x7fef2800bfc0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 shutdown_connections 2024-04-25T16:31:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fef1c0afc10 0x7fef1c0b2120 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fef341053c0 0x7fef341030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fef3410e7f0 0x7fef34103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 --2- 172.21.15.98:0/301068623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fef34105d10 0x7fef341035f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 >> 172.21.15.98:0/301068623 conn(0x7fef34078cd0 msgr2=0x7fef340fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 shutdown_connections 2024-04-25T16:31:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:26.205+0000 7fef3980b700 1 -- 172.21.15.98:0/301068623 wait complete. 2024-04-25T16:31:26.406 INFO:teuthology.orchestra.run.smithi098.stdout:77309411402 2024-04-25T16:31:27.213 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:26 smithi098 ceph-mon[29162]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T16:31:27.214 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:26 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/301068623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:31:27.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:26 smithi173 ceph-mon[32556]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T16:31:27.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:26 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/301068623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:31:27.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:26 smithi203 ceph-mon[33115]: pgmap v322: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-04-25T16:31:27.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:26 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/301068623' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:31:29.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:29 smithi098 ceph-mon[29162]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-25T16:31:29.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:29 smithi173 ceph-mon[32556]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-25T16:31:29.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:29 smithi203 ceph-mon[33115]: pgmap v323: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-04-25T16:31:30.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:30 smithi098 ceph-mon[29162]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:31:30.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:30 smithi203 ceph-mon[33115]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:31:30.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:30 smithi173 ceph-mon[32556]: pgmap v324: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:31:30.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.979+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1058512146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908103390 msgr2=0x7fa9081037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.979+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1058512146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908103390 0x7fa9081037a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa9040099c0 tx=0x7fa904009cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.979+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1058512146 shutdown_connections 2024-04-25T16:31:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.979+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1058512146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908108610 0x7fa90810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.979+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1058512146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908103390 0x7fa9081037a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.979+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1058512146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa908107cc0 0x7fa9081080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.979+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1058512146 >> 172.21.15.98:0/1058512146 conn(0x7fa9080fec90 msgr2=0x7fa9081010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.979+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1058512146 shutdown_connections 2024-04-25T16:31:30.981 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.980+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1058512146 wait complete. 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90f5ba700 1 Processor -- start 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90f5ba700 1 -- start start 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90f5ba700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908107cc0 0x7fa9081a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90f5ba700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908108610 0x7fa9081a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90f5ba700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa90819ab50 0x7fa90819af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90f5ba700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa908113140 con 0x7fa908108610 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90f5ba700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa908112fc0 con 0x7fa90819ab50 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90f5ba700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa9081132c0 con 0x7fa908107cc0 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90cb55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908108610 0x7fa9081a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90cb55700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908108610 0x7fa9081a0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57826/0 (socket says 172.21.15.98:57826) 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90cb55700 1 -- 172.21.15.98:0/1175127851 learned_addr learned my addr 172.21.15.98:0/1175127851 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:30.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.981+0000 7fa90d356700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908107cc0 0x7fa9081a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa90cb55700 1 -- 172.21.15.98:0/1175127851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908107cc0 msgr2=0x7fa9081a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa90cb55700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908107cc0 0x7fa9081a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa90cb55700 1 -- 172.21.15.98:0/1175127851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa90819ab50 msgr2=0x7fa90819af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa90cb55700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa90819ab50 0x7fa90819af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa90cb55700 1 -- 172.21.15.98:0/1175127851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90819b770 con 0x7fa908108610 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa90cb55700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908108610 0x7fa9081a0fc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800e970 tx=0x7fa8f800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa8fe7fc700 1 -- 172.21.15.98:0/1175127851 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f800cbd0 con 0x7fa908108610 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9081a7770 con 0x7fa908108610 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9081a7bd0 con 0x7fa908108610 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa8fe7fc700 1 -- 172.21.15.98:0/1175127851 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa8f800cd70 con 0x7fa908108610 2024-04-25T16:31:30.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.982+0000 7fa8fe7fc700 1 -- 172.21.15.98:0/1175127851 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8005560 con 0x7fa908108610 2024-04-25T16:31:30.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.983+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8ec005360 con 0x7fa908108610 2024-04-25T16:31:30.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.986+0000 7fa8fe7fc700 1 -- 172.21.15.98:0/1175127851 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa8f8004750 con 0x7fa908108610 2024-04-25T16:31:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.987+0000 7fa8fe7fc700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa8f40afc10 0x7fa8f40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.987+0000 7fa90d356700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa8f40afc10 0x7fa8f40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:30.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.987+0000 7fa8fe7fc700 1 -- 172.21.15.98:0/1175127851 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fa8f80dfa60 con 0x7fa908108610 2024-04-25T16:31:30.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.987+0000 7fa90d356700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa8f40afc10 0x7fa8f40b2120 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa904009990 tx=0x7fa90401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:30.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.990+0000 7fa8fe7fc700 1 -- 172.21.15.98:0/1175127851 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8f80a8180 con 0x7fa908108610 2024-04-25T16:31:31.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.209+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa8ec005920 con 0x7fa908108610 2024-04-25T16:31:31.210 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:31 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/813506965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:31:31.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.209+0000 7fa8fe7fc700 1 -- 172.21.15.98:0/1175127851 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa8f80b0030 con 0x7fa908108610 2024-04-25T16:31:31.210 INFO:teuthology.orchestra.run.smithi098.stdout:184683593783 2024-04-25T16:31:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.211+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa8f40afc10 msgr2=0x7fa8f40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.211+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa8f40afc10 0x7fa8f40b2120 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa904009990 tx=0x7fa90401a040 comp rx=0 tx=0).stop 2024-04-25T16:31:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.211+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908108610 msgr2=0x7fa9081a0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.211+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908108610 0x7fa9081a0fc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800e970 tx=0x7fa8f800ec80 comp rx=0 tx=0).stop 2024-04-25T16:31:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.211+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 shutdown_connections 2024-04-25T16:31:31.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.212+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa8f40afc10 0x7fa8f40b2120 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.212+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908107cc0 0x7fa9081a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.212+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa90819ab50 0x7fa90819af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.212+0000 7fa90f5ba700 1 --2- 172.21.15.98:0/1175127851 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa908108610 0x7fa9081a0fc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.212+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 >> 172.21.15.98:0/1175127851 conn(0x7fa9080fec90 msgr2=0x7fa90806b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.212+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 shutdown_connections 2024-04-25T16:31:31.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.212+0000 7fa90f5ba700 1 -- 172.21.15.98:0/1175127851 wait complete. 2024-04-25T16:31:31.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:31 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/813506965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:31:31.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:31 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/813506965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:31:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:32 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1175127851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:31:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:32 smithi203 ceph-mon[33115]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:31:32.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:32 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1175127851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:31:32.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:32 smithi098 ceph-mon[29162]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:31:32.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:32 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1175127851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:31:32.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:32 smithi173 ceph-mon[32556]: pgmap v325: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:31:33.214 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593778 got 184683593783 for osd.6 2024-04-25T16:31:33.214 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 -- 172.21.15.98:0/3106992134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0077430 msgr2=0x7f72d0110540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 --2- 172.21.15.98:0/3106992134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0077430 0x7f72d0110540 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009990 tx=0x7f72c4009ca0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 -- 172.21.15.98:0/3106992134 shutdown_connections 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 --2- 172.21.15.98:0/3106992134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d0110a80 0x7f72d0112ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 --2- 172.21.15.98:0/3106992134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d0076ae0 0x7f72d0076ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 --2- 172.21.15.98:0/3106992134 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0077430 0x7f72d0110540 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 -- 172.21.15.98:0/3106992134 >> 172.21.15.98:0/3106992134 conn(0x7f72d00ffd00 msgr2=0x7f72d0102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 -- 172.21.15.98:0/3106992134 shutdown_connections 2024-04-25T16:31:33.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.881+0000 7f72d55f9700 1 -- 172.21.15.98:0/3106992134 wait complete. 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.882+0000 7f72d55f9700 1 Processor -- start 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.882+0000 7f72d55f9700 1 -- start start 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.882+0000 7f72d55f9700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d0076ae0 0x7f72d019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.882+0000 7f72d55f9700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d0077430 0x7f72d019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.882+0000 7f72d55f9700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0110a80 0x7f72d01a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.882+0000 7f72d55f9700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f72d01163f0 con 0x7f72d0110a80 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.882+0000 7f72d55f9700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0116270 con 0x7f72d0076ae0 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.882+0000 7f72d55f9700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f72d0116570 con 0x7f72d0077430 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0110a80 0x7f72d01a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72ce7fc700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d0077430 0x7f72d019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0110a80 0x7f72d01a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57798/0 (socket says 172.21.15.98:57798) 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 -- 172.21.15.98:0/813506965 learned_addr learned my addr 172.21.15.98:0/813506965 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:33.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72ceffd700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d0076ae0 0x7f72d019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:33.794 INFO:teuthology.orchestra.run.smithi098.stdout:137438953537 2024-04-25T16:31:33.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 -- 172.21.15.98:0/813506965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d0077430 msgr2=0x7f72d019d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:33.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d0077430 0x7f72d019d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 -- 172.21.15.98:0/813506965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d0076ae0 msgr2=0x7f72d019ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:33.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d0076ae0 0x7f72d019ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 -- 172.21.15.98:0/813506965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d01a1d30 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72ce7fc700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d0077430 0x7f72d019d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72cf7fe700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0110a80 0x7f72d01a1690 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f72c800ef90 tx=0x7f72c800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72b7fff700 1 -- 172.21.15.98:0/813506965 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c80092e0 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d01a1fc0 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72b7fff700 1 -- 172.21.15.98:0/813506965 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f72c8004bb0 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.883+0000 7f72b7fff700 1 -- 172.21.15.98:0/813506965 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8004040 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.884+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d01a24d0 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.885+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d0051190 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.886+0000 7f72b7fff700 1 -- 172.21.15.98:0/813506965 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f72c801b030 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.887+0000 7f72b7fff700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f72b80afc10 0x7f72b80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.887+0000 7f72ceffd700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f72b80afc10 0x7f72b80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.887+0000 7f72b7fff700 1 -- 172.21.15.98:0/813506965 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f72c80de960 con 0x7f72d0110a80 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.887+0000 7f72ceffd700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f72b80afc10 0x7f72b80b2120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72c0005950 tx=0x7f72c00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:33.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:30.896+0000 7f72b7fff700 1 -- 172.21.15.98:0/813506965 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72c8010040 con 0x7f72d0110a80 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.113+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f72d002e510 con 0x7f72d0110a80 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.114+0000 7f72b7fff700 1 -- 172.21.15.98:0/813506965 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f72c80a7110 con 0x7f72d0110a80 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f72b80afc10 msgr2=0x7f72b80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f72b80afc10 0x7f72b80b2120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72c0005950 tx=0x7f72c00058e0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0110a80 msgr2=0x7f72d01a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0110a80 0x7f72d01a1690 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f72c800ef90 tx=0x7f72c800c3a0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 shutdown_connections 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f72b80afc10 0x7f72b80b2120 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f72d0110a80 0x7f72d01a1690 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72d0076ae0 0x7f72d019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 --2- 172.21.15.98:0/813506965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f72d0077430 0x7f72d019d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 >> 172.21.15.98:0/813506965 conn(0x7f72d00ffd00 msgr2=0x7f72d0100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 shutdown_connections 2024-04-25T16:31:33.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:31.116+0000 7f72d55f9700 1 -- 172.21.15.98:0/813506965 wait complete. 2024-04-25T16:31:35.109 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411402 for osd.2 2024-04-25T16:31:35.109 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:35.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:35 smithi098 ceph-mon[29162]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:31:35.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:35 smithi173 ceph-mon[32556]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:31:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.353+0000 7f38aa6c3700 1 -- 172.21.15.98:0/2699354146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a410e7a0 msgr2=0x7f38a4110c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.353+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/2699354146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a410e7a0 0x7f38a4110c80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f389c00b0a0 tx=0x7f389c00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.353+0000 7f38aa6c3700 1 -- 172.21.15.98:0/2699354146 shutdown_connections 2024-04-25T16:31:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.353+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/2699354146 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38a4105330 0x7f38a4105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.353+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/2699354146 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a410e7a0 0x7f38a4110c80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.353+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/2699354146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38a4105c80 0x7f38a410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.353+0000 7f38aa6c3700 1 -- 172.21.15.98:0/2699354146 >> 172.21.15.98:0/2699354146 conn(0x7f38a40fec90 msgr2=0x7f38a41010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:35.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.353+0000 7f38aa6c3700 1 -- 172.21.15.98:0/2699354146 shutdown_connections 2024-04-25T16:31:35.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.354+0000 7f38aa6c3700 1 -- 172.21.15.98:0/2699354146 wait complete. 2024-04-25T16:31:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.354+0000 7f38aa6c3700 1 Processor -- start 2024-04-25T16:31:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.354+0000 7f38aa6c3700 1 -- start start 2024-04-25T16:31:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38aa6c3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4105330 0x7f38a419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38aa6c3700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38a4105c80 0x7f38a419d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38aa6c3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38a410e7a0 0x7f38a41a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38aa6c3700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38a41141b0 con 0x7f38a4105330 2024-04-25T16:31:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38aa6c3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f38a4114030 con 0x7f38a410e7a0 2024-04-25T16:31:35.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38aa6c3700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f38a4114330 con 0x7f38a4105c80 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4105330 0x7f38a419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4105330 0x7f38a419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:57840/0 (socket says 172.21.15.98:57840) 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 -- 172.21.15.98:0/1790642576 learned_addr learned my addr 172.21.15.98:0/1790642576 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a37fe700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38a4105c80 0x7f38a419d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a8c60700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38a410e7a0 0x7f38a41a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 -- 172.21.15.98:0/1790642576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38a4105c80 msgr2=0x7f38a419d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38a4105c80 0x7f38a419d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 -- 172.21.15.98:0/1790642576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38a410e7a0 msgr2=0x7f38a41a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38a410e7a0 0x7f38a41a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 -- 172.21.15.98:0/1790642576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a41a1c30 con 0x7f38a4105330 2024-04-25T16:31:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.355+0000 7f38a3fff700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4105330 0x7f38a419cd60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f389400d950 tx=0x7f389400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.356+0000 7f38a17fa700 1 -- 172.21.15.98:0/1790642576 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3894016070 con 0x7f38a4105330 2024-04-25T16:31:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.356+0000 7f38a17fa700 1 -- 172.21.15.98:0/1790642576 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f389400fd20 con 0x7f38a4105330 2024-04-25T16:31:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.356+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a41a1ec0 con 0x7f38a4105330 2024-04-25T16:31:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.356+0000 7f38a17fa700 1 -- 172.21.15.98:0/1790642576 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389400e3b0 con 0x7f38a4105330 2024-04-25T16:31:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.356+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a41a9840 con 0x7f38a4105330 2024-04-25T16:31:35.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.357+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a4051190 con 0x7f38a4105330 2024-04-25T16:31:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.359+0000 7f38a17fa700 1 -- 172.21.15.98:0/1790642576 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3894019070 con 0x7f38a4105330 2024-04-25T16:31:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.359+0000 7f38a17fa700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f388c0afc90 0x7f388c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.360+0000 7f38a37fe700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f388c0afc90 0x7f388c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.360+0000 7f38a17fa700 1 -- 172.21.15.98:0/1790642576 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f38940deb40 con 0x7f38a4105330 2024-04-25T16:31:35.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.360+0000 7f38a37fe700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f388c0afc90 0x7f388c0b21a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f38a41049c0 tx=0x7f389800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:35.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.363+0000 7f38a17fa700 1 -- 172.21.15.98:0/1790642576 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38940a7260 con 0x7f38a4105330 2024-04-25T16:31:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:35 smithi203 ceph-mon[33115]: pgmap v326: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:31:35.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 -- 172.21.15.98:0/448760685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63181085e0 msgr2=0x7f631810ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 --2- 172.21.15.98:0/448760685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63181085e0 0x7f631810ff40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f631000b0d0 tx=0x7f631000b3e0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 -- 172.21.15.98:0/448760685 shutdown_connections 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 --2- 172.21.15.98:0/448760685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6318103360 0x7f6318103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 --2- 172.21.15.98:0/448760685 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f63181085e0 0x7f631810ff40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 --2- 172.21.15.98:0/448760685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6318107c90 0x7f63181080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 -- 172.21.15.98:0/448760685 >> 172.21.15.98:0/448760685 conn(0x7f63180fec90 msgr2=0x7f63181010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 -- 172.21.15.98:0/448760685 shutdown_connections 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.554+0000 7f631e381700 1 -- 172.21.15.98:0/448760685 wait complete. 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.555+0000 7f631e381700 1 Processor -- start 2024-04-25T16:31:35.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.555+0000 7f631e381700 1 -- start start 2024-04-25T16:31:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f631e381700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6318103360 0x7f631819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f631e381700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6318107c90 0x7f631819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f631e381700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63181085e0 0x7f63181998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f631e381700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f63181130c0 con 0x7f6318107c90 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f631e381700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6318112f40 con 0x7f6318103360 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f631e381700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6318113240 con 0x7f63181085e0 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6318107c90 0x7f631819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6318107c90 0x7f631819fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41550/0 (socket says 172.21.15.98:41550) 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 -- 172.21.15.98:0/3179711609 learned_addr learned my addr 172.21.15.98:0/3179711609 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f6317fff700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6318103360 0x7f631819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f631c91e700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63181085e0 0x7f63181998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 -- 172.21.15.98:0/3179711609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63181085e0 msgr2=0x7f63181998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63181085e0 0x7f63181998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 -- 172.21.15.98:0/3179711609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6318103360 msgr2=0x7f631819f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6318103360 0x7f631819f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 -- 172.21.15.98:0/3179711609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6318199f00 con 0x7f6318107c90 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f631c91e700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63181085e0 0x7f63181998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63177fe700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6318107c90 0x7f631819fbe0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f630c00d980 tx=0x7f630c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.556+0000 7f63157fa700 1 -- 172.21.15.98:0/3179711609 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c017070 con 0x7f6318107c90 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.557+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631819a1f0 con 0x7f6318107c90 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.557+0000 7f63157fa700 1 -- 172.21.15.98:0/3179711609 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f630c009dc0 con 0x7f6318107c90 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.557+0000 7f63157fa700 1 -- 172.21.15.98:0/3179711609 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630c00f3c0 con 0x7f6318107c90 2024-04-25T16:31:35.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.557+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631806b4e0 con 0x7f6318107c90 2024-04-25T16:31:35.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.558+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6318104810 con 0x7f6318107c90 2024-04-25T16:31:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.559+0000 7f63157fa700 1 -- 172.21.15.98:0/3179711609 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f630c00f560 con 0x7f6318107c90 2024-04-25T16:31:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.560+0000 7f63157fa700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63000afc90 0x7f63000b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.560+0000 7f6317fff700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63000afc90 0x7f63000b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.560+0000 7f63157fa700 1 -- 172.21.15.98:0/3179711609 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f630c0deb60 con 0x7f6318107c90 2024-04-25T16:31:35.561 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.560+0000 7f6317fff700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63000afc90 0x7f63000b21a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f631819aa00 tx=0x7f630800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:35.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.569+0000 7f63157fa700 1 -- 172.21.15.98:0/3179711609 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f630c0a7280 con 0x7f6318107c90 2024-04-25T16:31:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.579+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f38a402e510 con 0x7f38a4105330 2024-04-25T16:31:35.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.579+0000 7f38a17fa700 1 -- 172.21.15.98:0/1790642576 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f38940a6b60 con 0x7f38a4105330 2024-04-25T16:31:35.579 INFO:teuthology.orchestra.run.smithi098.stdout:283467841571 2024-04-25T16:31:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.581+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f388c0afc90 msgr2=0x7f388c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.581+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f388c0afc90 0x7f388c0b21a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f38a41049c0 tx=0x7f389800b410 comp rx=0 tx=0).stop 2024-04-25T16:31:35.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.581+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4105330 msgr2=0x7f38a419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.581+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4105330 0x7f38a419cd60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f389400d950 tx=0x7f389400dc60 comp rx=0 tx=0).stop 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.582+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 shutdown_connections 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.582+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f388c0afc90 0x7f388c0b21a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.582+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38a410e7a0 0x7f38a41a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.582+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38a4105330 0x7f38a419cd60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.582+0000 7f38aa6c3700 1 --2- 172.21.15.98:0/1790642576 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38a4105c80 0x7f38a419d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.582+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 >> 172.21.15.98:0/1790642576 conn(0x7f38a40fec90 msgr2=0x7f38a4103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.582+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 shutdown_connections 2024-04-25T16:31:35.582 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.582+0000 7f38aa6c3700 1 -- 172.21.15.98:0/1790642576 wait complete. 2024-04-25T16:31:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.778+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f631819ae20 con 0x7f6318107c90 2024-04-25T16:31:35.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.778+0000 7f63157fa700 1 -- 172.21.15.98:0/3179711609 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f631819ae20 con 0x7f6318107c90 2024-04-25T16:31:35.778 INFO:teuthology.orchestra.run.smithi098.stdout:309237645342 2024-04-25T16:31:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.780+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63000afc90 msgr2=0x7f63000b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.780+0000 7f631e381700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63000afc90 0x7f63000b21a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f631819aa00 tx=0x7f630800a380 comp rx=0 tx=0).stop 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.780+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6318107c90 msgr2=0x7f631819fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.780+0000 7f631e381700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6318107c90 0x7f631819fbe0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f630c00d980 tx=0x7f630c00dc90 comp rx=0 tx=0).stop 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.781+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 shutdown_connections 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.781+0000 7f631e381700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63000afc90 0x7f63000b21a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.781+0000 7f631e381700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63181085e0 0x7f63181998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.781+0000 7f631e381700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6318103360 0x7f631819f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.781+0000 7f631e381700 1 --2- 172.21.15.98:0/3179711609 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6318107c90 0x7f631819fbe0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.781+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 >> 172.21.15.98:0/3179711609 conn(0x7f63180fec90 msgr2=0x7f6318109540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.781+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 shutdown_connections 2024-04-25T16:31:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.781+0000 7f631e381700 1 -- 172.21.15.98:0/3179711609 wait complete. 2024-04-25T16:31:35.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.876+0000 7f8a02579700 1 -- 172.21.15.98:0/652492283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc107c90 msgr2=0x7f89fc1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.876+0000 7f8a02579700 1 --2- 172.21.15.98:0/652492283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc107c90 0x7f89fc1080a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f89f00099c0 tx=0x7f89f0009cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.876+0000 7f8a02579700 1 -- 172.21.15.98:0/652492283 shutdown_connections 2024-04-25T16:31:35.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.876+0000 7f8a02579700 1 --2- 172.21.15.98:0/652492283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89fc1085e0 0x7f89fc10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.876+0000 7f8a02579700 1 --2- 172.21.15.98:0/652492283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89fc103360 0x7f89fc103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.876+0000 7f8a02579700 1 --2- 172.21.15.98:0/652492283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc107c90 0x7f89fc1080a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.877+0000 7f8a02579700 1 -- 172.21.15.98:0/652492283 >> 172.21.15.98:0/652492283 conn(0x7f89fc0fec90 msgr2=0x7f89fc1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:35.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.877+0000 7f8a02579700 1 -- 172.21.15.98:0/652492283 shutdown_connections 2024-04-25T16:31:35.877 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.877+0000 7f8a02579700 1 -- 172.21.15.98:0/652492283 wait complete. 2024-04-25T16:31:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.877+0000 7f8a02579700 1 Processor -- start 2024-04-25T16:31:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.877+0000 7f8a02579700 1 -- start start 2024-04-25T16:31:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f8a02579700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc103360 0x7f89fc19f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f8a02579700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89fc107c90 0x7f89fc19fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f8a02579700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89fc1085e0 0x7f89fc199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f8a02579700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc1130f0 con 0x7f89fc103360 2024-04-25T16:31:35.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f8a02579700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc112f70 con 0x7f89fc107c90 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f8a02579700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f89fc113270 con 0x7f89fc1085e0 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f89fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc103360 0x7f89fc19f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f89fbfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc103360 0x7f89fc19f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41572/0 (socket says 172.21.15.98:41572) 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f89fbfff700 1 -- 172.21.15.98:0/4135065455 learned_addr learned my addr 172.21.15.98:0/4135065455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f89fb7fe700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89fc107c90 0x7f89fc19fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.878+0000 7f8a00b16700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89fc1085e0 0x7f89fc199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89fb7fe700 1 -- 172.21.15.98:0/4135065455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89fc1085e0 msgr2=0x7f89fc199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89fb7fe700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89fc1085e0 0x7f89fc199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89fb7fe700 1 -- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc103360 msgr2=0x7f89fc19f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89fb7fe700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc103360 0x7f89fc19f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89fb7fe700 1 -- 172.21.15.98:0/4135065455 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc19a000 con 0x7f89fc107c90 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89fbfff700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc103360 0x7f89fc19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:35.880 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89fb7fe700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89fc107c90 0x7f89fc19fc20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f89f00098e0 tx=0x7f89f0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89f97fa700 1 -- 172.21.15.98:0/4135065455 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0026070 con 0x7f89fc107c90 2024-04-25T16:31:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89fc19a290 con 0x7f89fc107c90 2024-04-25T16:31:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89fc1a6570 con 0x7f89fc107c90 2024-04-25T16:31:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89f97fa700 1 -- 172.21.15.98:0/4135065455 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f89f0003c50 con 0x7f89fc107c90 2024-04-25T16:31:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.879+0000 7f89f97fa700 1 -- 172.21.15.98:0/4135065455 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f000f900 con 0x7f89fc107c90 2024-04-25T16:31:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.880+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89fc104810 con 0x7f89fc107c90 2024-04-25T16:31:35.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.882+0000 7f89f97fa700 1 -- 172.21.15.98:0/4135065455 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f89f0029070 con 0x7f89fc107c90 2024-04-25T16:31:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.883+0000 7f89f97fa700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f89e40afc90 0x7f89e40b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.883+0000 7f89fbfff700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f89e40afc90 0x7f89e40b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.883+0000 7f89f97fa700 1 -- 172.21.15.98:0/4135065455 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f89f00dfa90 con 0x7f89fc107c90 2024-04-25T16:31:35.882 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.883+0000 7f89fbfff700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f89e40afc90 0x7f89e40b21a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f89fc1084e0 tx=0x7f89ec009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:35.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:35.886+0000 7f89f97fa700 1 -- 172.21.15.98:0/4135065455 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89f0013070 con 0x7f89fc107c90 2024-04-25T16:31:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.091+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f89fc02e510 con 0x7f89fc107c90 2024-04-25T16:31:36.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.092+0000 7f89f97fa700 1 -- 172.21.15.98:0/4135065455 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f89f00a81e0 con 0x7f89fc107c90 2024-04-25T16:31:36.092 INFO:teuthology.orchestra.run.smithi098.stdout:111669149766 2024-04-25T16:31:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f89e40afc90 msgr2=0x7f89e40b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f89e40afc90 0x7f89e40b21a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f89fc1084e0 tx=0x7f89ec009260 comp rx=0 tx=0).stop 2024-04-25T16:31:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89fc107c90 msgr2=0x7f89fc19fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89fc107c90 0x7f89fc19fc20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f89f00098e0 tx=0x7f89f0004270 comp rx=0 tx=0).stop 2024-04-25T16:31:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 shutdown_connections 2024-04-25T16:31:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f89e40afc90 0x7f89e40b21a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89fc1085e0 0x7f89fc199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f89fc103360 0x7f89fc19f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 --2- 172.21.15.98:0/4135065455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89fc107c90 0x7f89fc19fc20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.094+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 >> 172.21.15.98:0/4135065455 conn(0x7f89fc0fec90 msgr2=0x7f89fc1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.095+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 shutdown_connections 2024-04-25T16:31:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:36.095+0000 7f8a02579700 1 -- 172.21.15.98:0/4135065455 wait complete. 2024-04-25T16:31:36.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1790642576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:31:36.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3179711609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:31:36.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1790642576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:31:36.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3179711609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:31:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1790642576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:31:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3179711609' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:31:36.534 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645336 got 309237645342 for osd.11 2024-04-25T16:31:36.534 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:37.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:37 smithi098 ceph-mon[29162]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:31:37.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:37 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4135065455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:31:37.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:37 smithi173 ceph-mon[32556]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:31:37.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:37 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4135065455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:31:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:37 smithi203 ceph-mon[33115]: pgmap v327: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:31:37.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:37 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4135065455' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:31:38.248 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841565 got 283467841571 for osd.10 2024-04-25T16:31:38.249 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:38.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:38 smithi173 ceph-mon[32556]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:31:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:38 smithi203 ceph-mon[33115]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:31:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:38 smithi098 ceph-mon[29162]: pgmap v328: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:31:38.654 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953532 got 137438953537 for osd.4 2024-04-25T16:31:38.655 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 -- 172.21.15.98:0/2120641518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c54103000 msgr2=0x7f7c54110ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 --2- 172.21.15.98:0/2120641518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c54103000 0x7f7c54110ff0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00b0a0 tx=0x7f7c4c00b3b0 comp rx=0 tx=0).stop 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 -- 172.21.15.98:0/2120641518 shutdown_connections 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 --2- 172.21.15.98:0/2120641518 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c54103000 0x7f7c54110ff0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 --2- 172.21.15.98:0/2120641518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c5410a2b0 0x7f7c5410a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 --2- 172.21.15.98:0/2120641518 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c541026b0 0x7f7c54102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 -- 172.21.15.98:0/2120641518 >> 172.21.15.98:0/2120641518 conn(0x7f7c540fe5c0 msgr2=0x7f7c541009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 -- 172.21.15.98:0/2120641518 shutdown_connections 2024-04-25T16:31:38.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.885+0000 7f7c59731700 1 -- 172.21.15.98:0/2120641518 wait complete. 2024-04-25T16:31:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.886+0000 7f7c59731700 1 Processor -- start 2024-04-25T16:31:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.886+0000 7f7c59731700 1 -- start start 2024-04-25T16:31:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.886+0000 7f7c59731700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c541026b0 0x7f7c541a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.886+0000 7f7c59731700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103000 0x7f7c541a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.886+0000 7f7c59731700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5410a2b0 0x7f7c5419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:38.887 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.886+0000 7f7c59731700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7c54114280 con 0x7f7c5410a2b0 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.886+0000 7f7c59731700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7c54114100 con 0x7f7c54103000 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.886+0000 7f7c59731700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7c54114400 con 0x7f7c541026b0 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5410a2b0 0x7f7c5419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5410a2b0 0x7f7c5419aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41590/0 (socket says 172.21.15.98:41590) 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 -- 172.21.15.98:0/1911431921 learned_addr learned my addr 172.21.15.98:0/1911431921 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c52ffd700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c541026b0 0x7f7c541a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c527fc700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103000 0x7f7c541a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 -- 172.21.15.98:0/1911431921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c541026b0 msgr2=0x7f7c541a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c541026b0 0x7f7c541a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 -- 172.21.15.98:0/1911431921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103000 msgr2=0x7f7c541a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103000 0x7f7c541a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 -- 172.21.15.98:0/1911431921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5419b080 con 0x7f7c5410a2b0 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c537fe700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5410a2b0 0x7f7c5419aa40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0062a0 tx=0x7f7c4c004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:38.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c3bfff700 1 -- 172.21.15.98:0/1911431921 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c021070 con 0x7f7c5410a2b0 2024-04-25T16:31:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5419b310 con 0x7f7c5410a2b0 2024-04-25T16:31:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c3bfff700 1 -- 172.21.15.98:0/1911431921 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7c4c007d00 con 0x7f7c5410a2b0 2024-04-25T16:31:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c3bfff700 1 -- 172.21.15.98:0/1911431921 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c005280 con 0x7f7c5410a2b0 2024-04-25T16:31:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.887+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c5419b840 con 0x7f7c5410a2b0 2024-04-25T16:31:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.888+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c5410c720 con 0x7f7c5410a2b0 2024-04-25T16:31:38.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.890+0000 7f7c3bfff700 1 -- 172.21.15.98:0/1911431921 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7c4c024070 con 0x7f7c5410a2b0 2024-04-25T16:31:38.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.890+0000 7f7c3bfff700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c3c0afb80 0x7f7c3c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:38.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.891+0000 7f7c52ffd700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c3c0afb80 0x7f7c3c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:38.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.891+0000 7f7c3bfff700 1 -- 172.21.15.98:0/1911431921 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f7c4c00f020 con 0x7f7c5410a2b0 2024-04-25T16:31:38.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.891+0000 7f7c52ffd700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c3c0afb80 0x7f7c3c0b2090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7c44006fd0 tx=0x7f7c44008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:38.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:38.894+0000 7f7c3bfff700 1 -- 172.21.15.98:0/1911431921 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c4c02b020 con 0x7f7c5410a2b0 2024-04-25T16:31:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.103+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f7c5419bf70 con 0x7f7c5410a2b0 2024-04-25T16:31:39.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.103+0000 7f7c3bfff700 1 -- 172.21.15.98:0/1911431921 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f7c4c0a94b0 con 0x7f7c5410a2b0 2024-04-25T16:31:39.103 INFO:teuthology.orchestra.run.smithi098.stdout:257698037800 2024-04-25T16:31:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.105+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c3c0afb80 msgr2=0x7f7c3c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.105+0000 7f7c59731700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c3c0afb80 0x7f7c3c0b2090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7c44006fd0 tx=0x7f7c44008040 comp rx=0 tx=0).stop 2024-04-25T16:31:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.105+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5410a2b0 msgr2=0x7f7c5419aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.105+0000 7f7c59731700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5410a2b0 0x7f7c5419aa40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c0062a0 tx=0x7f7c4c004ed0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.106+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 shutdown_connections 2024-04-25T16:31:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.106+0000 7f7c59731700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7c3c0afb80 0x7f7c3c0b2090 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.106+0000 7f7c59731700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c541026b0 0x7f7c541a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.106+0000 7f7c59731700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7c5410a2b0 0x7f7c5419aa40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.106+0000 7f7c59731700 1 --2- 172.21.15.98:0/1911431921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c54103000 0x7f7c541a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.106+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 >> 172.21.15.98:0/1911431921 conn(0x7f7c540fe5c0 msgr2=0x7f7c54100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.106+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 shutdown_connections 2024-04-25T16:31:39.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.106+0000 7f7c59731700 1 -- 172.21.15.98:0/1911431921 wait complete. 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 -- 172.21.15.98:0/3862038979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348103b70 msgr2=0x7f7348103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3862038979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348103b70 0x7f7348103f80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f733c0099c0 tx=0x7f733c009cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 -- 172.21.15.98:0/3862038979 shutdown_connections 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3862038979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7348108630 0x7f7348110020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3862038979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348103b70 0x7f7348103f80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3862038979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348107ce0 0x7f73481080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 -- 172.21.15.98:0/3862038979 >> 172.21.15.98:0/3862038979 conn(0x7f73480fe5c0 msgr2=0x7f73481009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 -- 172.21.15.98:0/3862038979 shutdown_connections 2024-04-25T16:31:39.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.107+0000 7f734eb1e700 1 -- 172.21.15.98:0/3862038979 wait complete. 2024-04-25T16:31:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734eb1e700 1 Processor -- start 2024-04-25T16:31:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734eb1e700 1 -- start start 2024-04-25T16:31:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734eb1e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348103b70 0x7f734819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734eb1e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7348107ce0 0x7f734819c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734eb1e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348108630 0x7f734819c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734eb1e700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7348113120 con 0x7f7348108630 2024-04-25T16:31:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734eb1e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7348112fa0 con 0x7f7348107ce0 2024-04-25T16:31:39.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734eb1e700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f73481132a0 con 0x7f7348103b70 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734d0bb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348108630 0x7f734819c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.108+0000 7f734d0bb700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348108630 0x7f734819c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41602/0 (socket says 172.21.15.98:41602) 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734d0bb700 1 -- 172.21.15.98:0/3796396567 learned_addr learned my addr 172.21.15.98:0/3796396567 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734c8ba700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348103b70 0x7f734819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f7347fff700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7348107ce0 0x7f734819c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734d0bb700 1 -- 172.21.15.98:0/3796396567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348103b70 msgr2=0x7f734819bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734d0bb700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348103b70 0x7f734819bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734d0bb700 1 -- 172.21.15.98:0/3796396567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7348107ce0 msgr2=0x7f734819c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734d0bb700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7348107ce0 0x7f734819c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734d0bb700 1 -- 172.21.15.98:0/3796396567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73481999d0 con 0x7f7348108630 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734d0bb700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348108630 0x7f734819c600 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7340007ee0 tx=0x7f734000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f7345ffb700 1 -- 172.21.15.98:0/3796396567 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f734000dd20 con 0x7f7348108630 2024-04-25T16:31:39.110 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7348199cc0 con 0x7f7348108630 2024-04-25T16:31:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f7345ffb700 1 -- 172.21.15.98:0/3796396567 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7340004020 con 0x7f7348108630 2024-04-25T16:31:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.109+0000 7f7345ffb700 1 -- 172.21.15.98:0/3796396567 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7340015660 con 0x7f7348108630 2024-04-25T16:31:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.110+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f734819a200 con 0x7f7348108630 2024-04-25T16:31:39.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.111+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7348051190 con 0x7f7348108630 2024-04-25T16:31:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.113+0000 7f7345ffb700 1 -- 172.21.15.98:0/3796396567 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7340003a00 con 0x7f7348108630 2024-04-25T16:31:39.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.113+0000 7f7345ffb700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afc00 0x7f73380b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.113+0000 7f734c8ba700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afc00 0x7f73380b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.113+0000 7f7345ffb700 1 -- 172.21.15.98:0/3796396567 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f73400de830 con 0x7f7348108630 2024-04-25T16:31:39.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.114+0000 7f734c8ba700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afc00 0x7f73380b2110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f733c000c00 tx=0x7f733c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:39.118 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.118+0000 7f7345ffb700 1 -- 172.21.15.98:0/3796396567 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7340010070 con 0x7f7348108630 2024-04-25T16:31:39.199 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:39 smithi098 ceph-mon[29162]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:31:39.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.326+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f734819b5e0 con 0x7f7348108630 2024-04-25T16:31:39.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.326+0000 7f7345ffb700 1 -- 172.21.15.98:0/3796396567 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f73400a70b0 con 0x7f7348108630 2024-04-25T16:31:39.326 INFO:teuthology.orchestra.run.smithi098.stdout:158913790013 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afc00 msgr2=0x7f73380b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afc00 0x7f73380b2110 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f733c000c00 tx=0x7f733c005fb0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348108630 msgr2=0x7f734819c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348108630 0x7f734819c600 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7340007ee0 tx=0x7f734000d3b0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 shutdown_connections 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f73380afc00 0x7f73380b2110 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7348103b70 0x7f734819bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7348108630 0x7f734819c600 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 --2- 172.21.15.98:0/3796396567 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7348107ce0 0x7f734819c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:39.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 >> 172.21.15.98:0/3796396567 conn(0x7f73480fe5c0 msgr2=0x7f7348109590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 shutdown_connections 2024-04-25T16:31:39.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:39.328+0000 7f734eb1e700 1 -- 172.21.15.98:0/3796396567 wait complete. 2024-04-25T16:31:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:39 smithi203 ceph-mon[33115]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:31:39.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:39 smithi173 ceph-mon[32556]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:31:40.226 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:40 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1911431921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:31:40.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:40 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3796396567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:31:40.227 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:40 smithi098 ceph-mon[29162]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:31:40.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:40 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1911431921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:31:40.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:40 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3796396567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:31:40.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:40 smithi203 ceph-mon[33115]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:31:40.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:40 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1911431921' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:31:40.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:40 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3796396567' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:31:40.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:40 smithi173 ceph-mon[32556]: pgmap v329: 129 pgs: 129 active+clean; 646 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:31:40.650 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790008 got 158913790013 for osd.5 2024-04-25T16:31:40.651 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:41.252 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149761 got 111669149766 for osd.3 2024-04-25T16:31:41.253 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:41.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:41 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1816545539' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:31:41.433 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:41 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3250991959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:31:41.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:41 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1816545539' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:31:41.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:41 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3250991959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.238+0000 7f0d872f2700 1 -- 172.21.15.98:0/2973030913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 msgr2=0x7f0d8010ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.238+0000 7f0d872f2700 1 --2- 172.21.15.98:0/2973030913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 0x7f0d8010ff30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800b0d0 tx=0x7f0d7800b3e0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.239+0000 7f0d872f2700 1 -- 172.21.15.98:0/2973030913 shutdown_connections 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.239+0000 7f0d872f2700 1 --2- 172.21.15.98:0/2973030913 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 0x7f0d8010ff30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.239+0000 7f0d872f2700 1 --2- 172.21.15.98:0/2973030913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d80103350 0x7f0d80103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.239+0000 7f0d872f2700 1 --2- 172.21.15.98:0/2973030913 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d80107c80 0x7f0d80108090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.239+0000 7f0d872f2700 1 -- 172.21.15.98:0/2973030913 >> 172.21.15.98:0/2973030913 conn(0x7f0d800fec30 msgr2=0x7f0d80101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.239+0000 7f0d872f2700 1 -- 172.21.15.98:0/2973030913 shutdown_connections 2024-04-25T16:31:41.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.239+0000 7f0d872f2700 1 -- 172.21.15.98:0/2973030913 wait complete. 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d872f2700 1 Processor -- start 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d872f2700 1 -- start start 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d872f2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d80103350 0x7f0d8019f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d872f2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d80107c80 0x7f0d8019fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d872f2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 0x7f0d80199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d872f2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0d80113120 con 0x7f0d801085d0 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d872f2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0d80112fa0 con 0x7f0d80103350 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d872f2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0d801132a0 con 0x7f0d80107c80 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d8588f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 0x7f0d80199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d8588f700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 0x7f0d80199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41624/0 (socket says 172.21.15.98:41624) 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d8588f700 1 -- 172.21.15.98:0/1816545539 learned_addr learned my addr 172.21.15.98:0/1816545539 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d8508e700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d80103350 0x7f0d8019f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.240+0000 7f0d8488d700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d80107c80 0x7f0d8019fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d8588f700 1 -- 172.21.15.98:0/1816545539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d80107c80 msgr2=0x7f0d8019fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:41.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d8588f700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d80107c80 0x7f0d8019fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d8588f700 1 -- 172.21.15.98:0/1816545539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d80103350 msgr2=0x7f0d8019f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:41.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d8588f700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d80103350 0x7f0d8019f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d8588f700 1 -- 172.21.15.98:0/1816545539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d80199fb0 con 0x7f0d801085d0 2024-04-25T16:31:41.462 INFO:teuthology.orchestra.run.smithi098.stdout:55834574930 2024-04-25T16:31:41.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d8588f700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 0x7f0d80199900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0d78009850 tx=0x7f0d78007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:41.462 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d767fc700 1 -- 172.21.15.98:0/1816545539 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78004e60 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d767fc700 1 -- 172.21.15.98:0/1816545539 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0d7801a950 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d767fc700 1 -- 172.21.15.98:0/1816545539 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d78005710 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d8019a240 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.241+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d8006bf10 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.242+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d80104800 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.244+0000 7f0d767fc700 1 -- 172.21.15.98:0/1816545539 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0d780058b0 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.245+0000 7f0d767fc700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0d6c0afc00 0x7f0d6c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.245+0000 7f0d8508e700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0d6c0afc00 0x7f0d6c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.245+0000 7f0d767fc700 1 -- 172.21.15.98:0/1816545539 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f0d7800f030 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.245+0000 7f0d8508e700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0d6c0afc00 0x7f0d6c0b2110 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0d80106ca0 tx=0x7f0d7c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.248+0000 7f0d767fc700 1 -- 172.21.15.98:0/1816545539 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d780aac70 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.464+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0d8019ade0 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.464+0000 7f0d767fc700 1 -- 172.21.15.98:0/1816545539 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0d780aa570 con 0x7f0d801085d0 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0d6c0afc00 msgr2=0x7f0d6c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:41.463 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0d6c0afc00 0x7f0d6c0b2110 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0d80106ca0 tx=0x7f0d7c0073d0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 msgr2=0x7f0d80199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 0x7f0d80199900 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0d78009850 tx=0x7f0d78007d90 comp rx=0 tx=0).stop 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 shutdown_connections 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0d6c0afc00 0x7f0d6c0b2110 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0d801085d0 0x7f0d80199900 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d80103350 0x7f0d8019f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 --2- 172.21.15.98:0/1816545539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d80107c80 0x7f0d8019fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 >> 172.21.15.98:0/1816545539 conn(0x7f0d800fec30 msgr2=0x7f0d80101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 shutdown_connections 2024-04-25T16:31:41.464 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.466+0000 7f0d872f2700 1 -- 172.21.15.98:0/1816545539 wait complete. 2024-04-25T16:31:41.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:41 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1816545539' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:31:41.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:41 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3250991959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:31:42.083 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574930 for osd.1 2024-04-25T16:31:42.083 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.342+0000 7f95b3851700 1 -- 172.21.15.98:0/2867639044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac109710 msgr2=0x7f95ac111070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.342+0000 7f95b3851700 1 --2- 172.21.15.98:0/2867639044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac109710 0x7f95ac111070 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f95a400b0d0 tx=0x7f95a400b3e0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.342+0000 7f95b3851700 1 -- 172.21.15.98:0/2867639044 shutdown_connections 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.342+0000 7f95b3851700 1 --2- 172.21.15.98:0/2867639044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95ac104490 0x7f95ac1048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.342+0000 7f95b3851700 1 --2- 172.21.15.98:0/2867639044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac109710 0x7f95ac111070 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.342+0000 7f95b3851700 1 --2- 172.21.15.98:0/2867639044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95ac108dc0 0x7f95ac1091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.342+0000 7f95b3851700 1 -- 172.21.15.98:0/2867639044 >> 172.21.15.98:0/2867639044 conn(0x7f95ac0ffda0 msgr2=0x7f95ac1021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.343+0000 7f95b3851700 1 -- 172.21.15.98:0/2867639044 shutdown_connections 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.343+0000 7f95b3851700 1 -- 172.21.15.98:0/2867639044 wait complete. 2024-04-25T16:31:42.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.343+0000 7f95b3851700 1 Processor -- start 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.343+0000 7f95b3851700 1 -- start start 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b3851700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95ac104490 0x7f95ac076aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b3851700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac108dc0 0x7f95ac076fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b3851700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95ac109710 0x7f95ac077520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b3851700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac114210 con 0x7f95ac108dc0 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b3851700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac114090 con 0x7f95ac104490 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b3851700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f95ac114390 con 0x7f95ac109710 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac108dc0 0x7f95ac076fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac108dc0 0x7f95ac076fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41644/0 (socket says 172.21.15.98:41644) 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 -- 172.21.15.98:0/3250991959 learned_addr learned my addr 172.21.15.98:0/3250991959 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b15ed700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95ac104490 0x7f95ac076aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b1dee700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95ac109710 0x7f95ac077520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 -- 172.21.15.98:0/3250991959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95ac109710 msgr2=0x7f95ac077520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:42.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95ac109710 0x7f95ac077520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 -- 172.21.15.98:0/3250991959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95ac104490 msgr2=0x7f95ac076aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:42.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95ac104490 0x7f95ac076aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 -- 172.21.15.98:0/3250991959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ac1a8f20 con 0x7f95ac108dc0 2024-04-25T16:31:42.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b0dec700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac108dc0 0x7f95ac076fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f95a800e970 tx=0x7f95a800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:42.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95a27fc700 1 -- 172.21.15.98:0/3250991959 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a800cbd0 con 0x7f95ac108dc0 2024-04-25T16:31:42.160 INFO:teuthology.orchestra.run.smithi098.stdout:206158430259 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95ac1a9210 con 0x7f95ac108dc0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95a27fc700 1 -- 172.21.15.98:0/3250991959 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f95a800cd70 con 0x7f95ac108dc0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.344+0000 7f95a27fc700 1 -- 172.21.15.98:0/3250991959 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8005570 con 0x7f95ac108dc0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.345+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95ac1a97a0 con 0x7f95ac108dc0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.345+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95ac051190 con 0x7f95ac108dc0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.347+0000 7f95a27fc700 1 -- 172.21.15.98:0/3250991959 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f95a8004750 con 0x7f95ac108dc0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.348+0000 7f95a27fc700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f95980afb80 0x7f95980b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.348+0000 7f95b15ed700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f95980afb80 0x7f95980b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.348+0000 7f95a27fc700 1 -- 172.21.15.98:0/3250991959 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f95a80df2e0 con 0x7f95ac108dc0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.348+0000 7f95b15ed700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f95980afb80 0x7f95980b2090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f959c004630 tx=0x7f959c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.356+0000 7f95a27fc700 1 -- 172.21.15.98:0/3250991959 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95a8053ff0 con 0x7f95ac108dc0 2024-04-25T16:31:42.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.563+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f95ac077fd0 con 0x7f95ac108dc0 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.564+0000 7f95a27fc700 1 -- 172.21.15.98:0/3250991959 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f95a80a7b60 con 0x7f95ac108dc0 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f95980afb80 msgr2=0x7f95980b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f95980afb80 0x7f95980b2090 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f959c004630 tx=0x7f959c00b3c0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac108dc0 msgr2=0x7f95ac076fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac108dc0 0x7f95ac076fe0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f95a800e970 tx=0x7f95a800ec80 comp rx=0 tx=0).stop 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 shutdown_connections 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f95980afb80 0x7f95980b2090 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95ac109710 0x7f95ac077520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f95ac104490 0x7f95ac076aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 --2- 172.21.15.98:0/3250991959 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f95ac108dc0 0x7f95ac076fe0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 >> 172.21.15.98:0/3250991959 conn(0x7f95ac0ffda0 msgr2=0x7f95ac108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 shutdown_connections 2024-04-25T16:31:42.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:40.566+0000 7f95b3851700 1 -- 172.21.15.98:0/3250991959 wait complete. 2024-04-25T16:31:42.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:42 smithi203 ceph-mon[33115]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:31:42.461 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:42 smithi098 ceph-mon[29162]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:31:42.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:42 smithi173 ceph-mon[32556]: pgmap v330: 129 pgs: 129 active+clean; 646 KiB data, 382 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:31:42.780 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:42.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430254 got 206158430259 for osd.7 2024-04-25T16:31:42.781 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:43.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037793 got 257698037800 for osd.9 2024-04-25T16:31:43.257 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 -- 172.21.15.98:0/2578710294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318103b00 msgr2=0x7fc318103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 --2- 172.21.15.98:0/2578710294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318103b00 0x7fc318103f10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc3140099c0 tx=0x7fc314009cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 -- 172.21.15.98:0/2578710294 shutdown_connections 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 --2- 172.21.15.98:0/2578710294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc318108650 0x7fc31810fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 --2- 172.21.15.98:0/2578710294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318103b00 0x7fc318103f10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 --2- 172.21.15.98:0/2578710294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc318107d00 0x7fc318108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 -- 172.21.15.98:0/2578710294 >> 172.21.15.98:0/2578710294 conn(0x7fc3180fe370 msgr2=0x7fc3181007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 -- 172.21.15.98:0/2578710294 shutdown_connections 2024-04-25T16:31:44.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.927+0000 7fc31fd56700 1 -- 172.21.15.98:0/2578710294 wait complete. 2024-04-25T16:31:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31fd56700 1 Processor -- start 2024-04-25T16:31:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31fd56700 1 -- start start 2024-04-25T16:31:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31fd56700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc318103b00 0x7fc31819bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31fd56700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc318107d00 0x7fc31819c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31fd56700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318108650 0x7fc31819c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31fd56700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc318113170 con 0x7fc318108650 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31fd56700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc318112ff0 con 0x7fc318107d00 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31fd56700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc3181132f0 con 0x7fc318103b00 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31e2f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318108650 0x7fc31819c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31e2f3700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318108650 0x7fc31819c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41660/0 (socket says 172.21.15.98:41660) 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31e2f3700 1 -- 172.21.15.98:0/921687379 learned_addr learned my addr 172.21.15.98:0/921687379 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31d2f1700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc318107d00 0x7fc31819c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31daf2700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc318103b00 0x7fc31819bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31e2f3700 1 -- 172.21.15.98:0/921687379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc318103b00 msgr2=0x7fc31819bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31e2f3700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc318103b00 0x7fc31819bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31e2f3700 1 -- 172.21.15.98:0/921687379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc318107d00 msgr2=0x7fc31819c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31e2f3700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc318107d00 0x7fc31819c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.928+0000 7fc31e2f3700 1 -- 172.21.15.98:0/921687379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3181a6df0 con 0x7fc318108650 2024-04-25T16:31:44.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.929+0000 7fc31e2f3700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318108650 0x7fc31819c5d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc310007d70 tx=0x7fc31000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.929+0000 7fc30effd700 1 -- 172.21.15.98:0/921687379 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc310016020 con 0x7fc318108650 2024-04-25T16:31:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.929+0000 7fc30effd700 1 -- 172.21.15.98:0/921687379 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc31000ad40 con 0x7fc318108650 2024-04-25T16:31:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.929+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3181a70e0 con 0x7fc318108650 2024-04-25T16:31:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.929+0000 7fc30effd700 1 -- 172.21.15.98:0/921687379 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3100115e0 con 0x7fc318108650 2024-04-25T16:31:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.929+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3181a7620 con 0x7fc318108650 2024-04-25T16:31:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.930+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc31810b670 con 0x7fc318108650 2024-04-25T16:31:44.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.932+0000 7fc30effd700 1 -- 172.21.15.98:0/921687379 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc310019020 con 0x7fc318108650 2024-04-25T16:31:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.933+0000 7fc30effd700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3040afc00 0x7fc3040b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.933+0000 7fc31daf2700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3040afc00 0x7fc3040b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:44.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.933+0000 7fc30effd700 1 -- 172.21.15.98:0/921687379 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fc3100df2b0 con 0x7fc318108650 2024-04-25T16:31:44.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.933+0000 7fc31daf2700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3040afc00 0x7fc3040b2110 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc3140058e0 tx=0x7fc314005940 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:44.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:44.936+0000 7fc30effd700 1 -- 172.21.15.98:0/921687379 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc310020020 con 0x7fc318108650 2024-04-25T16:31:45.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.158+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fc318102d50 con 0x7fc318108650 2024-04-25T16:31:45.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.159+0000 7fc30effd700 1 -- 172.21.15.98:0/921687379 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc3100a7b30 con 0x7fc318108650 2024-04-25T16:31:45.158 INFO:teuthology.orchestra.run.smithi098.stdout:231928234031 2024-04-25T16:31:45.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3040afc00 msgr2=0x7fc3040b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:45.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3040afc00 0x7fc3040b2110 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc3140058e0 tx=0x7fc314005940 comp rx=0 tx=0).stop 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318108650 msgr2=0x7fc31819c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318108650 0x7fc31819c5d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc310007d70 tx=0x7fc31000a340 comp rx=0 tx=0).stop 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 shutdown_connections 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc3040afc00 0x7fc3040b2110 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc318103b00 0x7fc31819bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc318108650 0x7fc31819c5d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 --2- 172.21.15.98:0/921687379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc318107d00 0x7fc31819c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 >> 172.21.15.98:0/921687379 conn(0x7fc3180fe370 msgr2=0x7fc3180feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 shutdown_connections 2024-04-25T16:31:45.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:45.161+0000 7fc31fd56700 1 -- 172.21.15.98:0/921687379 wait complete. 2024-04-25T16:31:45.174 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:45 smithi098 ceph-mon[29162]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T16:31:45.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:45 smithi173 ceph-mon[32556]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T16:31:45.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:45 smithi203 ceph-mon[33115]: pgmap v331: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T16:31:45.948 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234025 got 231928234031 for osd.8 2024-04-25T16:31:45.948 DEBUG:teuthology.parallel:result is None 2024-04-25T16:31:45.948 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T16:31:45.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T16:31:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:46.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:46.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:46.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:46 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/921687379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T16:31:46.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:46 smithi098 ceph-mon[29162]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-25T16:31:46.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:31:46.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:31:46.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:46 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:46.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:46.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:46.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:46 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/921687379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T16:31:46.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:46 smithi203 ceph-mon[33115]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-25T16:31:46.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:31:46.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:31:46.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:46 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:46.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:46.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:46.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:46 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/921687379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T16:31:46.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:46 smithi173 ceph-mon[32556]: pgmap v332: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-25T16:31:46.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:31:46.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:31:46.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:46 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:31:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 -- 172.21.15.98:0/1448759489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102640 msgr2=0x7f31a0102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/1448759489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102640 0x7f31a0102a50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3194009990 tx=0x7f3194009ca0 comp rx=0 tx=0).stop 2024-04-25T16:31:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 -- 172.21.15.98:0/1448759489 shutdown_connections 2024-04-25T16:31:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/1448759489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31a0102f90 0x7f31a0111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/1448759489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a01091b0 0x7f31a01095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:47.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/1448759489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a0102640 0x7f31a0102a50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 -- 172.21.15.98:0/1448759489 >> 172.21.15.98:0/1448759489 conn(0x7f31a00fe370 msgr2=0x7f31a01007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 -- 172.21.15.98:0/1448759489 shutdown_connections 2024-04-25T16:31:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.997+0000 7f31a4e0a700 1 -- 172.21.15.98:0/1448759489 wait complete. 2024-04-25T16:31:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f31a4e0a700 1 Processor -- start 2024-04-25T16:31:47.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f31a4e0a700 1 -- start start 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f31a4e0a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a0102640 0x7f31a019ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f31a4e0a700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31a0102f90 0x7f31a019b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f31a4e0a700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01091b0 0x7f31a019f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f31a4e0a700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0114350 con 0x7f31a01091b0 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f31a4e0a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01141d0 con 0x7f31a0102640 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f31a4e0a700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f31a01144d0 con 0x7f31a0102f90 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f319ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01091b0 0x7f31a019f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f319e59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a0102640 0x7f31a019ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f319ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01091b0 0x7f31a019f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38216/0 (socket says 172.21.15.98:38216) 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f319ed9d700 1 -- 172.21.15.98:0/289645864 learned_addr learned my addr 172.21.15.98:0/289645864 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.998+0000 7f319dd9b700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31a0102f90 0x7f31a019b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f319ed9d700 1 -- 172.21.15.98:0/289645864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31a0102f90 msgr2=0x7f31a019b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f319ed9d700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31a0102f90 0x7f31a019b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:48.000 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f319ed9d700 1 -- 172.21.15.98:0/289645864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a0102640 msgr2=0x7f31a019ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f319ed9d700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a0102640 0x7f31a019ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f319ed9d700 1 -- 172.21.15.98:0/289645864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a019fc10 con 0x7f31a01091b0 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f319dd9b700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31a0102f90 0x7f31a019b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f319ed9d700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01091b0 0x7f31a019f570 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f319800b970 tx=0x7f319800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f318f7fe700 1 -- 172.21.15.98:0/289645864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198019070 con 0x7f31a01091b0 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a019ff00 con 0x7f31a01091b0 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f318f7fe700 1 -- 172.21.15.98:0/289645864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f319800cdc0 con 0x7f31a01091b0 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f318f7fe700 1 -- 172.21.15.98:0/289645864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198014e30 con 0x7f31a01091b0 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:47.999+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a01a7820 con 0x7f31a01091b0 2024-04-25T16:31:48.001 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.000+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a0051190 con 0x7f31a01091b0 2024-04-25T16:31:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.001+0000 7f318f7fe700 1 -- 172.21.15.98:0/289645864 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f319801c070 con 0x7f31a01091b0 2024-04-25T16:31:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.002+0000 7f318f7fe700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f31880afb80 0x7f31880b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.002+0000 7f319e59c700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f31880afb80 0x7f31880b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.002+0000 7f318f7fe700 1 -- 172.21.15.98:0/289645864 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f31980de740 con 0x7f31a01091b0 2024-04-25T16:31:48.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.003+0000 7f319e59c700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f31880afb80 0x7f31880b2090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f31a006c940 tx=0x7f319000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:48.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.006+0000 7f318f7fe700 1 -- 172.21.15.98:0/289645864 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f31980a6fc0 con 0x7f31a01091b0 2024-04-25T16:31:48.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.204+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f31a0063ad0 con 0x7f31880afb80 2024-04-25T16:31:48.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.213+0000 7f318f7fe700 1 -- 172.21.15.98:0/289645864 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340497 (secure 0 0 0) 0x7f31a0063ad0 con 0x7f31880afb80 2024-04-25T16:31:48.217 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f31880afb80 msgr2=0x7f31880b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f31880afb80 0x7f31880b2090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f31a006c940 tx=0x7f319000b410 comp rx=0 tx=0).stop 2024-04-25T16:31:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01091b0 msgr2=0x7f31a019f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01091b0 0x7f31a019f570 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f319800b970 tx=0x7f319800bc80 comp rx=0 tx=0).stop 2024-04-25T16:31:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 shutdown_connections 2024-04-25T16:31:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f31880afb80 0x7f31880b2090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f31a01091b0 0x7f31a019f570 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:48.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f31a0102640 0x7f31a019ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 --2- 172.21.15.98:0/289645864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f31a0102f90 0x7f31a019b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 >> 172.21.15.98:0/289645864 conn(0x7f31a00fe370 msgr2=0x7f31a00fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 shutdown_connections 2024-04-25T16:31:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:48.216+0000 7f31a4e0a700 1 -- 172.21.15.98:0/289645864 wait complete. 2024-04-25T16:31:48.221 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T16:31:48.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:48 smithi203 ceph-mon[33115]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T16:31:48.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:48 smithi098 ceph-mon[29162]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T16:31:48.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:48 smithi173 ceph-mon[32556]: pgmap v333: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-25T16:31:49.048 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-04-25T16:31:47.950451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661693,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":785,"num_read_kb":499,"num_write":376,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951957,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":813,"ondisk_log_size":813,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":395568,"kb_used_data":13136,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124546256,"statfs":{"total":1151940427776,"available":1151535366144,"internally_reserved":0,"allocated":13451264,"data_stored":7430249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":128,"num_read_kb":68,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003837"},"pg_stats":[{"pgid":"3.3f","version":"83'8","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.949318+0000","last_change":"2024-04-25T16:30:09.990700+0000","last_active":"2024-04-25T16:31:37.949318+0000","last_peered":"2024-04-25T16:31:37.949318+0000","last_clean":"2024-04-25T16:31:37.949318+0000","last_became_active":"2024-04-25T16:30:09.990225+0000","last_became_peered":"2024-04-25T16:30:09.990225+0000","last_unstale":"2024-04-25T16:31:37.949318+0000","last_undegraded":"2024-04-25T16:31:37.949318+0000","last_fullsized":"2024-04-25T16:31:37.949318+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:33.897277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.947429+0000","last_change":"2024-04-25T16:30:09.989618+0000","last_active":"2024-04-25T16:31:37.947429+0000","last_peered":"2024-04-25T16:31:37.947429+0000","last_clean":"2024-04-25T16:31:37.947429+0000","last_became_active":"2024-04-25T16:30:09.989270+0000","last_became_peered":"2024-04-25T16:30:09.989270+0000","last_unstale":"2024-04-25T16:31:37.947429+0000","last_undegraded":"2024-04-25T16:31:37.947429+0000","last_fullsized":"2024-04-25T16:31:37.947429+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:03.952233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.949748+0000","last_change":"2024-04-25T16:30:09.992442+0000","last_active":"2024-04-25T16:31:37.949748+0000","last_peered":"2024-04-25T16:31:37.949748+0000","last_clean":"2024-04-25T16:31:37.949748+0000","last_became_active":"2024-04-25T16:30:09.992115+0000","last_became_peered":"2024-04-25T16:30:09.992115+0000","last_unstale":"2024-04-25T16:31:37.949748+0000","last_undegraded":"2024-04-25T16:31:37.949748+0000","last_fullsized":"2024-04-25T16:31:37.949748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:00.668427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.943947+0000","last_change":"2024-04-25T16:31:09.006824+0000","last_active":"2024-04-25T16:31:37.943947+0000","last_peered":"2024-04-25T16:31:37.943947+0000","last_clean":"2024-04-25T16:31:37.943947+0000","last_became_active":"2024-04-25T16:31:09.005560+0000","last_became_peered":"2024-04-25T16:31:09.005560+0000","last_unstale":"2024-04-25T16:31:37.943947+0000","last_undegraded":"2024-04-25T16:31:37.943947+0000","last_fullsized":"2024-04-25T16:31:37.943947+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:41.979642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.928818+0000","last_change":"2024-04-25T16:30:09.991372+0000","last_active":"2024-04-25T16:31:37.928818+0000","last_peered":"2024-04-25T16:31:37.928818+0000","last_clean":"2024-04-25T16:31:37.928818+0000","last_became_active":"2024-04-25T16:30:09.991053+0000","last_became_peered":"2024-04-25T16:30:09.991053+0000","last_unstale":"2024-04-25T16:31:37.928818+0000","last_undegraded":"2024-04-25T16:31:37.928818+0000","last_fullsized":"2024-04-25T16:31:37.928818+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:28.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944476+0000","last_change":"2024-04-25T16:30:09.997563+0000","last_active":"2024-04-25T16:31:37.944476+0000","last_peered":"2024-04-25T16:31:37.944476+0000","last_clean":"2024-04-25T16:31:37.944476+0000","last_became_active":"2024-04-25T16:30:09.997158+0000","last_became_peered":"2024-04-25T16:30:09.997158+0000","last_unstale":"2024-04-25T16:31:37.944476+0000","last_undegraded":"2024-04-25T16:31:37.944476+0000","last_fullsized":"2024-04-25T16:31:37.944476+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:21.441957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.954206+0000","last_change":"2024-04-25T16:31:09.009888+0000","last_active":"2024-04-25T16:31:37.954206+0000","last_peered":"2024-04-25T16:31:37.954206+0000","last_clean":"2024-04-25T16:31:37.954206+0000","last_became_active":"2024-04-25T16:31:09.009201+0000","last_became_peered":"2024-04-25T16:31:09.009201+0000","last_unstale":"2024-04-25T16:31:37.954206+0000","last_undegraded":"2024-04-25T16:31:37.954206+0000","last_fullsized":"2024-04-25T16:31:37.954206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:52.171410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.948496+0000","last_change":"2024-04-25T16:30:09.994820+0000","last_active":"2024-04-25T16:31:37.948496+0000","last_peered":"2024-04-25T16:31:37.948496+0000","last_clean":"2024-04-25T16:31:37.948496+0000","last_became_active":"2024-04-25T16:30:09.994494+0000","last_became_peered":"2024-04-25T16:30:09.994494+0000","last_unstale":"2024-04-25T16:31:37.948496+0000","last_undegraded":"2024-04-25T16:31:37.948496+0000","last_fullsized":"2024-04-25T16:31:37.948496+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:39.373266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963287+0000","last_change":"2024-04-25T16:30:09.994870+0000","last_active":"2024-04-25T16:31:37.963287+0000","last_peered":"2024-04-25T16:31:37.963287+0000","last_clean":"2024-04-25T16:31:37.963287+0000","last_became_active":"2024-04-25T16:30:09.994521+0000","last_became_peered":"2024-04-25T16:30:09.994521+0000","last_unstale":"2024-04-25T16:31:37.963287+0000","last_undegraded":"2024-04-25T16:31:37.963287+0000","last_fullsized":"2024-04-25T16:31:37.963287+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:18.093904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":47,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.945320+0000","last_change":"2024-04-25T16:31:09.010148+0000","last_active":"2024-04-25T16:31:37.945320+0000","last_peered":"2024-04-25T16:31:37.945320+0000","last_clean":"2024-04-25T16:31:37.945320+0000","last_became_active":"2024-04-25T16:31:09.009655+0000","last_became_peered":"2024-04-25T16:31:09.009655+0000","last_unstale":"2024-04-25T16:31:37.945320+0000","last_undegraded":"2024-04-25T16:31:37.945320+0000","last_fullsized":"2024-04-25T16:31:37.945320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:18.637202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963613+0000","last_change":"2024-04-25T16:30:09.999519+0000","last_active":"2024-04-25T16:31:37.963613+0000","last_peered":"2024-04-25T16:31:37.963613+0000","last_clean":"2024-04-25T16:31:37.963613+0000","last_became_active":"2024-04-25T16:30:09.999158+0000","last_became_peered":"2024-04-25T16:30:09.999158+0000","last_unstale":"2024-04-25T16:31:37.963613+0000","last_undegraded":"2024-04-25T16:31:37.963613+0000","last_fullsized":"2024-04-25T16:31:37.963613+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:38:06.686541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944336+0000","last_change":"2024-04-25T16:30:10.005455+0000","last_active":"2024-04-25T16:31:37.944336+0000","last_peered":"2024-04-25T16:31:37.944336+0000","last_clean":"2024-04-25T16:31:37.944336+0000","last_became_active":"2024-04-25T16:30:10.004891+0000","last_became_peered":"2024-04-25T16:30:10.004891+0000","last_unstale":"2024-04-25T16:31:37.944336+0000","last_undegraded":"2024-04-25T16:31:37.944336+0000","last_fullsized":"2024-04-25T16:31:37.944336+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.598321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.943140+0000","last_change":"2024-04-25T16:30:09.992870+0000","last_active":"2024-04-25T16:31:37.943140+0000","last_peered":"2024-04-25T16:31:37.943140+0000","last_clean":"2024-04-25T16:31:37.943140+0000","last_became_active":"2024-04-25T16:30:09.992399+0000","last_became_peered":"2024-04-25T16:30:09.992399+0000","last_unstale":"2024-04-25T16:31:37.943140+0000","last_undegraded":"2024-04-25T16:31:37.943140+0000","last_fullsized":"2024-04-25T16:31:37.943140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.127287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.952996+0000","last_change":"2024-04-25T16:30:09.988707+0000","last_active":"2024-04-25T16:31:37.952996+0000","last_peered":"2024-04-25T16:31:37.952996+0000","last_clean":"2024-04-25T16:31:37.952996+0000","last_became_active":"2024-04-25T16:30:09.988386+0000","last_became_peered":"2024-04-25T16:30:09.988386+0000","last_unstale":"2024-04-25T16:31:37.952996+0000","last_undegraded":"2024-04-25T16:31:37.952996+0000","last_fullsized":"2024-04-25T16:31:37.952996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:03:15.973591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951178+0000","last_change":"2024-04-25T16:30:10.002780+0000","last_active":"2024-04-25T16:31:37.951178+0000","last_peered":"2024-04-25T16:31:37.951178+0000","last_clean":"2024-04-25T16:31:37.951178+0000","last_became_active":"2024-04-25T16:30:10.002055+0000","last_became_peered":"2024-04-25T16:30:10.002055+0000","last_unstale":"2024-04-25T16:31:37.951178+0000","last_undegraded":"2024-04-25T16:31:37.951178+0000","last_fullsized":"2024-04-25T16:31:37.951178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:23.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.962161+0000","last_change":"2024-04-25T16:30:09.998270+0000","last_active":"2024-04-25T16:31:37.962161+0000","last_peered":"2024-04-25T16:31:37.962161+0000","last_clean":"2024-04-25T16:31:37.962161+0000","last_became_active":"2024-04-25T16:30:09.997903+0000","last_became_peered":"2024-04-25T16:30:09.997903+0000","last_unstale":"2024-04-25T16:31:37.962161+0000","last_undegraded":"2024-04-25T16:31:37.962161+0000","last_fullsized":"2024-04-25T16:31:37.962161+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:14.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.949434+0000","last_change":"2024-04-25T16:30:09.988894+0000","last_active":"2024-04-25T16:31:37.949434+0000","last_peered":"2024-04-25T16:31:37.949434+0000","last_clean":"2024-04-25T16:31:37.949434+0000","last_became_active":"2024-04-25T16:30:09.988508+0000","last_became_peered":"2024-04-25T16:30:09.988508+0000","last_unstale":"2024-04-25T16:31:37.949434+0000","last_undegraded":"2024-04-25T16:31:37.949434+0000","last_fullsized":"2024-04-25T16:31:37.949434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:48.226193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.954793+0000","last_change":"2024-04-25T16:30:09.989910+0000","last_active":"2024-04-25T16:31:37.954793+0000","last_peered":"2024-04-25T16:31:37.954793+0000","last_clean":"2024-04-25T16:31:37.954793+0000","last_became_active":"2024-04-25T16:30:09.989560+0000","last_became_peered":"2024-04-25T16:30:09.989560+0000","last_unstale":"2024-04-25T16:31:37.954793+0000","last_undegraded":"2024-04-25T16:31:37.954793+0000","last_fullsized":"2024-04-25T16:31:37.954793+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:12.834034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.943030+0000","last_change":"2024-04-25T16:30:09.987930+0000","last_active":"2024-04-25T16:31:37.943030+0000","last_peered":"2024-04-25T16:31:37.943030+0000","last_clean":"2024-04-25T16:31:37.943030+0000","last_became_active":"2024-04-25T16:30:09.987595+0000","last_became_peered":"2024-04-25T16:30:09.987595+0000","last_unstale":"2024-04-25T16:31:37.943030+0000","last_undegraded":"2024-04-25T16:31:37.943030+0000","last_fullsized":"2024-04-25T16:31:37.943030+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:04.778172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001831+0000","last_change":"2024-04-25T16:30:14.006360+0000","last_active":"2024-04-25T16:31:09.001831+0000","last_peered":"2024-04-25T16:31:09.001831+0000","last_clean":"2024-04-25T16:31:09.001831+0000","last_became_active":"2024-04-25T16:30:14.005894+0000","last_became_peered":"2024-04-25T16:30:14.005894+0000","last_unstale":"2024-04-25T16:31:09.001831+0000","last_undegraded":"2024-04-25T16:31:09.001831+0000","last_fullsized":"2024-04-25T16:31:09.001831+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:33.150381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":108,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:43.164254+0000","last_change":"2024-04-25T16:30:08.992779+0000","last_active":"2024-04-25T16:31:43.164254+0000","last_peered":"2024-04-25T16:31:43.164254+0000","last_clean":"2024-04-25T16:31:43.164254+0000","last_became_active":"2024-04-25T16:30:08.992458+0000","last_became_peered":"2024-04-25T16:30:08.992458+0000","last_unstale":"2024-04-25T16:31:43.164254+0000","last_undegraded":"2024-04-25T16:31:43.164254+0000","last_fullsized":"2024-04-25T16:31:43.164254+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:21.430855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.946116+0000","last_change":"2024-04-25T16:30:09.995353+0000","last_active":"2024-04-25T16:31:37.946116+0000","last_peered":"2024-04-25T16:31:37.946116+0000","last_clean":"2024-04-25T16:31:37.946116+0000","last_became_active":"2024-04-25T16:30:09.995011+0000","last_became_peered":"2024-04-25T16:30:09.995011+0000","last_unstale":"2024-04-25T16:31:37.946116+0000","last_undegraded":"2024-04-25T16:31:37.946116+0000","last_fullsized":"2024-04-25T16:31:37.946116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:33.367268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.187252+0000","last_change":"2024-04-25T16:31:09.187252+0000","last_active":"2024-04-25T16:31:09.187252+0000","last_peered":"2024-04-25T16:31:09.187252+0000","last_clean":"2024-04-25T16:31:09.187252+0000","last_became_active":"2024-04-25T16:31:09.186689+0000","last_became_peered":"2024-04-25T16:31:09.186689+0000","last_unstale":"2024-04-25T16:31:09.187252+0000","last_undegraded":"2024-04-25T16:31:09.187252+0000","last_fullsized":"2024-04-25T16:31:09.187252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:18.287476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.185451+0000","last_change":"2024-04-25T16:30:08.997663+0000","last_active":"2024-04-25T16:31:09.185451+0000","last_peered":"2024-04-25T16:31:09.185451+0000","last_clean":"2024-04-25T16:31:09.185451+0000","last_became_active":"2024-04-25T16:30:08.997107+0000","last_became_peered":"2024-04-25T16:30:08.997107+0000","last_unstale":"2024-04-25T16:31:09.185451+0000","last_undegraded":"2024-04-25T16:31:09.185451+0000","last_fullsized":"2024-04-25T16:31:09.185451+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:25.856556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.962081+0000","last_change":"2024-04-25T16:30:09.987947+0000","last_active":"2024-04-25T16:31:37.962081+0000","last_peered":"2024-04-25T16:31:37.962081+0000","last_clean":"2024-04-25T16:31:37.962081+0000","last_became_active":"2024-04-25T16:30:09.987575+0000","last_became_peered":"2024-04-25T16:30:09.987575+0000","last_unstale":"2024-04-25T16:31:37.962081+0000","last_undegraded":"2024-04-25T16:31:37.962081+0000","last_fullsized":"2024-04-25T16:31:37.962081+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:28.443249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002770+0000","last_change":"2024-04-25T16:30:14.006711+0000","last_active":"2024-04-25T16:31:09.002770+0000","last_peered":"2024-04-25T16:31:09.002770+0000","last_clean":"2024-04-25T16:31:09.002770+0000","last_became_active":"2024-04-25T16:30:14.005500+0000","last_became_peered":"2024-04-25T16:30:14.005500+0000","last_unstale":"2024-04-25T16:31:09.002770+0000","last_undegraded":"2024-04-25T16:31:09.002770+0000","last_fullsized":"2024-04-25T16:31:09.002770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:58:26.288410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.007339+0000","last_change":"2024-04-25T16:30:08.993302+0000","last_active":"2024-04-25T16:31:09.007339+0000","last_peered":"2024-04-25T16:31:09.007339+0000","last_clean":"2024-04-25T16:31:09.007339+0000","last_became_active":"2024-04-25T16:30:08.992918+0000","last_became_peered":"2024-04-25T16:30:08.992918+0000","last_unstale":"2024-04-25T16:31:09.007339+0000","last_undegraded":"2024-04-25T16:31:09.007339+0000","last_fullsized":"2024-04-25T16:31:09.007339+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:18.843803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.961810+0000","last_change":"2024-04-25T16:30:09.988216+0000","last_active":"2024-04-25T16:31:37.961810+0000","last_peered":"2024-04-25T16:31:37.961810+0000","last_clean":"2024-04-25T16:31:37.961810+0000","last_became_active":"2024-04-25T16:30:09.987752+0000","last_became_peered":"2024-04-25T16:30:09.987752+0000","last_unstale":"2024-04-25T16:31:37.961810+0000","last_undegraded":"2024-04-25T16:31:37.961810+0000","last_fullsized":"2024-04-25T16:31:37.961810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:57:35.583000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002375+0000","last_change":"2024-04-25T16:30:14.001329+0000","last_active":"2024-04-25T16:31:09.002375+0000","last_peered":"2024-04-25T16:31:09.002375+0000","last_clean":"2024-04-25T16:31:09.002375+0000","last_became_active":"2024-04-25T16:30:14.000878+0000","last_became_peered":"2024-04-25T16:30:14.000878+0000","last_unstale":"2024-04-25T16:31:09.002375+0000","last_undegraded":"2024-04-25T16:31:09.002375+0000","last_fullsized":"2024-04-25T16:31:09.002375+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:07.946611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.009198+0000","last_change":"2024-04-25T16:30:08.989065+0000","last_active":"2024-04-25T16:31:09.009198+0000","last_peered":"2024-04-25T16:31:09.009198+0000","last_clean":"2024-04-25T16:31:09.009198+0000","last_became_active":"2024-04-25T16:30:08.988709+0000","last_became_peered":"2024-04-25T16:30:08.988709+0000","last_unstale":"2024-04-25T16:31:09.009198+0000","last_undegraded":"2024-04-25T16:31:09.009198+0000","last_fullsized":"2024-04-25T16:31:09.009198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:34.306581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.952822+0000","last_change":"2024-04-25T16:30:09.999641+0000","last_active":"2024-04-25T16:31:37.952822+0000","last_peered":"2024-04-25T16:31:37.952822+0000","last_clean":"2024-04-25T16:31:37.952822+0000","last_became_active":"2024-04-25T16:30:09.999319+0000","last_became_peered":"2024-04-25T16:30:09.999319+0000","last_unstale":"2024-04-25T16:31:37.952822+0000","last_undegraded":"2024-04-25T16:31:37.952822+0000","last_fullsized":"2024-04-25T16:31:37.952822+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:14.043201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002182+0000","last_change":"2024-04-25T16:30:13.999609+0000","last_active":"2024-04-25T16:31:09.002182+0000","last_peered":"2024-04-25T16:31:09.002182+0000","last_clean":"2024-04-25T16:31:09.002182+0000","last_became_active":"2024-04-25T16:30:13.999172+0000","last_became_peered":"2024-04-25T16:30:13.999172+0000","last_unstale":"2024-04-25T16:31:09.002182+0000","last_undegraded":"2024-04-25T16:31:09.002182+0000","last_fullsized":"2024-04-25T16:31:09.002182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:49.878153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'7","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:41.153775+0000","last_change":"2024-04-25T16:30:08.988919+0000","last_active":"2024-04-25T16:31:41.153775+0000","last_peered":"2024-04-25T16:31:41.153775+0000","last_clean":"2024-04-25T16:31:41.153775+0000","last_became_active":"2024-04-25T16:30:08.988218+0000","last_became_peered":"2024-04-25T16:30:08.988218+0000","last_unstale":"2024-04-25T16:31:41.153775+0000","last_undegraded":"2024-04-25T16:31:41.153775+0000","last_fullsized":"2024-04-25T16:31:41.153775+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:57:57.046055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.949422+0000","last_change":"2024-04-25T16:30:09.992498+0000","last_active":"2024-04-25T16:31:37.949422+0000","last_peered":"2024-04-25T16:31:37.949422+0000","last_clean":"2024-04-25T16:31:37.949422+0000","last_became_active":"2024-04-25T16:30:09.992190+0000","last_became_peered":"2024-04-25T16:30:09.992190+0000","last_unstale":"2024-04-25T16:31:37.949422+0000","last_undegraded":"2024-04-25T16:31:37.949422+0000","last_fullsized":"2024-04-25T16:31:37.949422+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:34.407963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002209+0000","last_change":"2024-04-25T16:30:14.007371+0000","last_active":"2024-04-25T16:31:09.002209+0000","last_peered":"2024-04-25T16:31:09.002209+0000","last_clean":"2024-04-25T16:31:09.002209+0000","last_became_active":"2024-04-25T16:30:14.006811+0000","last_became_peered":"2024-04-25T16:30:14.006811+0000","last_unstale":"2024-04-25T16:31:09.002209+0000","last_undegraded":"2024-04-25T16:31:09.002209+0000","last_fullsized":"2024-04-25T16:31:09.002209+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:01.473142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006206+0000","last_change":"2024-04-25T16:30:08.984080+0000","last_active":"2024-04-25T16:31:09.006206+0000","last_peered":"2024-04-25T16:31:09.006206+0000","last_clean":"2024-04-25T16:31:09.006206+0000","last_became_active":"2024-04-25T16:30:08.983702+0000","last_became_peered":"2024-04-25T16:30:08.983702+0000","last_unstale":"2024-04-25T16:31:09.006206+0000","last_undegraded":"2024-04-25T16:31:09.006206+0000","last_fullsized":"2024-04-25T16:31:09.006206+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:22.311986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.953706+0000","last_change":"2024-04-25T16:31:09.187653+0000","last_active":"2024-04-25T16:31:37.953706+0000","last_peered":"2024-04-25T16:31:37.953706+0000","last_clean":"2024-04-25T16:31:37.953706+0000","last_became_active":"2024-04-25T16:31:09.187183+0000","last_became_peered":"2024-04-25T16:31:09.187183+0000","last_unstale":"2024-04-25T16:31:37.953706+0000","last_undegraded":"2024-04-25T16:31:37.953706+0000","last_fullsized":"2024-04-25T16:31:37.953706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:44:35.157126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.010419+0000","last_change":"2024-04-25T16:30:14.003218+0000","last_active":"2024-04-25T16:31:09.010419+0000","last_peered":"2024-04-25T16:31:09.010419+0000","last_clean":"2024-04-25T16:31:09.010419+0000","last_became_active":"2024-04-25T16:30:14.002782+0000","last_became_peered":"2024-04-25T16:30:14.002782+0000","last_unstale":"2024-04-25T16:31:09.010419+0000","last_undegraded":"2024-04-25T16:31:09.010419+0000","last_fullsized":"2024-04-25T16:31:09.010419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005109+0000","last_change":"2024-04-25T16:30:08.993447+0000","last_active":"2024-04-25T16:31:09.005109+0000","last_peered":"2024-04-25T16:31:09.005109+0000","last_clean":"2024-04-25T16:31:09.005109+0000","last_became_active":"2024-04-25T16:30:08.993108+0000","last_became_peered":"2024-04-25T16:30:08.993108+0000","last_unstale":"2024-04-25T16:31:09.005109+0000","last_undegraded":"2024-04-25T16:31:09.005109+0000","last_fullsized":"2024-04-25T16:31:09.005109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:00.511392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.935592+0000","last_change":"2024-04-25T16:30:09.989863+0000","last_active":"2024-04-25T16:31:37.935592+0000","last_peered":"2024-04-25T16:31:37.935592+0000","last_clean":"2024-04-25T16:31:37.935592+0000","last_became_active":"2024-04-25T16:30:09.989520+0000","last_became_peered":"2024-04-25T16:30:09.989520+0000","last_unstale":"2024-04-25T16:31:37.935592+0000","last_undegraded":"2024-04-25T16:31:37.935592+0000","last_fullsized":"2024-04-25T16:31:37.935592+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:54.178298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.472395+0000","last_change":"2024-04-25T16:30:13.999904+0000","last_active":"2024-04-25T16:31:09.472395+0000","last_peered":"2024-04-25T16:31:09.472395+0000","last_clean":"2024-04-25T16:31:09.472395+0000","last_became_active":"2024-04-25T16:30:13.999468+0000","last_became_peered":"2024-04-25T16:30:13.999468+0000","last_unstale":"2024-04-25T16:31:09.472395+0000","last_undegraded":"2024-04-25T16:31:09.472395+0000","last_fullsized":"2024-04-25T16:31:09.472395+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:09.653365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005737+0000","last_change":"2024-04-25T16:30:08.987550+0000","last_active":"2024-04-25T16:31:09.005737+0000","last_peered":"2024-04-25T16:31:09.005737+0000","last_clean":"2024-04-25T16:31:09.005737+0000","last_became_active":"2024-04-25T16:30:08.987231+0000","last_became_peered":"2024-04-25T16:30:08.987231+0000","last_unstale":"2024-04-25T16:31:09.005737+0000","last_undegraded":"2024-04-25T16:31:09.005737+0000","last_fullsized":"2024-04-25T16:31:09.005737+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:01.110960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.942132+0000","last_change":"2024-04-25T16:30:09.991985+0000","last_active":"2024-04-25T16:31:37.942132+0000","last_peered":"2024-04-25T16:31:37.942132+0000","last_clean":"2024-04-25T16:31:37.942132+0000","last_became_active":"2024-04-25T16:30:09.991645+0000","last_became_peered":"2024-04-25T16:30:09.991645+0000","last_unstale":"2024-04-25T16:31:37.942132+0000","last_undegraded":"2024-04-25T16:31:37.942132+0000","last_fullsized":"2024-04-25T16:31:37.942132+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:30.419416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.185258+0000","last_change":"2024-04-25T16:30:13.999931+0000","last_active":"2024-04-25T16:31:09.185258+0000","last_peered":"2024-04-25T16:31:09.185258+0000","last_clean":"2024-04-25T16:31:09.185258+0000","last_became_active":"2024-04-25T16:30:13.997093+0000","last_became_peered":"2024-04-25T16:30:13.997093+0000","last_unstale":"2024-04-25T16:31:09.185258+0000","last_undegraded":"2024-04-25T16:31:09.185258+0000","last_fullsized":"2024-04-25T16:31:09.185258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:31.973370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'5","reported_seq":168,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:45.159660+0000","last_change":"2024-04-25T16:30:08.985721+0000","last_active":"2024-04-25T16:31:45.159660+0000","last_peered":"2024-04-25T16:31:45.159660+0000","last_clean":"2024-04-25T16:31:45.159660+0000","last_became_active":"2024-04-25T16:30:08.985097+0000","last_became_peered":"2024-04-25T16:30:08.985097+0000","last_unstale":"2024-04-25T16:31:45.159660+0000","last_undegraded":"2024-04-25T16:31:45.159660+0000","last_fullsized":"2024-04-25T16:31:45.159660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:00.713027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":136,"num_read_kb":136,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.961844+0000","last_change":"2024-04-25T16:30:09.992659+0000","last_active":"2024-04-25T16:31:37.961844+0000","last_peered":"2024-04-25T16:31:37.961844+0000","last_clean":"2024-04-25T16:31:37.961844+0000","last_became_active":"2024-04-25T16:30:09.985936+0000","last_became_peered":"2024-04-25T16:30:09.985936+0000","last_unstale":"2024-04-25T16:31:37.961844+0000","last_undegraded":"2024-04-25T16:31:37.961844+0000","last_fullsized":"2024-04-25T16:31:37.961844+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:10.808082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001423+0000","last_change":"2024-04-25T16:30:14.009352+0000","last_active":"2024-04-25T16:31:09.001423+0000","last_peered":"2024-04-25T16:31:09.001423+0000","last_clean":"2024-04-25T16:31:09.001423+0000","last_became_active":"2024-04-25T16:30:14.005250+0000","last_became_peered":"2024-04-25T16:30:14.005250+0000","last_unstale":"2024-04-25T16:31:09.001423+0000","last_undegraded":"2024-04-25T16:31:09.001423+0000","last_fullsized":"2024-04-25T16:31:09.001423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:23.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005543+0000","last_change":"2024-04-25T16:30:08.990547+0000","last_active":"2024-04-25T16:31:09.005543+0000","last_peered":"2024-04-25T16:31:09.005543+0000","last_clean":"2024-04-25T16:31:09.005543+0000","last_became_active":"2024-04-25T16:30:08.989927+0000","last_became_peered":"2024-04-25T16:30:08.989927+0000","last_unstale":"2024-04-25T16:31:09.005543+0000","last_undegraded":"2024-04-25T16:31:09.005543+0000","last_fullsized":"2024-04-25T16:31:09.005543+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:05.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.954774+0000","last_change":"2024-04-25T16:30:10.002459+0000","last_active":"2024-04-25T16:31:37.954774+0000","last_peered":"2024-04-25T16:31:37.954774+0000","last_clean":"2024-04-25T16:31:37.954774+0000","last_became_active":"2024-04-25T16:30:10.002005+0000","last_became_peered":"2024-04-25T16:30:10.002005+0000","last_unstale":"2024-04-25T16:31:37.954774+0000","last_undegraded":"2024-04-25T16:31:37.954774+0000","last_fullsized":"2024-04-25T16:31:37.954774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:01.220371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.186538+0000","last_change":"2024-04-25T16:31:09.186538+0000","last_active":"2024-04-25T16:31:09.186538+0000","last_peered":"2024-04-25T16:31:09.186538+0000","last_clean":"2024-04-25T16:31:09.186538+0000","last_became_active":"2024-04-25T16:31:09.185930+0000","last_became_peered":"2024-04-25T16:31:09.185930+0000","last_unstale":"2024-04-25T16:31:09.186538+0000","last_undegraded":"2024-04-25T16:31:09.186538+0000","last_fullsized":"2024-04-25T16:31:09.186538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:14:38.930206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005922+0000","last_change":"2024-04-25T16:30:08.985264+0000","last_active":"2024-04-25T16:31:09.005922+0000","last_peered":"2024-04-25T16:31:09.005922+0000","last_clean":"2024-04-25T16:31:09.005922+0000","last_became_active":"2024-04-25T16:30:08.984901+0000","last_became_peered":"2024-04-25T16:30:08.984901+0000","last_unstale":"2024-04-25T16:31:09.005922+0000","last_undegraded":"2024-04-25T16:31:09.005922+0000","last_fullsized":"2024-04-25T16:31:09.005922+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:15.271981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963583+0000","last_change":"2024-04-25T16:30:09.997002+0000","last_active":"2024-04-25T16:31:37.963583+0000","last_peered":"2024-04-25T16:31:37.963583+0000","last_clean":"2024-04-25T16:31:37.963583+0000","last_became_active":"2024-04-25T16:30:09.996636+0000","last_became_peered":"2024-04-25T16:30:09.996636+0000","last_unstale":"2024-04-25T16:31:37.963583+0000","last_undegraded":"2024-04-25T16:31:37.963583+0000","last_fullsized":"2024-04-25T16:31:37.963583+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:32.522050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001480+0000","last_change":"2024-04-25T16:30:14.006403+0000","last_active":"2024-04-25T16:31:09.001480+0000","last_peered":"2024-04-25T16:31:09.001480+0000","last_clean":"2024-04-25T16:31:09.001480+0000","last_became_active":"2024-04-25T16:30:14.004816+0000","last_became_peered":"2024-04-25T16:30:14.004816+0000","last_unstale":"2024-04-25T16:31:09.001480+0000","last_undegraded":"2024-04-25T16:31:09.001480+0000","last_fullsized":"2024-04-25T16:31:09.001480+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:51:10.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'38","reported_seq":208,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:45.173672+0000","last_change":"2024-04-25T16:30:08.986119+0000","last_active":"2024-04-25T16:31:45.173672+0000","last_peered":"2024-04-25T16:31:45.173672+0000","last_clean":"2024-04-25T16:31:45.173672+0000","last_became_active":"2024-04-25T16:30:08.985806+0000","last_became_peered":"2024-04-25T16:30:08.985806+0000","last_unstale":"2024-04-25T16:31:45.173672+0000","last_undegraded":"2024-04-25T16:31:45.173672+0000","last_fullsized":"2024-04-25T16:31:45.173672+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:17.125886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":134,"num_write":40,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951203+0000","last_change":"2024-04-25T16:30:09.996891+0000","last_active":"2024-04-25T16:31:37.951203+0000","last_peered":"2024-04-25T16:31:37.951203+0000","last_clean":"2024-04-25T16:31:37.951203+0000","last_became_active":"2024-04-25T16:30:09.995851+0000","last_became_peered":"2024-04-25T16:30:09.995851+0000","last_unstale":"2024-04-25T16:31:37.951203+0000","last_undegraded":"2024-04-25T16:31:37.951203+0000","last_fullsized":"2024-04-25T16:31:37.951203+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:42.602854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006493+0000","last_change":"2024-04-25T16:30:13.998277+0000","last_active":"2024-04-25T16:31:09.006493+0000","last_peered":"2024-04-25T16:31:09.006493+0000","last_clean":"2024-04-25T16:31:09.006493+0000","last_became_active":"2024-04-25T16:30:13.997313+0000","last_became_peered":"2024-04-25T16:30:13.997313+0000","last_unstale":"2024-04-25T16:31:09.006493+0000","last_undegraded":"2024-04-25T16:31:09.006493+0000","last_fullsized":"2024-04-25T16:31:09.006493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:56.379806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002447+0000","last_change":"2024-04-25T16:30:08.986377+0000","last_active":"2024-04-25T16:31:09.002447+0000","last_peered":"2024-04-25T16:31:09.002447+0000","last_clean":"2024-04-25T16:31:09.002447+0000","last_became_active":"2024-04-25T16:30:08.985892+0000","last_became_peered":"2024-04-25T16:30:08.985892+0000","last_unstale":"2024-04-25T16:31:09.002447+0000","last_undegraded":"2024-04-25T16:31:09.002447+0000","last_fullsized":"2024-04-25T16:31:09.002447+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:53.651789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951284+0000","last_change":"2024-04-25T16:31:09.010551+0000","last_active":"2024-04-25T16:31:37.951284+0000","last_peered":"2024-04-25T16:31:37.951284+0000","last_clean":"2024-04-25T16:31:37.951284+0000","last_became_active":"2024-04-25T16:31:09.010141+0000","last_became_peered":"2024-04-25T16:31:09.010141+0000","last_unstale":"2024-04-25T16:31:37.951284+0000","last_undegraded":"2024-04-25T16:31:37.951284+0000","last_fullsized":"2024-04-25T16:31:37.951284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:54.650080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.010236+0000","last_change":"2024-04-25T16:30:14.006064+0000","last_active":"2024-04-25T16:31:09.010236+0000","last_peered":"2024-04-25T16:31:09.010236+0000","last_clean":"2024-04-25T16:31:09.010236+0000","last_became_active":"2024-04-25T16:30:14.005545+0000","last_became_peered":"2024-04-25T16:30:14.005545+0000","last_unstale":"2024-04-25T16:31:09.010236+0000","last_undegraded":"2024-04-25T16:31:09.010236+0000","last_fullsized":"2024-04-25T16:31:09.010236+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:36.611008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006844+0000","last_change":"2024-04-25T16:30:08.988413+0000","last_active":"2024-04-25T16:31:09.006844+0000","last_peered":"2024-04-25T16:31:09.006844+0000","last_clean":"2024-04-25T16:31:09.006844+0000","last_became_active":"2024-04-25T16:30:08.988075+0000","last_became_peered":"2024-04-25T16:30:08.988075+0000","last_unstale":"2024-04-25T16:31:09.006844+0000","last_undegraded":"2024-04-25T16:31:09.006844+0000","last_fullsized":"2024-04-25T16:31:09.006844+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:16.635876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.961181+0000","last_change":"2024-04-25T16:30:09.997697+0000","last_active":"2024-04-25T16:31:37.961181+0000","last_peered":"2024-04-25T16:31:37.961181+0000","last_clean":"2024-04-25T16:31:37.961181+0000","last_became_active":"2024-04-25T16:30:09.997086+0000","last_became_peered":"2024-04-25T16:30:09.997086+0000","last_unstale":"2024-04-25T16:31:37.961181+0000","last_undegraded":"2024-04-25T16:31:37.961181+0000","last_fullsized":"2024-04-25T16:31:37.961181+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:35.345902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002285+0000","last_change":"2024-04-25T16:30:14.010486+0000","last_active":"2024-04-25T16:31:09.002285+0000","last_peered":"2024-04-25T16:31:09.002285+0000","last_clean":"2024-04-25T16:31:09.002285+0000","last_became_active":"2024-04-25T16:30:14.009273+0000","last_became_peered":"2024-04-25T16:30:14.009273+0000","last_unstale":"2024-04-25T16:31:09.002285+0000","last_undegraded":"2024-04-25T16:31:09.002285+0000","last_fullsized":"2024-04-25T16:31:09.002285+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:11.863575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"83'7","reported_seq":107,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:46.174301+0000","last_change":"2024-04-25T16:30:08.986661+0000","last_active":"2024-04-25T16:31:46.174301+0000","last_peered":"2024-04-25T16:31:46.174301+0000","last_clean":"2024-04-25T16:31:46.174301+0000","last_became_active":"2024-04-25T16:30:08.986306+0000","last_became_peered":"2024-04-25T16:30:08.986306+0000","last_unstale":"2024-04-25T16:31:46.174301+0000","last_undegraded":"2024-04-25T16:31:46.174301+0000","last_fullsized":"2024-04-25T16:31:46.174301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:44.363693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.954252+0000","last_change":"2024-04-25T16:30:09.993443+0000","last_active":"2024-04-25T16:31:37.954252+0000","last_peered":"2024-04-25T16:31:37.954252+0000","last_clean":"2024-04-25T16:31:37.954252+0000","last_became_active":"2024-04-25T16:30:09.993031+0000","last_became_peered":"2024-04-25T16:30:09.993031+0000","last_unstale":"2024-04-25T16:31:37.954252+0000","last_undegraded":"2024-04-25T16:31:37.954252+0000","last_fullsized":"2024-04-25T16:31:37.954252+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:22.908713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005959+0000","last_change":"2024-04-25T16:30:14.003183+0000","last_active":"2024-04-25T16:31:09.005959+0000","last_peered":"2024-04-25T16:31:09.005959+0000","last_clean":"2024-04-25T16:31:09.005959+0000","last_became_active":"2024-04-25T16:30:14.002182+0000","last_became_peered":"2024-04-25T16:30:14.002182+0000","last_unstale":"2024-04-25T16:31:09.005959+0000","last_undegraded":"2024-04-25T16:31:09.005959+0000","last_fullsized":"2024-04-25T16:31:09.005959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:08.616556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006777+0000","last_change":"2024-04-25T16:30:08.995860+0000","last_active":"2024-04-25T16:31:09.006777+0000","last_peered":"2024-04-25T16:31:09.006777+0000","last_clean":"2024-04-25T16:31:09.006777+0000","last_became_active":"2024-04-25T16:30:08.995487+0000","last_became_peered":"2024-04-25T16:30:08.995487+0000","last_unstale":"2024-04-25T16:31:09.006777+0000","last_undegraded":"2024-04-25T16:31:09.006777+0000","last_fullsized":"2024-04-25T16:31:09.006777+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:48.206502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.947991+0000","last_change":"2024-04-25T16:30:09.998820+0000","last_active":"2024-04-25T16:31:37.947991+0000","last_peered":"2024-04-25T16:31:37.947991+0000","last_clean":"2024-04-25T16:31:37.947991+0000","last_became_active":"2024-04-25T16:30:09.998448+0000","last_became_peered":"2024-04-25T16:30:09.998448+0000","last_unstale":"2024-04-25T16:31:37.947991+0000","last_undegraded":"2024-04-25T16:31:37.947991+0000","last_fullsized":"2024-04-25T16:31:37.947991+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:54.361250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005702+0000","last_change":"2024-04-25T16:30:13.992379+0000","last_active":"2024-04-25T16:31:09.005702+0000","last_peered":"2024-04-25T16:31:09.005702+0000","last_clean":"2024-04-25T16:31:09.005702+0000","last_became_active":"2024-04-25T16:30:13.991940+0000","last_became_peered":"2024-04-25T16:30:13.991940+0000","last_unstale":"2024-04-25T16:31:09.005702+0000","last_undegraded":"2024-04-25T16:31:09.005702+0000","last_fullsized":"2024-04-25T16:31:09.005702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:59.152384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.008399+0000","last_change":"2024-04-25T16:30:08.987701+0000","last_active":"2024-04-25T16:31:09.008399+0000","last_peered":"2024-04-25T16:31:09.008399+0000","last_clean":"2024-04-25T16:31:09.008399+0000","last_became_active":"2024-04-25T16:30:08.987376+0000","last_became_peered":"2024-04-25T16:30:08.987376+0000","last_unstale":"2024-04-25T16:31:09.008399+0000","last_undegraded":"2024-04-25T16:31:09.008399+0000","last_fullsized":"2024-04-25T16:31:09.008399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:21.810100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.945676+0000","last_change":"2024-04-25T16:30:09.993005+0000","last_active":"2024-04-25T16:31:37.945676+0000","last_peered":"2024-04-25T16:31:37.945676+0000","last_clean":"2024-04-25T16:31:37.945676+0000","last_became_active":"2024-04-25T16:30:09.992619+0000","last_became_peered":"2024-04-25T16:30:09.992619+0000","last_unstale":"2024-04-25T16:31:37.945676+0000","last_undegraded":"2024-04-25T16:31:37.945676+0000","last_fullsized":"2024-04-25T16:31:37.945676+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:39:40.998241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006207+0000","last_change":"2024-04-25T16:30:14.147895+0000","last_active":"2024-04-25T16:31:09.006207+0000","last_peered":"2024-04-25T16:31:09.006207+0000","last_clean":"2024-04-25T16:31:09.006207+0000","last_became_active":"2024-04-25T16:30:14.147298+0000","last_became_peered":"2024-04-25T16:30:14.147298+0000","last_unstale":"2024-04-25T16:31:09.006207+0000","last_undegraded":"2024-04-25T16:31:09.006207+0000","last_fullsized":"2024-04-25T16:31:09.006207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:06.269587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001713+0000","last_change":"2024-04-25T16:30:08.982683+0000","last_active":"2024-04-25T16:31:09.001713+0000","last_peered":"2024-04-25T16:31:09.001713+0000","last_clean":"2024-04-25T16:31:09.001713+0000","last_became_active":"2024-04-25T16:30:08.982236+0000","last_became_peered":"2024-04-25T16:30:08.982236+0000","last_unstale":"2024-04-25T16:31:09.001713+0000","last_undegraded":"2024-04-25T16:31:09.001713+0000","last_fullsized":"2024-04-25T16:31:09.001713+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:48.781791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.939293+0000","last_change":"2024-04-25T16:30:09.986734+0000","last_active":"2024-04-25T16:31:37.939293+0000","last_peered":"2024-04-25T16:31:37.939293+0000","last_clean":"2024-04-25T16:31:37.939293+0000","last_became_active":"2024-04-25T16:30:09.986337+0000","last_became_peered":"2024-04-25T16:30:09.986337+0000","last_unstale":"2024-04-25T16:31:37.939293+0000","last_undegraded":"2024-04-25T16:31:37.939293+0000","last_fullsized":"2024-04-25T16:31:37.939293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.678009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005340+0000","last_change":"2024-04-25T16:30:14.003324+0000","last_active":"2024-04-25T16:31:09.005340+0000","last_peered":"2024-04-25T16:31:09.005340+0000","last_clean":"2024-04-25T16:31:09.005340+0000","last_became_active":"2024-04-25T16:30:14.002904+0000","last_became_peered":"2024-04-25T16:30:14.002904+0000","last_unstale":"2024-04-25T16:31:09.005340+0000","last_undegraded":"2024-04-25T16:31:09.005340+0000","last_fullsized":"2024-04-25T16:31:09.005340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:28.750102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":98,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:41.162285+0000","last_change":"2024-04-25T16:30:08.993469+0000","last_active":"2024-04-25T16:31:41.162285+0000","last_peered":"2024-04-25T16:31:41.162285+0000","last_clean":"2024-04-25T16:31:41.162285+0000","last_became_active":"2024-04-25T16:30:08.993126+0000","last_became_peered":"2024-04-25T16:30:08.993126+0000","last_unstale":"2024-04-25T16:31:41.162285+0000","last_undegraded":"2024-04-25T16:31:41.162285+0000","last_fullsized":"2024-04-25T16:31:41.162285+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:10.274645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.945673+0000","last_change":"2024-04-25T16:30:10.000597+0000","last_active":"2024-04-25T16:31:37.945673+0000","last_peered":"2024-04-25T16:31:37.945673+0000","last_clean":"2024-04-25T16:31:37.945673+0000","last_became_active":"2024-04-25T16:30:09.998892+0000","last_became_peered":"2024-04-25T16:30:09.998892+0000","last_unstale":"2024-04-25T16:31:37.945673+0000","last_undegraded":"2024-04-25T16:31:37.945673+0000","last_fullsized":"2024-04-25T16:31:37.945673+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:09.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005789+0000","last_change":"2024-04-25T16:30:14.147240+0000","last_active":"2024-04-25T16:31:09.005789+0000","last_peered":"2024-04-25T16:31:09.005789+0000","last_clean":"2024-04-25T16:31:09.005789+0000","last_became_active":"2024-04-25T16:30:14.146793+0000","last_became_peered":"2024-04-25T16:30:14.146793+0000","last_unstale":"2024-04-25T16:31:09.005789+0000","last_undegraded":"2024-04-25T16:31:09.005789+0000","last_fullsized":"2024-04-25T16:31:09.005789+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:55.713386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944046+0000","last_change":"2024-04-25T16:30:09.994595+0000","last_active":"2024-04-25T16:31:37.944046+0000","last_peered":"2024-04-25T16:31:37.944046+0000","last_clean":"2024-04-25T16:31:37.944046+0000","last_became_active":"2024-04-25T16:30:09.994305+0000","last_became_peered":"2024-04-25T16:30:09.994305+0000","last_unstale":"2024-04-25T16:31:37.944046+0000","last_undegraded":"2024-04-25T16:31:37.944046+0000","last_fullsized":"2024-04-25T16:31:37.944046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:27.839500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'6","reported_seq":100,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:42.165886+0000","last_change":"2024-04-25T16:30:08.984203+0000","last_active":"2024-04-25T16:31:42.165886+0000","last_peered":"2024-04-25T16:31:42.165886+0000","last_clean":"2024-04-25T16:31:42.165886+0000","last_became_active":"2024-04-25T16:30:08.982883+0000","last_became_peered":"2024-04-25T16:30:08.982883+0000","last_unstale":"2024-04-25T16:31:42.165886+0000","last_undegraded":"2024-04-25T16:31:42.165886+0000","last_fullsized":"2024-04-25T16:31:42.165886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:18.447624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":67,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.017505+0000","last_change":"2024-04-25T16:31:09.017505+0000","last_active":"2024-04-25T16:31:09.017505+0000","last_peered":"2024-04-25T16:31:09.017505+0000","last_clean":"2024-04-25T16:31:09.017505+0000","last_became_active":"2024-04-25T16:31:09.011812+0000","last_became_peered":"2024-04-25T16:31:09.011812+0000","last_unstale":"2024-04-25T16:31:09.017505+0000","last_undegraded":"2024-04-25T16:31:09.017505+0000","last_fullsized":"2024-04-25T16:31:09.017505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:02.404908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.948845+0000","last_change":"2024-04-25T16:30:09.990998+0000","last_active":"2024-04-25T16:31:37.948845+0000","last_peered":"2024-04-25T16:31:37.948845+0000","last_clean":"2024-04-25T16:31:37.948845+0000","last_became_active":"2024-04-25T16:30:09.990644+0000","last_became_peered":"2024-04-25T16:30:09.990644+0000","last_unstale":"2024-04-25T16:31:37.948845+0000","last_undegraded":"2024-04-25T16:31:37.948845+0000","last_fullsized":"2024-04-25T16:31:37.948845+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:42.175520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002912+0000","last_change":"2024-04-25T16:30:08.988315+0000","last_active":"2024-04-25T16:31:09.002912+0000","last_peered":"2024-04-25T16:31:09.002912+0000","last_clean":"2024-04-25T16:31:09.002912+0000","last_became_active":"2024-04-25T16:30:08.987930+0000","last_became_peered":"2024-04-25T16:30:08.987930+0000","last_unstale":"2024-04-25T16:31:09.002912+0000","last_undegraded":"2024-04-25T16:31:09.002912+0000","last_fullsized":"2024-04-25T16:31:09.002912+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:44.884602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005926+0000","last_change":"2024-04-25T16:28:58.219510+0000","last_active":"2024-04-25T16:31:09.005926+0000","last_peered":"2024-04-25T16:31:09.005926+0000","last_clean":"2024-04-25T16:31:09.005926+0000","last_became_active":"2024-04-25T16:28:58.213888+0000","last_became_peered":"2024-04-25T16:28:58.213888+0000","last_unstale":"2024-04-25T16:31:09.005926+0000","last_undegraded":"2024-04-25T16:31:09.005926+0000","last_fullsized":"2024-04-25T16:31:09.005926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_clean_scrub_stamp":"2024-04-25T16:25:33.939186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:15.378406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.012831+0000","last_change":"2024-04-25T16:31:09.012831+0000","last_active":"2024-04-25T16:31:09.012831+0000","last_peered":"2024-04-25T16:31:09.012831+0000","last_clean":"2024-04-25T16:31:09.012831+0000","last_became_active":"2024-04-25T16:31:09.012375+0000","last_became_peered":"2024-04-25T16:31:09.012375+0000","last_unstale":"2024-04-25T16:31:09.012831+0000","last_undegraded":"2024-04-25T16:31:09.012831+0000","last_fullsized":"2024-04-25T16:31:09.012831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:02.214734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944037+0000","last_change":"2024-04-25T16:30:10.005210+0000","last_active":"2024-04-25T16:31:37.944037+0000","last_peered":"2024-04-25T16:31:37.944037+0000","last_clean":"2024-04-25T16:31:37.944037+0000","last_became_active":"2024-04-25T16:30:10.004936+0000","last_became_peered":"2024-04-25T16:30:10.004936+0000","last_unstale":"2024-04-25T16:31:37.944037+0000","last_undegraded":"2024-04-25T16:31:37.944037+0000","last_fullsized":"2024-04-25T16:31:37.944037+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:53.292675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005622+0000","last_change":"2024-04-25T16:30:08.990162+0000","last_active":"2024-04-25T16:31:09.005622+0000","last_peered":"2024-04-25T16:31:09.005622+0000","last_clean":"2024-04-25T16:31:09.005622+0000","last_became_active":"2024-04-25T16:30:08.989613+0000","last_became_peered":"2024-04-25T16:30:08.989613+0000","last_unstale":"2024-04-25T16:31:09.005622+0000","last_undegraded":"2024-04-25T16:31:09.005622+0000","last_fullsized":"2024-04-25T16:31:09.005622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:41.595381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.010109+0000","last_change":"2024-04-25T16:30:14.001850+0000","last_active":"2024-04-25T16:31:09.010109+0000","last_peered":"2024-04-25T16:31:09.010109+0000","last_clean":"2024-04-25T16:31:09.010109+0000","last_became_active":"2024-04-25T16:30:14.001348+0000","last_became_peered":"2024-04-25T16:30:14.001348+0000","last_unstale":"2024-04-25T16:31:09.010109+0000","last_undegraded":"2024-04-25T16:31:09.010109+0000","last_fullsized":"2024-04-25T16:31:09.010109+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:56:41.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.007060+0000","last_change":"2024-04-25T16:30:08.995833+0000","last_active":"2024-04-25T16:31:09.007060+0000","last_peered":"2024-04-25T16:31:09.007060+0000","last_clean":"2024-04-25T16:31:09.007060+0000","last_became_active":"2024-04-25T16:30:08.995545+0000","last_became_peered":"2024-04-25T16:30:08.995545+0000","last_unstale":"2024-04-25T16:31:09.007060+0000","last_undegraded":"2024-04-25T16:31:09.007060+0000","last_fullsized":"2024-04-25T16:31:09.007060+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:31.452219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.936626+0000","last_change":"2024-04-25T16:30:09.990225+0000","last_active":"2024-04-25T16:31:37.936626+0000","last_peered":"2024-04-25T16:31:37.936626+0000","last_clean":"2024-04-25T16:31:37.936626+0000","last_became_active":"2024-04-25T16:30:09.989859+0000","last_became_peered":"2024-04-25T16:30:09.989859+0000","last_unstale":"2024-04-25T16:31:37.936626+0000","last_undegraded":"2024-04-25T16:31:37.936626+0000","last_fullsized":"2024-04-25T16:31:37.936626+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:40.589395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.185069+0000","last_change":"2024-04-25T16:30:14.147223+0000","last_active":"2024-04-25T16:31:09.185069+0000","last_peered":"2024-04-25T16:31:09.185069+0000","last_clean":"2024-04-25T16:31:09.185069+0000","last_became_active":"2024-04-25T16:30:14.146821+0000","last_became_peered":"2024-04-25T16:30:14.146821+0000","last_unstale":"2024-04-25T16:31:09.185069+0000","last_undegraded":"2024-04-25T16:31:09.185069+0000","last_fullsized":"2024-04-25T16:31:09.185069+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:01.289165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'13","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002175+0000","last_change":"2024-04-25T16:30:08.993175+0000","last_active":"2024-04-25T16:31:09.002175+0000","last_peered":"2024-04-25T16:31:09.002175+0000","last_clean":"2024-04-25T16:31:09.002175+0000","last_became_active":"2024-04-25T16:30:08.992145+0000","last_became_peered":"2024-04-25T16:30:08.992145+0000","last_unstale":"2024-04-25T16:31:09.002175+0000","last_undegraded":"2024-04-25T16:31:09.002175+0000","last_fullsized":"2024-04-25T16:31:09.002175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:06.951435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.947300+0000","last_change":"2024-04-25T16:30:09.986088+0000","last_active":"2024-04-25T16:31:37.947300+0000","last_peered":"2024-04-25T16:31:37.947300+0000","last_clean":"2024-04-25T16:31:37.947300+0000","last_became_active":"2024-04-25T16:30:09.985256+0000","last_became_peered":"2024-04-25T16:30:09.985256+0000","last_unstale":"2024-04-25T16:31:37.947300+0000","last_undegraded":"2024-04-25T16:31:37.947300+0000","last_fullsized":"2024-04-25T16:31:37.947300+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:29.197558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001709+0000","last_change":"2024-04-25T16:30:14.000699+0000","last_active":"2024-04-25T16:31:09.001709+0000","last_peered":"2024-04-25T16:31:09.001709+0000","last_clean":"2024-04-25T16:31:09.001709+0000","last_became_active":"2024-04-25T16:30:14.000255+0000","last_became_peered":"2024-04-25T16:30:14.000255+0000","last_unstale":"2024-04-25T16:31:09.001709+0000","last_undegraded":"2024-04-25T16:31:09.001709+0000","last_fullsized":"2024-04-25T16:31:09.001709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:33.436693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.003974+0000","last_change":"2024-04-25T16:30:08.981926+0000","last_active":"2024-04-25T16:31:09.003974+0000","last_peered":"2024-04-25T16:31:09.003974+0000","last_clean":"2024-04-25T16:31:09.003974+0000","last_became_active":"2024-04-25T16:30:08.981590+0000","last_became_peered":"2024-04-25T16:30:08.981590+0000","last_unstale":"2024-04-25T16:31:09.003974+0000","last_undegraded":"2024-04-25T16:31:09.003974+0000","last_fullsized":"2024-04-25T16:31:09.003974+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:57.818774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951291+0000","last_change":"2024-04-25T16:30:10.003998+0000","last_active":"2024-04-25T16:31:37.951291+0000","last_peered":"2024-04-25T16:31:37.951291+0000","last_clean":"2024-04-25T16:31:37.951291+0000","last_became_active":"2024-04-25T16:30:10.003626+0000","last_became_peered":"2024-04-25T16:30:10.003626+0000","last_unstale":"2024-04-25T16:31:37.951291+0000","last_undegraded":"2024-04-25T16:31:37.951291+0000","last_fullsized":"2024-04-25T16:31:37.951291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:10.686092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002820+0000","last_change":"2024-04-25T16:30:14.149278+0000","last_active":"2024-04-25T16:31:09.002820+0000","last_peered":"2024-04-25T16:31:09.002820+0000","last_clean":"2024-04-25T16:31:09.002820+0000","last_became_active":"2024-04-25T16:30:14.148654+0000","last_became_peered":"2024-04-25T16:30:14.148654+0000","last_unstale":"2024-04-25T16:31:09.002820+0000","last_undegraded":"2024-04-25T16:31:09.002820+0000","last_fullsized":"2024-04-25T16:31:09.002820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:24.823222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.184459+0000","last_change":"2024-04-25T16:30:08.981612+0000","last_active":"2024-04-25T16:31:09.184459+0000","last_peered":"2024-04-25T16:31:09.184459+0000","last_clean":"2024-04-25T16:31:09.184459+0000","last_became_active":"2024-04-25T16:30:08.981222+0000","last_became_peered":"2024-04-25T16:30:08.981222+0000","last_unstale":"2024-04-25T16:31:09.184459+0000","last_undegraded":"2024-04-25T16:31:09.184459+0000","last_fullsized":"2024-04-25T16:31:09.184459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:14.914658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.907101+0000","last_change":"2024-04-25T16:30:09.989592+0000","last_active":"2024-04-25T16:31:37.907101+0000","last_peered":"2024-04-25T16:31:37.907101+0000","last_clean":"2024-04-25T16:31:37.907101+0000","last_became_active":"2024-04-25T16:30:09.989222+0000","last_became_peered":"2024-04-25T16:30:09.989222+0000","last_unstale":"2024-04-25T16:31:37.907101+0000","last_undegraded":"2024-04-25T16:31:37.907101+0000","last_fullsized":"2024-04-25T16:31:37.907101+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:52.403239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002346+0000","last_change":"2024-04-25T16:30:14.010480+0000","last_active":"2024-04-25T16:31:09.002346+0000","last_peered":"2024-04-25T16:31:09.002346+0000","last_clean":"2024-04-25T16:31:09.002346+0000","last_became_active":"2024-04-25T16:30:14.009023+0000","last_became_peered":"2024-04-25T16:30:14.009023+0000","last_unstale":"2024-04-25T16:31:09.002346+0000","last_undegraded":"2024-04-25T16:31:09.002346+0000","last_fullsized":"2024-04-25T16:31:09.002346+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:10.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.185374+0000","last_change":"2024-04-25T16:30:08.986849+0000","last_active":"2024-04-25T16:31:09.185374+0000","last_peered":"2024-04-25T16:31:09.185374+0000","last_clean":"2024-04-25T16:31:09.185374+0000","last_became_active":"2024-04-25T16:30:08.986479+0000","last_became_peered":"2024-04-25T16:30:08.986479+0000","last_unstale":"2024-04-25T16:31:09.185374+0000","last_undegraded":"2024-04-25T16:31:09.185374+0000","last_fullsized":"2024-04-25T16:31:09.185374+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.950372+0000","last_change":"2024-04-25T16:30:09.990927+0000","last_active":"2024-04-25T16:31:37.950372+0000","last_peered":"2024-04-25T16:31:37.950372+0000","last_clean":"2024-04-25T16:31:37.950372+0000","last_became_active":"2024-04-25T16:30:09.990482+0000","last_became_peered":"2024-04-25T16:30:09.990482+0000","last_unstale":"2024-04-25T16:31:37.950372+0000","last_undegraded":"2024-04-25T16:31:37.950372+0000","last_fullsized":"2024-04-25T16:31:37.950372+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:18.649721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.003105+0000","last_change":"2024-04-25T16:30:14.006159+0000","last_active":"2024-04-25T16:31:09.003105+0000","last_peered":"2024-04-25T16:31:09.003105+0000","last_clean":"2024-04-25T16:31:09.003105+0000","last_became_active":"2024-04-25T16:30:14.005413+0000","last_became_peered":"2024-04-25T16:30:14.005413+0000","last_unstale":"2024-04-25T16:31:09.003105+0000","last_undegraded":"2024-04-25T16:31:09.003105+0000","last_fullsized":"2024-04-25T16:31:09.003105+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:52.934385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001913+0000","last_change":"2024-04-25T16:30:08.986207+0000","last_active":"2024-04-25T16:31:09.001913+0000","last_peered":"2024-04-25T16:31:09.001913+0000","last_clean":"2024-04-25T16:31:09.001913+0000","last_became_active":"2024-04-25T16:30:08.985857+0000","last_became_peered":"2024-04-25T16:30:08.985857+0000","last_unstale":"2024-04-25T16:31:09.001913+0000","last_undegraded":"2024-04-25T16:31:09.001913+0000","last_fullsized":"2024-04-25T16:31:09.001913+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:45.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.939780+0000","last_change":"2024-04-25T16:30:09.993566+0000","last_active":"2024-04-25T16:31:37.939780+0000","last_peered":"2024-04-25T16:31:37.939780+0000","last_clean":"2024-04-25T16:31:37.939780+0000","last_became_active":"2024-04-25T16:30:09.992250+0000","last_became_peered":"2024-04-25T16:30:09.992250+0000","last_unstale":"2024-04-25T16:31:37.939780+0000","last_undegraded":"2024-04-25T16:31:37.939780+0000","last_fullsized":"2024-04-25T16:31:37.939780+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:33.279300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002881+0000","last_change":"2024-04-25T16:30:14.147443+0000","last_active":"2024-04-25T16:31:09.002881+0000","last_peered":"2024-04-25T16:31:09.002881+0000","last_clean":"2024-04-25T16:31:09.002881+0000","last_became_active":"2024-04-25T16:30:14.146838+0000","last_became_peered":"2024-04-25T16:30:14.146838+0000","last_unstale":"2024-04-25T16:31:09.002881+0000","last_undegraded":"2024-04-25T16:31:09.002881+0000","last_fullsized":"2024-04-25T16:31:09.002881+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:51.532685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.015526+0000","last_change":"2024-04-25T16:31:09.015526+0000","last_active":"2024-04-25T16:31:09.015526+0000","last_peered":"2024-04-25T16:31:09.015526+0000","last_clean":"2024-04-25T16:31:09.015526+0000","last_became_active":"2024-04-25T16:31:09.007972+0000","last_became_peered":"2024-04-25T16:31:09.007972+0000","last_unstale":"2024-04-25T16:31:09.015526+0000","last_undegraded":"2024-04-25T16:31:09.015526+0000","last_fullsized":"2024-04-25T16:31:09.015526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:56.464426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963806+0000","last_change":"2024-04-25T16:30:09.998486+0000","last_active":"2024-04-25T16:31:37.963806+0000","last_peered":"2024-04-25T16:31:37.963806+0000","last_clean":"2024-04-25T16:31:37.963806+0000","last_became_active":"2024-04-25T16:30:09.998154+0000","last_became_peered":"2024-04-25T16:30:09.998154+0000","last_unstale":"2024-04-25T16:31:37.963806+0000","last_undegraded":"2024-04-25T16:31:37.963806+0000","last_fullsized":"2024-04-25T16:31:37.963806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:55.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.184576+0000","last_change":"2024-04-25T16:30:13.995864+0000","last_active":"2024-04-25T16:31:09.184576+0000","last_peered":"2024-04-25T16:31:09.184576+0000","last_clean":"2024-04-25T16:31:09.184576+0000","last_became_active":"2024-04-25T16:30:13.995326+0000","last_became_peered":"2024-04-25T16:30:13.995326+0000","last_unstale":"2024-04-25T16:31:09.184576+0000","last_undegraded":"2024-04-25T16:31:09.184576+0000","last_fullsized":"2024-04-25T16:31:09.184576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:11.718826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":99,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:41.183807+0000","last_change":"2024-04-25T16:30:08.984189+0000","last_active":"2024-04-25T16:31:41.183807+0000","last_peered":"2024-04-25T16:31:41.183807+0000","last_clean":"2024-04-25T16:31:41.183807+0000","last_became_active":"2024-04-25T16:30:08.983821+0000","last_became_peered":"2024-04-25T16:30:08.983821+0000","last_unstale":"2024-04-25T16:31:41.183807+0000","last_undegraded":"2024-04-25T16:31:41.183807+0000","last_fullsized":"2024-04-25T16:31:41.183807+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:51.315397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963625+0000","last_change":"2024-04-25T16:30:09.998763+0000","last_active":"2024-04-25T16:31:37.963625+0000","last_peered":"2024-04-25T16:31:37.963625+0000","last_clean":"2024-04-25T16:31:37.963625+0000","last_became_active":"2024-04-25T16:30:09.998439+0000","last_became_peered":"2024-04-25T16:30:09.998439+0000","last_unstale":"2024-04-25T16:31:37.963625+0000","last_undegraded":"2024-04-25T16:31:37.963625+0000","last_fullsized":"2024-04-25T16:31:37.963625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:14.279883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.472083+0000","last_change":"2024-04-25T16:30:14.005427+0000","last_active":"2024-04-25T16:31:09.472083+0000","last_peered":"2024-04-25T16:31:09.472083+0000","last_clean":"2024-04-25T16:31:09.472083+0000","last_became_active":"2024-04-25T16:30:14.005006+0000","last_became_peered":"2024-04-25T16:30:14.005006+0000","last_unstale":"2024-04-25T16:31:09.472083+0000","last_undegraded":"2024-04-25T16:31:09.472083+0000","last_fullsized":"2024-04-25T16:31:09.472083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:49.563365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006221+0000","last_change":"2024-04-25T16:30:08.985557+0000","last_active":"2024-04-25T16:31:09.006221+0000","last_peered":"2024-04-25T16:31:09.006221+0000","last_clean":"2024-04-25T16:31:09.006221+0000","last_became_active":"2024-04-25T16:30:08.985215+0000","last_became_peered":"2024-04-25T16:30:08.985215+0000","last_unstale":"2024-04-25T16:31:09.006221+0000","last_undegraded":"2024-04-25T16:31:09.006221+0000","last_fullsized":"2024-04-25T16:31:09.006221+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:51.504089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.947056+0000","last_change":"2024-04-25T16:30:09.991336+0000","last_active":"2024-04-25T16:31:37.947056+0000","last_peered":"2024-04-25T16:31:37.947056+0000","last_clean":"2024-04-25T16:31:37.947056+0000","last_became_active":"2024-04-25T16:30:09.990819+0000","last_became_peered":"2024-04-25T16:30:09.990819+0000","last_unstale":"2024-04-25T16:31:37.947056+0000","last_undegraded":"2024-04-25T16:31:37.947056+0000","last_fullsized":"2024-04-25T16:31:37.947056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:45.988061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.004993+0000","last_change":"2024-04-25T16:30:13.998463+0000","last_active":"2024-04-25T16:31:09.004993+0000","last_peered":"2024-04-25T16:31:09.004993+0000","last_clean":"2024-04-25T16:31:09.004993+0000","last_became_active":"2024-04-25T16:30:13.997952+0000","last_became_peered":"2024-04-25T16:30:13.997952+0000","last_unstale":"2024-04-25T16:31:09.004993+0000","last_undegraded":"2024-04-25T16:31:09.004993+0000","last_fullsized":"2024-04-25T16:31:09.004993+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:58.507942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005908+0000","last_change":"2024-04-25T16:30:08.987232+0000","last_active":"2024-04-25T16:31:09.005908+0000","last_peered":"2024-04-25T16:31:09.005908+0000","last_clean":"2024-04-25T16:31:09.005908+0000","last_became_active":"2024-04-25T16:30:08.986873+0000","last_became_peered":"2024-04-25T16:30:08.986873+0000","last_unstale":"2024-04-25T16:31:09.005908+0000","last_undegraded":"2024-04-25T16:31:09.005908+0000","last_fullsized":"2024-04-25T16:31:09.005908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:40.746479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.956754+0000","last_change":"2024-04-25T16:30:09.988566+0000","last_active":"2024-04-25T16:31:37.956754+0000","last_peered":"2024-04-25T16:31:37.956754+0000","last_clean":"2024-04-25T16:31:37.956754+0000","last_became_active":"2024-04-25T16:30:09.988164+0000","last_became_peered":"2024-04-25T16:30:09.988164+0000","last_unstale":"2024-04-25T16:31:37.956754+0000","last_undegraded":"2024-04-25T16:31:37.956754+0000","last_fullsized":"2024-04-25T16:31:37.956754+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:40.476766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.950950+0000","last_change":"2024-04-25T16:30:09.996349+0000","last_active":"2024-04-25T16:31:37.950950+0000","last_peered":"2024-04-25T16:31:37.950950+0000","last_clean":"2024-04-25T16:31:37.950950+0000","last_became_active":"2024-04-25T16:30:09.994974+0000","last_became_peered":"2024-04-25T16:30:09.994974+0000","last_unstale":"2024-04-25T16:31:37.950950+0000","last_undegraded":"2024-04-25T16:31:37.950950+0000","last_fullsized":"2024-04-25T16:31:37.950950+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:46.929057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951802+0000","last_change":"2024-04-25T16:30:09.997837+0000","last_active":"2024-04-25T16:31:37.951802+0000","last_peered":"2024-04-25T16:31:37.951802+0000","last_clean":"2024-04-25T16:31:37.951802+0000","last_became_active":"2024-04-25T16:30:09.997361+0000","last_became_peered":"2024-04-25T16:30:09.997361+0000","last_unstale":"2024-04-25T16:31:37.951802+0000","last_undegraded":"2024-04-25T16:31:37.951802+0000","last_fullsized":"2024-04-25T16:31:37.951802+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:35.698528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.942275+0000","last_change":"2024-04-25T16:30:09.991794+0000","last_active":"2024-04-25T16:31:37.942275+0000","last_peered":"2024-04-25T16:31:37.942275+0000","last_clean":"2024-04-25T16:31:37.942275+0000","last_became_active":"2024-04-25T16:30:09.991440+0000","last_became_peered":"2024-04-25T16:30:09.991440+0000","last_unstale":"2024-04-25T16:31:37.942275+0000","last_undegraded":"2024-04-25T16:31:37.942275+0000","last_fullsized":"2024-04-25T16:31:37.942275+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:11.322254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951162+0000","last_change":"2024-04-25T16:30:10.002364+0000","last_active":"2024-04-25T16:31:37.951162+0000","last_peered":"2024-04-25T16:31:37.951162+0000","last_clean":"2024-04-25T16:31:37.951162+0000","last_became_active":"2024-04-25T16:30:10.001957+0000","last_became_peered":"2024-04-25T16:30:10.001957+0000","last_unstale":"2024-04-25T16:31:37.951162+0000","last_undegraded":"2024-04-25T16:31:37.951162+0000","last_fullsized":"2024-04-25T16:31:37.951162+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:19.150175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944711+0000","last_change":"2024-04-25T16:30:10.004150+0000","last_active":"2024-04-25T16:31:37.944711+0000","last_peered":"2024-04-25T16:31:37.944711+0000","last_clean":"2024-04-25T16:31:37.944711+0000","last_became_active":"2024-04-25T16:30:10.003733+0000","last_became_peered":"2024-04-25T16:30:10.003733+0000","last_unstale":"2024-04-25T16:31:37.944711+0000","last_undegraded":"2024-04-25T16:31:37.944711+0000","last_fullsized":"2024-04-25T16:31:37.944711+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:11.850366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.943924+0000","last_change":"2024-04-25T16:30:09.991056+0000","last_active":"2024-04-25T16:31:37.943924+0000","last_peered":"2024-04-25T16:31:37.943924+0000","last_clean":"2024-04-25T16:31:37.943924+0000","last_became_active":"2024-04-25T16:30:09.988708+0000","last_became_peered":"2024-04-25T16:30:09.988708+0000","last_unstale":"2024-04-25T16:31:37.943924+0000","last_undegraded":"2024-04-25T16:31:37.943924+0000","last_fullsized":"2024-04-25T16:31:37.943924+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:13.062269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.948518+0000","last_change":"2024-04-25T16:30:09.996571+0000","last_active":"2024-04-25T16:31:37.948518+0000","last_peered":"2024-04-25T16:31:37.948518+0000","last_clean":"2024-04-25T16:31:37.948518+0000","last_became_active":"2024-04-25T16:30:09.996206+0000","last_became_peered":"2024-04-25T16:30:09.996206+0000","last_unstale":"2024-04-25T16:31:37.948518+0000","last_undegraded":"2024-04-25T16:31:37.948518+0000","last_fullsized":"2024-04-25T16:31:37.948518+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:07.107949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.942401+0000","last_change":"2024-04-25T16:30:09.995067+0000","last_active":"2024-04-25T16:31:37.942401+0000","last_peered":"2024-04-25T16:31:37.942401+0000","last_clean":"2024-04-25T16:31:37.942401+0000","last_became_active":"2024-04-25T16:30:09.994678+0000","last_became_peered":"2024-04-25T16:30:09.994678+0000","last_unstale":"2024-04-25T16:31:37.942401+0000","last_undegraded":"2024-04-25T16:31:37.942401+0000","last_fullsized":"2024-04-25T16:31:37.942401+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:09.000869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.953243+0000","last_change":"2024-04-25T16:30:09.996821+0000","last_active":"2024-04-25T16:31:37.953243+0000","last_peered":"2024-04-25T16:31:37.953243+0000","last_clean":"2024-04-25T16:31:37.953243+0000","last_became_active":"2024-04-25T16:30:09.996464+0000","last_became_peered":"2024-04-25T16:30:09.996464+0000","last_unstale":"2024-04-25T16:31:37.953243+0000","last_undegraded":"2024-04-25T16:31:37.953243+0000","last_fullsized":"2024-04-25T16:31:37.953243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:54.961544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.948787+0000","last_change":"2024-04-25T16:30:09.991733+0000","last_active":"2024-04-25T16:31:37.948787+0000","last_peered":"2024-04-25T16:31:37.948787+0000","last_clean":"2024-04-25T16:31:37.948787+0000","last_became_active":"2024-04-25T16:30:09.991401+0000","last_became_peered":"2024-04-25T16:30:09.991401+0000","last_unstale":"2024-04-25T16:31:37.948787+0000","last_undegraded":"2024-04-25T16:31:37.948787+0000","last_fullsized":"2024-04-25T16:31:37.948787+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:14.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.964113+0000","last_change":"2024-04-25T16:30:09.999087+0000","last_active":"2024-04-25T16:31:37.964113+0000","last_peered":"2024-04-25T16:31:37.964113+0000","last_clean":"2024-04-25T16:31:37.964113+0000","last_became_active":"2024-04-25T16:30:09.998725+0000","last_became_peered":"2024-04-25T16:30:09.998725+0000","last_unstale":"2024-04-25T16:31:37.964113+0000","last_undegraded":"2024-04-25T16:31:37.964113+0000","last_fullsized":"2024-04-25T16:31:37.964113+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:19.155171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.950902+0000","last_change":"2024-04-25T16:30:09.995880+0000","last_active":"2024-04-25T16:31:37.950902+0000","last_peered":"2024-04-25T16:31:37.950902+0000","last_clean":"2024-04-25T16:31:37.950902+0000","last_became_active":"2024-04-25T16:30:09.995514+0000","last_became_peered":"2024-04-25T16:30:09.995514+0000","last_unstale":"2024-04-25T16:31:37.950902+0000","last_undegraded":"2024-04-25T16:31:37.950902+0000","last_fullsized":"2024-04-25T16:31:37.950902+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.946875+0000","last_change":"2024-04-25T16:30:09.998326+0000","last_active":"2024-04-25T16:31:37.946875+0000","last_peered":"2024-04-25T16:31:37.946875+0000","last_clean":"2024-04-25T16:31:37.946875+0000","last_became_active":"2024-04-25T16:30:09.995803+0000","last_became_peered":"2024-04-25T16:30:09.995803+0000","last_unstale":"2024-04-25T16:31:37.946875+0000","last_undegraded":"2024-04-25T16:31:37.946875+0000","last_fullsized":"2024-04-25T16:31:37.946875+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:46.983947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71091,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":709,"num_read_kb":435,"num_write":251,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":213273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":456380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841574,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36936,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708216,"statfs":{"total":95995035648,"available":95957213184,"internally_reserved":0,"allocated":1638400,"data_stored":1111429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037802,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":460406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234032,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":456688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430260,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1053858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593786,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":451814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":516449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":453270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":517201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":451522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953541,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":456420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790015,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36860,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708292,"statfs":{"total":95995035648,"available":95957291008,"internally_reserved":0,"allocated":1560576,"data_stored":1044812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:31:49.054 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T16:31:49.298 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:49.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:49 smithi098 ceph-mon[29162]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:31:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:49 smithi203 ceph-mon[33115]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:31:49.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:49 smithi173 ceph-mon[32556]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:31:50.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:50 smithi203 ceph-mon[33115]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:31:50.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:50 smithi098 ceph-mon[29162]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:31:50.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:50 smithi173 ceph-mon[32556]: pgmap v334: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:31:51.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 -- 172.21.15.98:0/2452297275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90102990 msgr2=0x7f6e90102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:51.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2452297275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90102990 0x7f6e90102da0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6e800099c0 tx=0x7f6e80009cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 -- 172.21.15.98:0/2452297275 shutdown_connections 2024-04-25T16:31:51.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2452297275 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e90109700 0x7f6e90111060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2452297275 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90102990 0x7f6e90102da0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2452297275 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e90108db0 0x7f6e901091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 -- 172.21.15.98:0/2452297275 >> 172.21.15.98:0/2452297275 conn(0x7f6e900fe370 msgr2=0x7f6e901007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 -- 172.21.15.98:0/2452297275 shutdown_connections 2024-04-25T16:31:51.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.182+0000 7f6e94a58700 1 -- 172.21.15.98:0/2452297275 wait complete. 2024-04-25T16:31:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.183+0000 7f6e94a58700 1 Processor -- start 2024-04-25T16:31:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.183+0000 7f6e94a58700 1 -- start start 2024-04-25T16:31:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.183+0000 7f6e94a58700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e90102990 0x7f6e901a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:51.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.183+0000 7f6e94a58700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e90108db0 0x7f6e901a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.183+0000 7f6e94a58700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90109700 0x7f6e9019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.183+0000 7f6e94a58700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e90114230 con 0x7f6e90109700 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.183+0000 7f6e94a58700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901140b0 con 0x7f6e90108db0 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.183+0000 7f6e94a58700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e901143b0 con 0x7f6e90102990 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90109700 0x7f6e9019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e90102990 0x7f6e901a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90109700 0x7f6e9019aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38230/0 (socket says 172.21.15.98:38230) 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8e59c700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e90102990 0x7f6e901a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:39102/0 (socket says 172.21.15.98:39102) 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 -- 172.21.15.98:0/2277932691 learned_addr learned my addr 172.21.15.98:0/2277932691 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8dd9b700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e90108db0 0x7f6e901a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 -- 172.21.15.98:0/2277932691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e90102990 msgr2=0x7f6e901a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e90102990 0x7f6e901a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 -- 172.21.15.98:0/2277932691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e90108db0 msgr2=0x7f6e901a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e90108db0 0x7f6e901a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 -- 172.21.15.98:0/2277932691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e9019b140 con 0x7f6e90109700 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8e59c700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e90102990 0x7f6e901a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e8ed9d700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90109700 0x7f6e9019aa90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800ef30 tx=0x7f6e8800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e7f7fe700 1 -- 172.21.15.98:0/2277932691 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e880092e0 con 0x7f6e90109700 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e9019b3d0 con 0x7f6e90109700 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e7f7fe700 1 -- 172.21.15.98:0/2277932691 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e88004bb0 con 0x7f6e90109700 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e7f7fe700 1 -- 172.21.15.98:0/2277932691 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88004020 con 0x7f6e90109700 2024-04-25T16:31:51.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.184+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e9019b8c0 con 0x7f6e90109700 2024-04-25T16:31:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.185+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e90051190 con 0x7f6e90109700 2024-04-25T16:31:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.187+0000 7f6e7f7fe700 1 -- 172.21.15.98:0/2277932691 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6e8801b030 con 0x7f6e90109700 2024-04-25T16:31:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.187+0000 7f6e7f7fe700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e780afb80 0x7f6e780b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.188+0000 7f6e7f7fe700 1 -- 172.21.15.98:0/2277932691 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f6e880df390 con 0x7f6e90109700 2024-04-25T16:31:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.188+0000 7f6e8e59c700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e780afb80 0x7f6e780b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:51.187 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.188+0000 7f6e8e59c700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e780afb80 0x7f6e780b2090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6e80000c00 tx=0x7f6e8001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:51.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.191+0000 7f6e7f7fe700 1 -- 172.21.15.98:0/2277932691 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e9019b8c0 con 0x7f6e90109700 2024-04-25T16:31:51.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.385+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6e90063ad0 con 0x7f6e780afb80 2024-04-25T16:31:51.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.394+0000 7f6e7f7fe700 1 -- 172.21.15.98:0/2277932691 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340499 (secure 0 0 0) 0x7f6e8007b230 con 0x7f6e780afb80 2024-04-25T16:31:51.398 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e780afb80 msgr2=0x7f6e780b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:51.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e780afb80 0x7f6e780b2090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6e80000c00 tx=0x7f6e8001a040 comp rx=0 tx=0).stop 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90109700 msgr2=0x7f6e9019aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90109700 0x7f6e9019aa90 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800ef30 tx=0x7f6e8800c3a0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 shutdown_connections 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e780afb80 0x7f6e780b2090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e90102990 0x7f6e901a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e90109700 0x7f6e9019aa90 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 --2- 172.21.15.98:0/2277932691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e90108db0 0x7f6e901a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 >> 172.21.15.98:0/2277932691 conn(0x7f6e900fe370 msgr2=0x7f6e900feb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 shutdown_connections 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:51.397+0000 7f6e94a58700 1 -- 172.21.15.98:0/2277932691 wait complete. 2024-04-25T16:31:51.402 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T16:31:52.197 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-04-25T16:31:49.951063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661693,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":810,"num_read_kb":509,"num_write":376,"num_write_kb":2300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2951957,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":813,"ondisk_log_size":813,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":395568,"kb_used_data":13136,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124546256,"statfs":{"total":1151940427776,"available":1151535366144,"internally_reserved":0,"allocated":13451264,"data_stored":7430249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":128,"num_read_kb":63,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003972"},"pg_stats":[{"pgid":"3.3f","version":"83'8","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.949318+0000","last_change":"2024-04-25T16:30:09.990700+0000","last_active":"2024-04-25T16:31:37.949318+0000","last_peered":"2024-04-25T16:31:37.949318+0000","last_clean":"2024-04-25T16:31:37.949318+0000","last_became_active":"2024-04-25T16:30:09.990225+0000","last_became_peered":"2024-04-25T16:30:09.990225+0000","last_unstale":"2024-04-25T16:31:37.949318+0000","last_undegraded":"2024-04-25T16:31:37.949318+0000","last_fullsized":"2024-04-25T16:31:37.949318+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:33.897277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.947429+0000","last_change":"2024-04-25T16:30:09.989618+0000","last_active":"2024-04-25T16:31:37.947429+0000","last_peered":"2024-04-25T16:31:37.947429+0000","last_clean":"2024-04-25T16:31:37.947429+0000","last_became_active":"2024-04-25T16:30:09.989270+0000","last_became_peered":"2024-04-25T16:30:09.989270+0000","last_unstale":"2024-04-25T16:31:37.947429+0000","last_undegraded":"2024-04-25T16:31:37.947429+0000","last_fullsized":"2024-04-25T16:31:37.947429+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:03.952233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.949748+0000","last_change":"2024-04-25T16:30:09.992442+0000","last_active":"2024-04-25T16:31:37.949748+0000","last_peered":"2024-04-25T16:31:37.949748+0000","last_clean":"2024-04-25T16:31:37.949748+0000","last_became_active":"2024-04-25T16:30:09.992115+0000","last_became_peered":"2024-04-25T16:30:09.992115+0000","last_unstale":"2024-04-25T16:31:37.949748+0000","last_undegraded":"2024-04-25T16:31:37.949748+0000","last_fullsized":"2024-04-25T16:31:37.949748+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:00.668427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.943947+0000","last_change":"2024-04-25T16:31:09.006824+0000","last_active":"2024-04-25T16:31:37.943947+0000","last_peered":"2024-04-25T16:31:37.943947+0000","last_clean":"2024-04-25T16:31:37.943947+0000","last_became_active":"2024-04-25T16:31:09.005560+0000","last_became_peered":"2024-04-25T16:31:09.005560+0000","last_unstale":"2024-04-25T16:31:37.943947+0000","last_undegraded":"2024-04-25T16:31:37.943947+0000","last_fullsized":"2024-04-25T16:31:37.943947+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:41.979642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.928818+0000","last_change":"2024-04-25T16:30:09.991372+0000","last_active":"2024-04-25T16:31:37.928818+0000","last_peered":"2024-04-25T16:31:37.928818+0000","last_clean":"2024-04-25T16:31:37.928818+0000","last_became_active":"2024-04-25T16:30:09.991053+0000","last_became_peered":"2024-04-25T16:30:09.991053+0000","last_unstale":"2024-04-25T16:31:37.928818+0000","last_undegraded":"2024-04-25T16:31:37.928818+0000","last_fullsized":"2024-04-25T16:31:37.928818+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:28.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944476+0000","last_change":"2024-04-25T16:30:09.997563+0000","last_active":"2024-04-25T16:31:37.944476+0000","last_peered":"2024-04-25T16:31:37.944476+0000","last_clean":"2024-04-25T16:31:37.944476+0000","last_became_active":"2024-04-25T16:30:09.997158+0000","last_became_peered":"2024-04-25T16:30:09.997158+0000","last_unstale":"2024-04-25T16:31:37.944476+0000","last_undegraded":"2024-04-25T16:31:37.944476+0000","last_fullsized":"2024-04-25T16:31:37.944476+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:21.441957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.954206+0000","last_change":"2024-04-25T16:31:09.009888+0000","last_active":"2024-04-25T16:31:37.954206+0000","last_peered":"2024-04-25T16:31:37.954206+0000","last_clean":"2024-04-25T16:31:37.954206+0000","last_became_active":"2024-04-25T16:31:09.009201+0000","last_became_peered":"2024-04-25T16:31:09.009201+0000","last_unstale":"2024-04-25T16:31:37.954206+0000","last_undegraded":"2024-04-25T16:31:37.954206+0000","last_fullsized":"2024-04-25T16:31:37.954206+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:52.171410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.948496+0000","last_change":"2024-04-25T16:30:09.994820+0000","last_active":"2024-04-25T16:31:37.948496+0000","last_peered":"2024-04-25T16:31:37.948496+0000","last_clean":"2024-04-25T16:31:37.948496+0000","last_became_active":"2024-04-25T16:30:09.994494+0000","last_became_peered":"2024-04-25T16:30:09.994494+0000","last_unstale":"2024-04-25T16:31:37.948496+0000","last_undegraded":"2024-04-25T16:31:37.948496+0000","last_fullsized":"2024-04-25T16:31:37.948496+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:39.373266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963287+0000","last_change":"2024-04-25T16:30:09.994870+0000","last_active":"2024-04-25T16:31:37.963287+0000","last_peered":"2024-04-25T16:31:37.963287+0000","last_clean":"2024-04-25T16:31:37.963287+0000","last_became_active":"2024-04-25T16:30:09.994521+0000","last_became_peered":"2024-04-25T16:30:09.994521+0000","last_unstale":"2024-04-25T16:31:37.963287+0000","last_undegraded":"2024-04-25T16:31:37.963287+0000","last_fullsized":"2024-04-25T16:31:37.963287+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:18.093904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":47,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.945320+0000","last_change":"2024-04-25T16:31:09.010148+0000","last_active":"2024-04-25T16:31:37.945320+0000","last_peered":"2024-04-25T16:31:37.945320+0000","last_clean":"2024-04-25T16:31:37.945320+0000","last_became_active":"2024-04-25T16:31:09.009655+0000","last_became_peered":"2024-04-25T16:31:09.009655+0000","last_unstale":"2024-04-25T16:31:37.945320+0000","last_undegraded":"2024-04-25T16:31:37.945320+0000","last_fullsized":"2024-04-25T16:31:37.945320+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:18.637202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963613+0000","last_change":"2024-04-25T16:30:09.999519+0000","last_active":"2024-04-25T16:31:37.963613+0000","last_peered":"2024-04-25T16:31:37.963613+0000","last_clean":"2024-04-25T16:31:37.963613+0000","last_became_active":"2024-04-25T16:30:09.999158+0000","last_became_peered":"2024-04-25T16:30:09.999158+0000","last_unstale":"2024-04-25T16:31:37.963613+0000","last_undegraded":"2024-04-25T16:31:37.963613+0000","last_fullsized":"2024-04-25T16:31:37.963613+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:38:06.686541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944336+0000","last_change":"2024-04-25T16:30:10.005455+0000","last_active":"2024-04-25T16:31:37.944336+0000","last_peered":"2024-04-25T16:31:37.944336+0000","last_clean":"2024-04-25T16:31:37.944336+0000","last_became_active":"2024-04-25T16:30:10.004891+0000","last_became_peered":"2024-04-25T16:30:10.004891+0000","last_unstale":"2024-04-25T16:31:37.944336+0000","last_undegraded":"2024-04-25T16:31:37.944336+0000","last_fullsized":"2024-04-25T16:31:37.944336+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.598321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.943140+0000","last_change":"2024-04-25T16:30:09.992870+0000","last_active":"2024-04-25T16:31:37.943140+0000","last_peered":"2024-04-25T16:31:37.943140+0000","last_clean":"2024-04-25T16:31:37.943140+0000","last_became_active":"2024-04-25T16:30:09.992399+0000","last_became_peered":"2024-04-25T16:30:09.992399+0000","last_unstale":"2024-04-25T16:31:37.943140+0000","last_undegraded":"2024-04-25T16:31:37.943140+0000","last_fullsized":"2024-04-25T16:31:37.943140+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.127287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.952996+0000","last_change":"2024-04-25T16:30:09.988707+0000","last_active":"2024-04-25T16:31:37.952996+0000","last_peered":"2024-04-25T16:31:37.952996+0000","last_clean":"2024-04-25T16:31:37.952996+0000","last_became_active":"2024-04-25T16:30:09.988386+0000","last_became_peered":"2024-04-25T16:30:09.988386+0000","last_unstale":"2024-04-25T16:31:37.952996+0000","last_undegraded":"2024-04-25T16:31:37.952996+0000","last_fullsized":"2024-04-25T16:31:37.952996+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:03:15.973591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951178+0000","last_change":"2024-04-25T16:30:10.002780+0000","last_active":"2024-04-25T16:31:37.951178+0000","last_peered":"2024-04-25T16:31:37.951178+0000","last_clean":"2024-04-25T16:31:37.951178+0000","last_became_active":"2024-04-25T16:30:10.002055+0000","last_became_peered":"2024-04-25T16:30:10.002055+0000","last_unstale":"2024-04-25T16:31:37.951178+0000","last_undegraded":"2024-04-25T16:31:37.951178+0000","last_fullsized":"2024-04-25T16:31:37.951178+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:23.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.962161+0000","last_change":"2024-04-25T16:30:09.998270+0000","last_active":"2024-04-25T16:31:37.962161+0000","last_peered":"2024-04-25T16:31:37.962161+0000","last_clean":"2024-04-25T16:31:37.962161+0000","last_became_active":"2024-04-25T16:30:09.997903+0000","last_became_peered":"2024-04-25T16:30:09.997903+0000","last_unstale":"2024-04-25T16:31:37.962161+0000","last_undegraded":"2024-04-25T16:31:37.962161+0000","last_fullsized":"2024-04-25T16:31:37.962161+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:14.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.949434+0000","last_change":"2024-04-25T16:30:09.988894+0000","last_active":"2024-04-25T16:31:37.949434+0000","last_peered":"2024-04-25T16:31:37.949434+0000","last_clean":"2024-04-25T16:31:37.949434+0000","last_became_active":"2024-04-25T16:30:09.988508+0000","last_became_peered":"2024-04-25T16:30:09.988508+0000","last_unstale":"2024-04-25T16:31:37.949434+0000","last_undegraded":"2024-04-25T16:31:37.949434+0000","last_fullsized":"2024-04-25T16:31:37.949434+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:48.226193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.954793+0000","last_change":"2024-04-25T16:30:09.989910+0000","last_active":"2024-04-25T16:31:37.954793+0000","last_peered":"2024-04-25T16:31:37.954793+0000","last_clean":"2024-04-25T16:31:37.954793+0000","last_became_active":"2024-04-25T16:30:09.989560+0000","last_became_peered":"2024-04-25T16:30:09.989560+0000","last_unstale":"2024-04-25T16:31:37.954793+0000","last_undegraded":"2024-04-25T16:31:37.954793+0000","last_fullsized":"2024-04-25T16:31:37.954793+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:12.834034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.943030+0000","last_change":"2024-04-25T16:30:09.987930+0000","last_active":"2024-04-25T16:31:37.943030+0000","last_peered":"2024-04-25T16:31:37.943030+0000","last_clean":"2024-04-25T16:31:37.943030+0000","last_became_active":"2024-04-25T16:30:09.987595+0000","last_became_peered":"2024-04-25T16:30:09.987595+0000","last_unstale":"2024-04-25T16:31:37.943030+0000","last_undegraded":"2024-04-25T16:31:37.943030+0000","last_fullsized":"2024-04-25T16:31:37.943030+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:04.778172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001831+0000","last_change":"2024-04-25T16:30:14.006360+0000","last_active":"2024-04-25T16:31:09.001831+0000","last_peered":"2024-04-25T16:31:09.001831+0000","last_clean":"2024-04-25T16:31:09.001831+0000","last_became_active":"2024-04-25T16:30:14.005894+0000","last_became_peered":"2024-04-25T16:30:14.005894+0000","last_unstale":"2024-04-25T16:31:09.001831+0000","last_undegraded":"2024-04-25T16:31:09.001831+0000","last_fullsized":"2024-04-25T16:31:09.001831+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:33.150381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":113,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:48.171576+0000","last_change":"2024-04-25T16:30:08.992779+0000","last_active":"2024-04-25T16:31:48.171576+0000","last_peered":"2024-04-25T16:31:48.171576+0000","last_clean":"2024-04-25T16:31:48.171576+0000","last_became_active":"2024-04-25T16:30:08.992458+0000","last_became_peered":"2024-04-25T16:30:08.992458+0000","last_unstale":"2024-04-25T16:31:48.171576+0000","last_undegraded":"2024-04-25T16:31:48.171576+0000","last_fullsized":"2024-04-25T16:31:48.171576+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:21.430855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":73,"num_read_kb":73,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.946116+0000","last_change":"2024-04-25T16:30:09.995353+0000","last_active":"2024-04-25T16:31:37.946116+0000","last_peered":"2024-04-25T16:31:37.946116+0000","last_clean":"2024-04-25T16:31:37.946116+0000","last_became_active":"2024-04-25T16:30:09.995011+0000","last_became_peered":"2024-04-25T16:30:09.995011+0000","last_unstale":"2024-04-25T16:31:37.946116+0000","last_undegraded":"2024-04-25T16:31:37.946116+0000","last_fullsized":"2024-04-25T16:31:37.946116+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:33.367268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.187252+0000","last_change":"2024-04-25T16:31:09.187252+0000","last_active":"2024-04-25T16:31:09.187252+0000","last_peered":"2024-04-25T16:31:09.187252+0000","last_clean":"2024-04-25T16:31:09.187252+0000","last_became_active":"2024-04-25T16:31:09.186689+0000","last_became_peered":"2024-04-25T16:31:09.186689+0000","last_unstale":"2024-04-25T16:31:09.187252+0000","last_undegraded":"2024-04-25T16:31:09.187252+0000","last_fullsized":"2024-04-25T16:31:09.187252+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:18.287476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.185451+0000","last_change":"2024-04-25T16:30:08.997663+0000","last_active":"2024-04-25T16:31:09.185451+0000","last_peered":"2024-04-25T16:31:09.185451+0000","last_clean":"2024-04-25T16:31:09.185451+0000","last_became_active":"2024-04-25T16:30:08.997107+0000","last_became_peered":"2024-04-25T16:30:08.997107+0000","last_unstale":"2024-04-25T16:31:09.185451+0000","last_undegraded":"2024-04-25T16:31:09.185451+0000","last_fullsized":"2024-04-25T16:31:09.185451+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:25.856556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.962081+0000","last_change":"2024-04-25T16:30:09.987947+0000","last_active":"2024-04-25T16:31:37.962081+0000","last_peered":"2024-04-25T16:31:37.962081+0000","last_clean":"2024-04-25T16:31:37.962081+0000","last_became_active":"2024-04-25T16:30:09.987575+0000","last_became_peered":"2024-04-25T16:30:09.987575+0000","last_unstale":"2024-04-25T16:31:37.962081+0000","last_undegraded":"2024-04-25T16:31:37.962081+0000","last_fullsized":"2024-04-25T16:31:37.962081+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:28.443249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002770+0000","last_change":"2024-04-25T16:30:14.006711+0000","last_active":"2024-04-25T16:31:09.002770+0000","last_peered":"2024-04-25T16:31:09.002770+0000","last_clean":"2024-04-25T16:31:09.002770+0000","last_became_active":"2024-04-25T16:30:14.005500+0000","last_became_peered":"2024-04-25T16:30:14.005500+0000","last_unstale":"2024-04-25T16:31:09.002770+0000","last_undegraded":"2024-04-25T16:31:09.002770+0000","last_fullsized":"2024-04-25T16:31:09.002770+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:58:26.288410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.007339+0000","last_change":"2024-04-25T16:30:08.993302+0000","last_active":"2024-04-25T16:31:09.007339+0000","last_peered":"2024-04-25T16:31:09.007339+0000","last_clean":"2024-04-25T16:31:09.007339+0000","last_became_active":"2024-04-25T16:30:08.992918+0000","last_became_peered":"2024-04-25T16:30:08.992918+0000","last_unstale":"2024-04-25T16:31:09.007339+0000","last_undegraded":"2024-04-25T16:31:09.007339+0000","last_fullsized":"2024-04-25T16:31:09.007339+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:36:18.843803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.961810+0000","last_change":"2024-04-25T16:30:09.988216+0000","last_active":"2024-04-25T16:31:37.961810+0000","last_peered":"2024-04-25T16:31:37.961810+0000","last_clean":"2024-04-25T16:31:37.961810+0000","last_became_active":"2024-04-25T16:30:09.987752+0000","last_became_peered":"2024-04-25T16:30:09.987752+0000","last_unstale":"2024-04-25T16:31:37.961810+0000","last_undegraded":"2024-04-25T16:31:37.961810+0000","last_fullsized":"2024-04-25T16:31:37.961810+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:57:35.583000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002375+0000","last_change":"2024-04-25T16:30:14.001329+0000","last_active":"2024-04-25T16:31:09.002375+0000","last_peered":"2024-04-25T16:31:09.002375+0000","last_clean":"2024-04-25T16:31:09.002375+0000","last_became_active":"2024-04-25T16:30:14.000878+0000","last_became_peered":"2024-04-25T16:30:14.000878+0000","last_unstale":"2024-04-25T16:31:09.002375+0000","last_undegraded":"2024-04-25T16:31:09.002375+0000","last_fullsized":"2024-04-25T16:31:09.002375+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:07.946611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.009198+0000","last_change":"2024-04-25T16:30:08.989065+0000","last_active":"2024-04-25T16:31:09.009198+0000","last_peered":"2024-04-25T16:31:09.009198+0000","last_clean":"2024-04-25T16:31:09.009198+0000","last_became_active":"2024-04-25T16:30:08.988709+0000","last_became_peered":"2024-04-25T16:30:08.988709+0000","last_unstale":"2024-04-25T16:31:09.009198+0000","last_undegraded":"2024-04-25T16:31:09.009198+0000","last_fullsized":"2024-04-25T16:31:09.009198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:34.306581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.952822+0000","last_change":"2024-04-25T16:30:09.999641+0000","last_active":"2024-04-25T16:31:37.952822+0000","last_peered":"2024-04-25T16:31:37.952822+0000","last_clean":"2024-04-25T16:31:37.952822+0000","last_became_active":"2024-04-25T16:30:09.999319+0000","last_became_peered":"2024-04-25T16:30:09.999319+0000","last_unstale":"2024-04-25T16:31:37.952822+0000","last_undegraded":"2024-04-25T16:31:37.952822+0000","last_fullsized":"2024-04-25T16:31:37.952822+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:14.043201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002182+0000","last_change":"2024-04-25T16:30:13.999609+0000","last_active":"2024-04-25T16:31:09.002182+0000","last_peered":"2024-04-25T16:31:09.002182+0000","last_clean":"2024-04-25T16:31:09.002182+0000","last_became_active":"2024-04-25T16:30:13.999172+0000","last_became_peered":"2024-04-25T16:30:13.999172+0000","last_unstale":"2024-04-25T16:31:09.002182+0000","last_undegraded":"2024-04-25T16:31:09.002182+0000","last_fullsized":"2024-04-25T16:31:09.002182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:49.878153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"83'7","reported_seq":107,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:46.161631+0000","last_change":"2024-04-25T16:30:08.988919+0000","last_active":"2024-04-25T16:31:46.161631+0000","last_peered":"2024-04-25T16:31:46.161631+0000","last_clean":"2024-04-25T16:31:46.161631+0000","last_became_active":"2024-04-25T16:30:08.988218+0000","last_became_peered":"2024-04-25T16:30:08.988218+0000","last_unstale":"2024-04-25T16:31:46.161631+0000","last_undegraded":"2024-04-25T16:31:46.161631+0000","last_fullsized":"2024-04-25T16:31:46.161631+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:57:57.046055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.949422+0000","last_change":"2024-04-25T16:30:09.992498+0000","last_active":"2024-04-25T16:31:37.949422+0000","last_peered":"2024-04-25T16:31:37.949422+0000","last_clean":"2024-04-25T16:31:37.949422+0000","last_became_active":"2024-04-25T16:30:09.992190+0000","last_became_peered":"2024-04-25T16:30:09.992190+0000","last_unstale":"2024-04-25T16:31:37.949422+0000","last_undegraded":"2024-04-25T16:31:37.949422+0000","last_fullsized":"2024-04-25T16:31:37.949422+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:34.407963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002209+0000","last_change":"2024-04-25T16:30:14.007371+0000","last_active":"2024-04-25T16:31:09.002209+0000","last_peered":"2024-04-25T16:31:09.002209+0000","last_clean":"2024-04-25T16:31:09.002209+0000","last_became_active":"2024-04-25T16:30:14.006811+0000","last_became_peered":"2024-04-25T16:30:14.006811+0000","last_unstale":"2024-04-25T16:31:09.002209+0000","last_undegraded":"2024-04-25T16:31:09.002209+0000","last_fullsized":"2024-04-25T16:31:09.002209+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:01.473142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006206+0000","last_change":"2024-04-25T16:30:08.984080+0000","last_active":"2024-04-25T16:31:09.006206+0000","last_peered":"2024-04-25T16:31:09.006206+0000","last_clean":"2024-04-25T16:31:09.006206+0000","last_became_active":"2024-04-25T16:30:08.983702+0000","last_became_peered":"2024-04-25T16:30:08.983702+0000","last_unstale":"2024-04-25T16:31:09.006206+0000","last_undegraded":"2024-04-25T16:31:09.006206+0000","last_fullsized":"2024-04-25T16:31:09.006206+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:22.311986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.953706+0000","last_change":"2024-04-25T16:31:09.187653+0000","last_active":"2024-04-25T16:31:37.953706+0000","last_peered":"2024-04-25T16:31:37.953706+0000","last_clean":"2024-04-25T16:31:37.953706+0000","last_became_active":"2024-04-25T16:31:09.187183+0000","last_became_peered":"2024-04-25T16:31:09.187183+0000","last_unstale":"2024-04-25T16:31:37.953706+0000","last_undegraded":"2024-04-25T16:31:37.953706+0000","last_fullsized":"2024-04-25T16:31:37.953706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:44:35.157126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.010419+0000","last_change":"2024-04-25T16:30:14.003218+0000","last_active":"2024-04-25T16:31:09.010419+0000","last_peered":"2024-04-25T16:31:09.010419+0000","last_clean":"2024-04-25T16:31:09.010419+0000","last_became_active":"2024-04-25T16:30:14.002782+0000","last_became_peered":"2024-04-25T16:30:14.002782+0000","last_unstale":"2024-04-25T16:31:09.010419+0000","last_undegraded":"2024-04-25T16:31:09.010419+0000","last_fullsized":"2024-04-25T16:31:09.010419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005109+0000","last_change":"2024-04-25T16:30:08.993447+0000","last_active":"2024-04-25T16:31:09.005109+0000","last_peered":"2024-04-25T16:31:09.005109+0000","last_clean":"2024-04-25T16:31:09.005109+0000","last_became_active":"2024-04-25T16:30:08.993108+0000","last_became_peered":"2024-04-25T16:30:08.993108+0000","last_unstale":"2024-04-25T16:31:09.005109+0000","last_undegraded":"2024-04-25T16:31:09.005109+0000","last_fullsized":"2024-04-25T16:31:09.005109+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:00.511392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.935592+0000","last_change":"2024-04-25T16:30:09.989863+0000","last_active":"2024-04-25T16:31:37.935592+0000","last_peered":"2024-04-25T16:31:37.935592+0000","last_clean":"2024-04-25T16:31:37.935592+0000","last_became_active":"2024-04-25T16:30:09.989520+0000","last_became_peered":"2024-04-25T16:30:09.989520+0000","last_unstale":"2024-04-25T16:31:37.935592+0000","last_undegraded":"2024-04-25T16:31:37.935592+0000","last_fullsized":"2024-04-25T16:31:37.935592+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:54.178298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.472395+0000","last_change":"2024-04-25T16:30:13.999904+0000","last_active":"2024-04-25T16:31:09.472395+0000","last_peered":"2024-04-25T16:31:09.472395+0000","last_clean":"2024-04-25T16:31:09.472395+0000","last_became_active":"2024-04-25T16:30:13.999468+0000","last_became_peered":"2024-04-25T16:30:13.999468+0000","last_unstale":"2024-04-25T16:31:09.472395+0000","last_undegraded":"2024-04-25T16:31:09.472395+0000","last_fullsized":"2024-04-25T16:31:09.472395+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:09.653365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005737+0000","last_change":"2024-04-25T16:30:08.987550+0000","last_active":"2024-04-25T16:31:09.005737+0000","last_peered":"2024-04-25T16:31:09.005737+0000","last_clean":"2024-04-25T16:31:09.005737+0000","last_became_active":"2024-04-25T16:30:08.987231+0000","last_became_peered":"2024-04-25T16:30:08.987231+0000","last_unstale":"2024-04-25T16:31:09.005737+0000","last_undegraded":"2024-04-25T16:31:09.005737+0000","last_fullsized":"2024-04-25T16:31:09.005737+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:01.110960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.942132+0000","last_change":"2024-04-25T16:30:09.991985+0000","last_active":"2024-04-25T16:31:37.942132+0000","last_peered":"2024-04-25T16:31:37.942132+0000","last_clean":"2024-04-25T16:31:37.942132+0000","last_became_active":"2024-04-25T16:30:09.991645+0000","last_became_peered":"2024-04-25T16:30:09.991645+0000","last_unstale":"2024-04-25T16:31:37.942132+0000","last_undegraded":"2024-04-25T16:31:37.942132+0000","last_fullsized":"2024-04-25T16:31:37.942132+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:30.419416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.185258+0000","last_change":"2024-04-25T16:30:13.999931+0000","last_active":"2024-04-25T16:31:09.185258+0000","last_peered":"2024-04-25T16:31:09.185258+0000","last_clean":"2024-04-25T16:31:09.185258+0000","last_became_active":"2024-04-25T16:30:13.997093+0000","last_became_peered":"2024-04-25T16:30:13.997093+0000","last_unstale":"2024-04-25T16:31:09.185258+0000","last_undegraded":"2024-04-25T16:31:09.185258+0000","last_fullsized":"2024-04-25T16:31:09.185258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:31.973370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"83'5","reported_seq":168,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:45.159660+0000","last_change":"2024-04-25T16:30:08.985721+0000","last_active":"2024-04-25T16:31:45.159660+0000","last_peered":"2024-04-25T16:31:45.159660+0000","last_clean":"2024-04-25T16:31:45.159660+0000","last_became_active":"2024-04-25T16:30:08.985097+0000","last_became_peered":"2024-04-25T16:30:08.985097+0000","last_unstale":"2024-04-25T16:31:45.159660+0000","last_undegraded":"2024-04-25T16:31:45.159660+0000","last_fullsized":"2024-04-25T16:31:45.159660+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:00.713027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":136,"num_read_kb":136,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.961844+0000","last_change":"2024-04-25T16:30:09.992659+0000","last_active":"2024-04-25T16:31:37.961844+0000","last_peered":"2024-04-25T16:31:37.961844+0000","last_clean":"2024-04-25T16:31:37.961844+0000","last_became_active":"2024-04-25T16:30:09.985936+0000","last_became_peered":"2024-04-25T16:30:09.985936+0000","last_unstale":"2024-04-25T16:31:37.961844+0000","last_undegraded":"2024-04-25T16:31:37.961844+0000","last_fullsized":"2024-04-25T16:31:37.961844+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:10.808082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001423+0000","last_change":"2024-04-25T16:30:14.009352+0000","last_active":"2024-04-25T16:31:09.001423+0000","last_peered":"2024-04-25T16:31:09.001423+0000","last_clean":"2024-04-25T16:31:09.001423+0000","last_became_active":"2024-04-25T16:30:14.005250+0000","last_became_peered":"2024-04-25T16:30:14.005250+0000","last_unstale":"2024-04-25T16:31:09.001423+0000","last_undegraded":"2024-04-25T16:31:09.001423+0000","last_fullsized":"2024-04-25T16:31:09.001423+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:23.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005543+0000","last_change":"2024-04-25T16:30:08.990547+0000","last_active":"2024-04-25T16:31:09.005543+0000","last_peered":"2024-04-25T16:31:09.005543+0000","last_clean":"2024-04-25T16:31:09.005543+0000","last_became_active":"2024-04-25T16:30:08.989927+0000","last_became_peered":"2024-04-25T16:30:08.989927+0000","last_unstale":"2024-04-25T16:31:09.005543+0000","last_undegraded":"2024-04-25T16:31:09.005543+0000","last_fullsized":"2024-04-25T16:31:09.005543+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:05.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.954774+0000","last_change":"2024-04-25T16:30:10.002459+0000","last_active":"2024-04-25T16:31:37.954774+0000","last_peered":"2024-04-25T16:31:37.954774+0000","last_clean":"2024-04-25T16:31:37.954774+0000","last_became_active":"2024-04-25T16:30:10.002005+0000","last_became_peered":"2024-04-25T16:30:10.002005+0000","last_unstale":"2024-04-25T16:31:37.954774+0000","last_undegraded":"2024-04-25T16:31:37.954774+0000","last_fullsized":"2024-04-25T16:31:37.954774+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:01.220371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.186538+0000","last_change":"2024-04-25T16:31:09.186538+0000","last_active":"2024-04-25T16:31:09.186538+0000","last_peered":"2024-04-25T16:31:09.186538+0000","last_clean":"2024-04-25T16:31:09.186538+0000","last_became_active":"2024-04-25T16:31:09.185930+0000","last_became_peered":"2024-04-25T16:31:09.185930+0000","last_unstale":"2024-04-25T16:31:09.186538+0000","last_undegraded":"2024-04-25T16:31:09.186538+0000","last_fullsized":"2024-04-25T16:31:09.186538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:14:38.930206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005922+0000","last_change":"2024-04-25T16:30:08.985264+0000","last_active":"2024-04-25T16:31:09.005922+0000","last_peered":"2024-04-25T16:31:09.005922+0000","last_clean":"2024-04-25T16:31:09.005922+0000","last_became_active":"2024-04-25T16:30:08.984901+0000","last_became_peered":"2024-04-25T16:30:08.984901+0000","last_unstale":"2024-04-25T16:31:09.005922+0000","last_undegraded":"2024-04-25T16:31:09.005922+0000","last_fullsized":"2024-04-25T16:31:09.005922+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:15.271981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963583+0000","last_change":"2024-04-25T16:30:09.997002+0000","last_active":"2024-04-25T16:31:37.963583+0000","last_peered":"2024-04-25T16:31:37.963583+0000","last_clean":"2024-04-25T16:31:37.963583+0000","last_became_active":"2024-04-25T16:30:09.996636+0000","last_became_peered":"2024-04-25T16:30:09.996636+0000","last_unstale":"2024-04-25T16:31:37.963583+0000","last_undegraded":"2024-04-25T16:31:37.963583+0000","last_fullsized":"2024-04-25T16:31:37.963583+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:32.522050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001480+0000","last_change":"2024-04-25T16:30:14.006403+0000","last_active":"2024-04-25T16:31:09.001480+0000","last_peered":"2024-04-25T16:31:09.001480+0000","last_clean":"2024-04-25T16:31:09.001480+0000","last_became_active":"2024-04-25T16:30:14.004816+0000","last_became_peered":"2024-04-25T16:30:14.004816+0000","last_unstale":"2024-04-25T16:31:09.001480+0000","last_undegraded":"2024-04-25T16:31:09.001480+0000","last_fullsized":"2024-04-25T16:31:09.001480+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:51:10.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"83'38","reported_seq":208,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:45.173672+0000","last_change":"2024-04-25T16:30:08.986119+0000","last_active":"2024-04-25T16:31:45.173672+0000","last_peered":"2024-04-25T16:31:45.173672+0000","last_clean":"2024-04-25T16:31:45.173672+0000","last_became_active":"2024-04-25T16:30:08.985806+0000","last_became_peered":"2024-04-25T16:30:08.985806+0000","last_unstale":"2024-04-25T16:31:45.173672+0000","last_undegraded":"2024-04-25T16:31:45.173672+0000","last_fullsized":"2024-04-25T16:31:45.173672+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:17.125886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":134,"num_write":40,"num_write_kb":80,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951203+0000","last_change":"2024-04-25T16:30:09.996891+0000","last_active":"2024-04-25T16:31:37.951203+0000","last_peered":"2024-04-25T16:31:37.951203+0000","last_clean":"2024-04-25T16:31:37.951203+0000","last_became_active":"2024-04-25T16:30:09.995851+0000","last_became_peered":"2024-04-25T16:30:09.995851+0000","last_unstale":"2024-04-25T16:31:37.951203+0000","last_undegraded":"2024-04-25T16:31:37.951203+0000","last_fullsized":"2024-04-25T16:31:37.951203+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:42.602854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006493+0000","last_change":"2024-04-25T16:30:13.998277+0000","last_active":"2024-04-25T16:31:09.006493+0000","last_peered":"2024-04-25T16:31:09.006493+0000","last_clean":"2024-04-25T16:31:09.006493+0000","last_became_active":"2024-04-25T16:30:13.997313+0000","last_became_peered":"2024-04-25T16:30:13.997313+0000","last_unstale":"2024-04-25T16:31:09.006493+0000","last_undegraded":"2024-04-25T16:31:09.006493+0000","last_fullsized":"2024-04-25T16:31:09.006493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:56.379806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002447+0000","last_change":"2024-04-25T16:30:08.986377+0000","last_active":"2024-04-25T16:31:09.002447+0000","last_peered":"2024-04-25T16:31:09.002447+0000","last_clean":"2024-04-25T16:31:09.002447+0000","last_became_active":"2024-04-25T16:30:08.985892+0000","last_became_peered":"2024-04-25T16:30:08.985892+0000","last_unstale":"2024-04-25T16:31:09.002447+0000","last_undegraded":"2024-04-25T16:31:09.002447+0000","last_fullsized":"2024-04-25T16:31:09.002447+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:53.651789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951284+0000","last_change":"2024-04-25T16:31:09.010551+0000","last_active":"2024-04-25T16:31:37.951284+0000","last_peered":"2024-04-25T16:31:37.951284+0000","last_clean":"2024-04-25T16:31:37.951284+0000","last_became_active":"2024-04-25T16:31:09.010141+0000","last_became_peered":"2024-04-25T16:31:09.010141+0000","last_unstale":"2024-04-25T16:31:37.951284+0000","last_undegraded":"2024-04-25T16:31:37.951284+0000","last_fullsized":"2024-04-25T16:31:37.951284+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:54.650080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.010236+0000","last_change":"2024-04-25T16:30:14.006064+0000","last_active":"2024-04-25T16:31:09.010236+0000","last_peered":"2024-04-25T16:31:09.010236+0000","last_clean":"2024-04-25T16:31:09.010236+0000","last_became_active":"2024-04-25T16:30:14.005545+0000","last_became_peered":"2024-04-25T16:30:14.005545+0000","last_unstale":"2024-04-25T16:31:09.010236+0000","last_undegraded":"2024-04-25T16:31:09.010236+0000","last_fullsized":"2024-04-25T16:31:09.010236+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:36.611008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006844+0000","last_change":"2024-04-25T16:30:08.988413+0000","last_active":"2024-04-25T16:31:09.006844+0000","last_peered":"2024-04-25T16:31:09.006844+0000","last_clean":"2024-04-25T16:31:09.006844+0000","last_became_active":"2024-04-25T16:30:08.988075+0000","last_became_peered":"2024-04-25T16:30:08.988075+0000","last_unstale":"2024-04-25T16:31:09.006844+0000","last_undegraded":"2024-04-25T16:31:09.006844+0000","last_fullsized":"2024-04-25T16:31:09.006844+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:16.635876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.961181+0000","last_change":"2024-04-25T16:30:09.997697+0000","last_active":"2024-04-25T16:31:37.961181+0000","last_peered":"2024-04-25T16:31:37.961181+0000","last_clean":"2024-04-25T16:31:37.961181+0000","last_became_active":"2024-04-25T16:30:09.997086+0000","last_became_peered":"2024-04-25T16:30:09.997086+0000","last_unstale":"2024-04-25T16:31:37.961181+0000","last_undegraded":"2024-04-25T16:31:37.961181+0000","last_fullsized":"2024-04-25T16:31:37.961181+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:35.345902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002285+0000","last_change":"2024-04-25T16:30:14.010486+0000","last_active":"2024-04-25T16:31:09.002285+0000","last_peered":"2024-04-25T16:31:09.002285+0000","last_clean":"2024-04-25T16:31:09.002285+0000","last_became_active":"2024-04-25T16:30:14.009273+0000","last_became_peered":"2024-04-25T16:30:14.009273+0000","last_unstale":"2024-04-25T16:31:09.002285+0000","last_undegraded":"2024-04-25T16:31:09.002285+0000","last_fullsized":"2024-04-25T16:31:09.002285+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:11.863575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"83'7","reported_seq":107,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:46.174301+0000","last_change":"2024-04-25T16:30:08.986661+0000","last_active":"2024-04-25T16:31:46.174301+0000","last_peered":"2024-04-25T16:31:46.174301+0000","last_clean":"2024-04-25T16:31:46.174301+0000","last_became_active":"2024-04-25T16:30:08.986306+0000","last_became_peered":"2024-04-25T16:30:08.986306+0000","last_unstale":"2024-04-25T16:31:46.174301+0000","last_undegraded":"2024-04-25T16:31:46.174301+0000","last_fullsized":"2024-04-25T16:31:46.174301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:44.363693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.954252+0000","last_change":"2024-04-25T16:30:09.993443+0000","last_active":"2024-04-25T16:31:37.954252+0000","last_peered":"2024-04-25T16:31:37.954252+0000","last_clean":"2024-04-25T16:31:37.954252+0000","last_became_active":"2024-04-25T16:30:09.993031+0000","last_became_peered":"2024-04-25T16:30:09.993031+0000","last_unstale":"2024-04-25T16:31:37.954252+0000","last_undegraded":"2024-04-25T16:31:37.954252+0000","last_fullsized":"2024-04-25T16:31:37.954252+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:22.908713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005959+0000","last_change":"2024-04-25T16:30:14.003183+0000","last_active":"2024-04-25T16:31:09.005959+0000","last_peered":"2024-04-25T16:31:09.005959+0000","last_clean":"2024-04-25T16:31:09.005959+0000","last_became_active":"2024-04-25T16:30:14.002182+0000","last_became_peered":"2024-04-25T16:30:14.002182+0000","last_unstale":"2024-04-25T16:31:09.005959+0000","last_undegraded":"2024-04-25T16:31:09.005959+0000","last_fullsized":"2024-04-25T16:31:09.005959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:08.616556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006777+0000","last_change":"2024-04-25T16:30:08.995860+0000","last_active":"2024-04-25T16:31:09.006777+0000","last_peered":"2024-04-25T16:31:09.006777+0000","last_clean":"2024-04-25T16:31:09.006777+0000","last_became_active":"2024-04-25T16:30:08.995487+0000","last_became_peered":"2024-04-25T16:30:08.995487+0000","last_unstale":"2024-04-25T16:31:09.006777+0000","last_undegraded":"2024-04-25T16:31:09.006777+0000","last_fullsized":"2024-04-25T16:31:09.006777+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:48.206502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.947991+0000","last_change":"2024-04-25T16:30:09.998820+0000","last_active":"2024-04-25T16:31:37.947991+0000","last_peered":"2024-04-25T16:31:37.947991+0000","last_clean":"2024-04-25T16:31:37.947991+0000","last_became_active":"2024-04-25T16:30:09.998448+0000","last_became_peered":"2024-04-25T16:30:09.998448+0000","last_unstale":"2024-04-25T16:31:37.947991+0000","last_undegraded":"2024-04-25T16:31:37.947991+0000","last_fullsized":"2024-04-25T16:31:37.947991+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:54.361250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005702+0000","last_change":"2024-04-25T16:30:13.992379+0000","last_active":"2024-04-25T16:31:09.005702+0000","last_peered":"2024-04-25T16:31:09.005702+0000","last_clean":"2024-04-25T16:31:09.005702+0000","last_became_active":"2024-04-25T16:30:13.991940+0000","last_became_peered":"2024-04-25T16:30:13.991940+0000","last_unstale":"2024-04-25T16:31:09.005702+0000","last_undegraded":"2024-04-25T16:31:09.005702+0000","last_fullsized":"2024-04-25T16:31:09.005702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:59.152384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.008399+0000","last_change":"2024-04-25T16:30:08.987701+0000","last_active":"2024-04-25T16:31:09.008399+0000","last_peered":"2024-04-25T16:31:09.008399+0000","last_clean":"2024-04-25T16:31:09.008399+0000","last_became_active":"2024-04-25T16:30:08.987376+0000","last_became_peered":"2024-04-25T16:30:08.987376+0000","last_unstale":"2024-04-25T16:31:09.008399+0000","last_undegraded":"2024-04-25T16:31:09.008399+0000","last_fullsized":"2024-04-25T16:31:09.008399+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:21.810100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.945676+0000","last_change":"2024-04-25T16:30:09.993005+0000","last_active":"2024-04-25T16:31:37.945676+0000","last_peered":"2024-04-25T16:31:37.945676+0000","last_clean":"2024-04-25T16:31:37.945676+0000","last_became_active":"2024-04-25T16:30:09.992619+0000","last_became_peered":"2024-04-25T16:30:09.992619+0000","last_unstale":"2024-04-25T16:31:37.945676+0000","last_undegraded":"2024-04-25T16:31:37.945676+0000","last_fullsized":"2024-04-25T16:31:37.945676+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:39:40.998241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006207+0000","last_change":"2024-04-25T16:30:14.147895+0000","last_active":"2024-04-25T16:31:09.006207+0000","last_peered":"2024-04-25T16:31:09.006207+0000","last_clean":"2024-04-25T16:31:09.006207+0000","last_became_active":"2024-04-25T16:30:14.147298+0000","last_became_peered":"2024-04-25T16:30:14.147298+0000","last_unstale":"2024-04-25T16:31:09.006207+0000","last_undegraded":"2024-04-25T16:31:09.006207+0000","last_fullsized":"2024-04-25T16:31:09.006207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:06.269587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":28,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001713+0000","last_change":"2024-04-25T16:30:08.982683+0000","last_active":"2024-04-25T16:31:09.001713+0000","last_peered":"2024-04-25T16:31:09.001713+0000","last_clean":"2024-04-25T16:31:09.001713+0000","last_became_active":"2024-04-25T16:30:08.982236+0000","last_became_peered":"2024-04-25T16:30:08.982236+0000","last_unstale":"2024-04-25T16:31:09.001713+0000","last_undegraded":"2024-04-25T16:31:09.001713+0000","last_fullsized":"2024-04-25T16:31:09.001713+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:48.781791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.939293+0000","last_change":"2024-04-25T16:30:09.986734+0000","last_active":"2024-04-25T16:31:37.939293+0000","last_peered":"2024-04-25T16:31:37.939293+0000","last_clean":"2024-04-25T16:31:37.939293+0000","last_became_active":"2024-04-25T16:30:09.986337+0000","last_became_peered":"2024-04-25T16:30:09.986337+0000","last_unstale":"2024-04-25T16:31:37.939293+0000","last_undegraded":"2024-04-25T16:31:37.939293+0000","last_fullsized":"2024-04-25T16:31:37.939293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.678009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005340+0000","last_change":"2024-04-25T16:30:14.003324+0000","last_active":"2024-04-25T16:31:09.005340+0000","last_peered":"2024-04-25T16:31:09.005340+0000","last_clean":"2024-04-25T16:31:09.005340+0000","last_became_active":"2024-04-25T16:30:14.002904+0000","last_became_peered":"2024-04-25T16:30:14.002904+0000","last_unstale":"2024-04-25T16:31:09.005340+0000","last_undegraded":"2024-04-25T16:31:09.005340+0000","last_fullsized":"2024-04-25T16:31:09.005340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:28.750102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":103,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:46.169598+0000","last_change":"2024-04-25T16:30:08.993469+0000","last_active":"2024-04-25T16:31:46.169598+0000","last_peered":"2024-04-25T16:31:46.169598+0000","last_clean":"2024-04-25T16:31:46.169598+0000","last_became_active":"2024-04-25T16:30:08.993126+0000","last_became_peered":"2024-04-25T16:30:08.993126+0000","last_unstale":"2024-04-25T16:31:46.169598+0000","last_undegraded":"2024-04-25T16:31:46.169598+0000","last_fullsized":"2024-04-25T16:31:46.169598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:10.274645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.945673+0000","last_change":"2024-04-25T16:30:10.000597+0000","last_active":"2024-04-25T16:31:37.945673+0000","last_peered":"2024-04-25T16:31:37.945673+0000","last_clean":"2024-04-25T16:31:37.945673+0000","last_became_active":"2024-04-25T16:30:09.998892+0000","last_became_peered":"2024-04-25T16:30:09.998892+0000","last_unstale":"2024-04-25T16:31:37.945673+0000","last_undegraded":"2024-04-25T16:31:37.945673+0000","last_fullsized":"2024-04-25T16:31:37.945673+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:09.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005789+0000","last_change":"2024-04-25T16:30:14.147240+0000","last_active":"2024-04-25T16:31:09.005789+0000","last_peered":"2024-04-25T16:31:09.005789+0000","last_clean":"2024-04-25T16:31:09.005789+0000","last_became_active":"2024-04-25T16:30:14.146793+0000","last_became_peered":"2024-04-25T16:30:14.146793+0000","last_unstale":"2024-04-25T16:31:09.005789+0000","last_undegraded":"2024-04-25T16:31:09.005789+0000","last_fullsized":"2024-04-25T16:31:09.005789+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:55.713386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944046+0000","last_change":"2024-04-25T16:30:09.994595+0000","last_active":"2024-04-25T16:31:37.944046+0000","last_peered":"2024-04-25T16:31:37.944046+0000","last_clean":"2024-04-25T16:31:37.944046+0000","last_became_active":"2024-04-25T16:30:09.994305+0000","last_became_peered":"2024-04-25T16:30:09.994305+0000","last_unstale":"2024-04-25T16:31:37.944046+0000","last_undegraded":"2024-04-25T16:31:37.944046+0000","last_fullsized":"2024-04-25T16:31:37.944046+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:27.839500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"83'6","reported_seq":105,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:47.175193+0000","last_change":"2024-04-25T16:30:08.984203+0000","last_active":"2024-04-25T16:31:47.175193+0000","last_peered":"2024-04-25T16:31:47.175193+0000","last_clean":"2024-04-25T16:31:47.175193+0000","last_became_active":"2024-04-25T16:30:08.982883+0000","last_became_peered":"2024-04-25T16:30:08.982883+0000","last_unstale":"2024-04-25T16:31:47.175193+0000","last_undegraded":"2024-04-25T16:31:47.175193+0000","last_fullsized":"2024-04-25T16:31:47.175193+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:18.447624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.017505+0000","last_change":"2024-04-25T16:31:09.017505+0000","last_active":"2024-04-25T16:31:09.017505+0000","last_peered":"2024-04-25T16:31:09.017505+0000","last_clean":"2024-04-25T16:31:09.017505+0000","last_became_active":"2024-04-25T16:31:09.011812+0000","last_became_peered":"2024-04-25T16:31:09.011812+0000","last_unstale":"2024-04-25T16:31:09.017505+0000","last_undegraded":"2024-04-25T16:31:09.017505+0000","last_fullsized":"2024-04-25T16:31:09.017505+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:02.404908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.948845+0000","last_change":"2024-04-25T16:30:09.990998+0000","last_active":"2024-04-25T16:31:37.948845+0000","last_peered":"2024-04-25T16:31:37.948845+0000","last_clean":"2024-04-25T16:31:37.948845+0000","last_became_active":"2024-04-25T16:30:09.990644+0000","last_became_peered":"2024-04-25T16:30:09.990644+0000","last_unstale":"2024-04-25T16:31:37.948845+0000","last_undegraded":"2024-04-25T16:31:37.948845+0000","last_fullsized":"2024-04-25T16:31:37.948845+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:42.175520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002912+0000","last_change":"2024-04-25T16:30:08.988315+0000","last_active":"2024-04-25T16:31:09.002912+0000","last_peered":"2024-04-25T16:31:09.002912+0000","last_clean":"2024-04-25T16:31:09.002912+0000","last_became_active":"2024-04-25T16:30:08.987930+0000","last_became_peered":"2024-04-25T16:30:08.987930+0000","last_unstale":"2024-04-25T16:31:09.002912+0000","last_undegraded":"2024-04-25T16:31:09.002912+0000","last_fullsized":"2024-04-25T16:31:09.002912+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:44.884602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":102,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005926+0000","last_change":"2024-04-25T16:28:58.219510+0000","last_active":"2024-04-25T16:31:09.005926+0000","last_peered":"2024-04-25T16:31:09.005926+0000","last_clean":"2024-04-25T16:31:09.005926+0000","last_became_active":"2024-04-25T16:28:58.213888+0000","last_became_peered":"2024-04-25T16:28:58.213888+0000","last_unstale":"2024-04-25T16:31:09.005926+0000","last_undegraded":"2024-04-25T16:31:09.005926+0000","last_fullsized":"2024-04-25T16:31:09.005926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_clean_scrub_stamp":"2024-04-25T16:25:33.939186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:15.378406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.012831+0000","last_change":"2024-04-25T16:31:09.012831+0000","last_active":"2024-04-25T16:31:09.012831+0000","last_peered":"2024-04-25T16:31:09.012831+0000","last_clean":"2024-04-25T16:31:09.012831+0000","last_became_active":"2024-04-25T16:31:09.012375+0000","last_became_peered":"2024-04-25T16:31:09.012375+0000","last_unstale":"2024-04-25T16:31:09.012831+0000","last_undegraded":"2024-04-25T16:31:09.012831+0000","last_fullsized":"2024-04-25T16:31:09.012831+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:02.214734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944037+0000","last_change":"2024-04-25T16:30:10.005210+0000","last_active":"2024-04-25T16:31:37.944037+0000","last_peered":"2024-04-25T16:31:37.944037+0000","last_clean":"2024-04-25T16:31:37.944037+0000","last_became_active":"2024-04-25T16:30:10.004936+0000","last_became_peered":"2024-04-25T16:30:10.004936+0000","last_unstale":"2024-04-25T16:31:37.944037+0000","last_undegraded":"2024-04-25T16:31:37.944037+0000","last_fullsized":"2024-04-25T16:31:37.944037+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:53.292675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005622+0000","last_change":"2024-04-25T16:30:08.990162+0000","last_active":"2024-04-25T16:31:09.005622+0000","last_peered":"2024-04-25T16:31:09.005622+0000","last_clean":"2024-04-25T16:31:09.005622+0000","last_became_active":"2024-04-25T16:30:08.989613+0000","last_became_peered":"2024-04-25T16:30:08.989613+0000","last_unstale":"2024-04-25T16:31:09.005622+0000","last_undegraded":"2024-04-25T16:31:09.005622+0000","last_fullsized":"2024-04-25T16:31:09.005622+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:41.595381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.010109+0000","last_change":"2024-04-25T16:30:14.001850+0000","last_active":"2024-04-25T16:31:09.010109+0000","last_peered":"2024-04-25T16:31:09.010109+0000","last_clean":"2024-04-25T16:31:09.010109+0000","last_became_active":"2024-04-25T16:30:14.001348+0000","last_became_peered":"2024-04-25T16:30:14.001348+0000","last_unstale":"2024-04-25T16:31:09.010109+0000","last_undegraded":"2024-04-25T16:31:09.010109+0000","last_fullsized":"2024-04-25T16:31:09.010109+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:56:41.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.007060+0000","last_change":"2024-04-25T16:30:08.995833+0000","last_active":"2024-04-25T16:31:09.007060+0000","last_peered":"2024-04-25T16:31:09.007060+0000","last_clean":"2024-04-25T16:31:09.007060+0000","last_became_active":"2024-04-25T16:30:08.995545+0000","last_became_peered":"2024-04-25T16:30:08.995545+0000","last_unstale":"2024-04-25T16:31:09.007060+0000","last_undegraded":"2024-04-25T16:31:09.007060+0000","last_fullsized":"2024-04-25T16:31:09.007060+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:31.452219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.936626+0000","last_change":"2024-04-25T16:30:09.990225+0000","last_active":"2024-04-25T16:31:37.936626+0000","last_peered":"2024-04-25T16:31:37.936626+0000","last_clean":"2024-04-25T16:31:37.936626+0000","last_became_active":"2024-04-25T16:30:09.989859+0000","last_became_peered":"2024-04-25T16:30:09.989859+0000","last_unstale":"2024-04-25T16:31:37.936626+0000","last_undegraded":"2024-04-25T16:31:37.936626+0000","last_fullsized":"2024-04-25T16:31:37.936626+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:40.589395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.185069+0000","last_change":"2024-04-25T16:30:14.147223+0000","last_active":"2024-04-25T16:31:09.185069+0000","last_peered":"2024-04-25T16:31:09.185069+0000","last_clean":"2024-04-25T16:31:09.185069+0000","last_became_active":"2024-04-25T16:30:14.146821+0000","last_became_peered":"2024-04-25T16:30:14.146821+0000","last_unstale":"2024-04-25T16:31:09.185069+0000","last_undegraded":"2024-04-25T16:31:09.185069+0000","last_fullsized":"2024-04-25T16:31:09.185069+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:01.289165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'13","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002175+0000","last_change":"2024-04-25T16:30:08.993175+0000","last_active":"2024-04-25T16:31:09.002175+0000","last_peered":"2024-04-25T16:31:09.002175+0000","last_clean":"2024-04-25T16:31:09.002175+0000","last_became_active":"2024-04-25T16:30:08.992145+0000","last_became_peered":"2024-04-25T16:30:08.992145+0000","last_unstale":"2024-04-25T16:31:09.002175+0000","last_undegraded":"2024-04-25T16:31:09.002175+0000","last_fullsized":"2024-04-25T16:31:09.002175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:21:06.951435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.947300+0000","last_change":"2024-04-25T16:30:09.986088+0000","last_active":"2024-04-25T16:31:37.947300+0000","last_peered":"2024-04-25T16:31:37.947300+0000","last_clean":"2024-04-25T16:31:37.947300+0000","last_became_active":"2024-04-25T16:30:09.985256+0000","last_became_peered":"2024-04-25T16:30:09.985256+0000","last_unstale":"2024-04-25T16:31:37.947300+0000","last_undegraded":"2024-04-25T16:31:37.947300+0000","last_fullsized":"2024-04-25T16:31:37.947300+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:29.197558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001709+0000","last_change":"2024-04-25T16:30:14.000699+0000","last_active":"2024-04-25T16:31:09.001709+0000","last_peered":"2024-04-25T16:31:09.001709+0000","last_clean":"2024-04-25T16:31:09.001709+0000","last_became_active":"2024-04-25T16:30:14.000255+0000","last_became_peered":"2024-04-25T16:30:14.000255+0000","last_unstale":"2024-04-25T16:31:09.001709+0000","last_undegraded":"2024-04-25T16:31:09.001709+0000","last_fullsized":"2024-04-25T16:31:09.001709+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:33.436693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.003974+0000","last_change":"2024-04-25T16:30:08.981926+0000","last_active":"2024-04-25T16:31:09.003974+0000","last_peered":"2024-04-25T16:31:09.003974+0000","last_clean":"2024-04-25T16:31:09.003974+0000","last_became_active":"2024-04-25T16:30:08.981590+0000","last_became_peered":"2024-04-25T16:30:08.981590+0000","last_unstale":"2024-04-25T16:31:09.003974+0000","last_undegraded":"2024-04-25T16:31:09.003974+0000","last_fullsized":"2024-04-25T16:31:09.003974+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:57.818774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951291+0000","last_change":"2024-04-25T16:30:10.003998+0000","last_active":"2024-04-25T16:31:37.951291+0000","last_peered":"2024-04-25T16:31:37.951291+0000","last_clean":"2024-04-25T16:31:37.951291+0000","last_became_active":"2024-04-25T16:30:10.003626+0000","last_became_peered":"2024-04-25T16:30:10.003626+0000","last_unstale":"2024-04-25T16:31:37.951291+0000","last_undegraded":"2024-04-25T16:31:37.951291+0000","last_fullsized":"2024-04-25T16:31:37.951291+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:10.686092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002820+0000","last_change":"2024-04-25T16:30:14.149278+0000","last_active":"2024-04-25T16:31:09.002820+0000","last_peered":"2024-04-25T16:31:09.002820+0000","last_clean":"2024-04-25T16:31:09.002820+0000","last_became_active":"2024-04-25T16:30:14.148654+0000","last_became_peered":"2024-04-25T16:30:14.148654+0000","last_unstale":"2024-04-25T16:31:09.002820+0000","last_undegraded":"2024-04-25T16:31:09.002820+0000","last_fullsized":"2024-04-25T16:31:09.002820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:24.823222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.184459+0000","last_change":"2024-04-25T16:30:08.981612+0000","last_active":"2024-04-25T16:31:09.184459+0000","last_peered":"2024-04-25T16:31:09.184459+0000","last_clean":"2024-04-25T16:31:09.184459+0000","last_became_active":"2024-04-25T16:30:08.981222+0000","last_became_peered":"2024-04-25T16:30:08.981222+0000","last_unstale":"2024-04-25T16:31:09.184459+0000","last_undegraded":"2024-04-25T16:31:09.184459+0000","last_fullsized":"2024-04-25T16:31:09.184459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:14.914658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.907101+0000","last_change":"2024-04-25T16:30:09.989592+0000","last_active":"2024-04-25T16:31:37.907101+0000","last_peered":"2024-04-25T16:31:37.907101+0000","last_clean":"2024-04-25T16:31:37.907101+0000","last_became_active":"2024-04-25T16:30:09.989222+0000","last_became_peered":"2024-04-25T16:30:09.989222+0000","last_unstale":"2024-04-25T16:31:37.907101+0000","last_undegraded":"2024-04-25T16:31:37.907101+0000","last_fullsized":"2024-04-25T16:31:37.907101+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:52.403239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002346+0000","last_change":"2024-04-25T16:30:14.010480+0000","last_active":"2024-04-25T16:31:09.002346+0000","last_peered":"2024-04-25T16:31:09.002346+0000","last_clean":"2024-04-25T16:31:09.002346+0000","last_became_active":"2024-04-25T16:30:14.009023+0000","last_became_peered":"2024-04-25T16:30:14.009023+0000","last_unstale":"2024-04-25T16:31:09.002346+0000","last_undegraded":"2024-04-25T16:31:09.002346+0000","last_fullsized":"2024-04-25T16:31:09.002346+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:10.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.185374+0000","last_change":"2024-04-25T16:30:08.986849+0000","last_active":"2024-04-25T16:31:09.185374+0000","last_peered":"2024-04-25T16:31:09.185374+0000","last_clean":"2024-04-25T16:31:09.185374+0000","last_became_active":"2024-04-25T16:30:08.986479+0000","last_became_peered":"2024-04-25T16:30:08.986479+0000","last_unstale":"2024-04-25T16:31:09.185374+0000","last_undegraded":"2024-04-25T16:31:09.185374+0000","last_fullsized":"2024-04-25T16:31:09.185374+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.950372+0000","last_change":"2024-04-25T16:30:09.990927+0000","last_active":"2024-04-25T16:31:37.950372+0000","last_peered":"2024-04-25T16:31:37.950372+0000","last_clean":"2024-04-25T16:31:37.950372+0000","last_became_active":"2024-04-25T16:30:09.990482+0000","last_became_peered":"2024-04-25T16:30:09.990482+0000","last_unstale":"2024-04-25T16:31:37.950372+0000","last_undegraded":"2024-04-25T16:31:37.950372+0000","last_fullsized":"2024-04-25T16:31:37.950372+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:18.649721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.003105+0000","last_change":"2024-04-25T16:30:14.006159+0000","last_active":"2024-04-25T16:31:09.003105+0000","last_peered":"2024-04-25T16:31:09.003105+0000","last_clean":"2024-04-25T16:31:09.003105+0000","last_became_active":"2024-04-25T16:30:14.005413+0000","last_became_peered":"2024-04-25T16:30:14.005413+0000","last_unstale":"2024-04-25T16:31:09.003105+0000","last_undegraded":"2024-04-25T16:31:09.003105+0000","last_fullsized":"2024-04-25T16:31:09.003105+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:52.934385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.001913+0000","last_change":"2024-04-25T16:30:08.986207+0000","last_active":"2024-04-25T16:31:09.001913+0000","last_peered":"2024-04-25T16:31:09.001913+0000","last_clean":"2024-04-25T16:31:09.001913+0000","last_became_active":"2024-04-25T16:30:08.985857+0000","last_became_peered":"2024-04-25T16:30:08.985857+0000","last_unstale":"2024-04-25T16:31:09.001913+0000","last_undegraded":"2024-04-25T16:31:09.001913+0000","last_fullsized":"2024-04-25T16:31:09.001913+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:45.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.939780+0000","last_change":"2024-04-25T16:30:09.993566+0000","last_active":"2024-04-25T16:31:37.939780+0000","last_peered":"2024-04-25T16:31:37.939780+0000","last_clean":"2024-04-25T16:31:37.939780+0000","last_became_active":"2024-04-25T16:30:09.992250+0000","last_became_peered":"2024-04-25T16:30:09.992250+0000","last_unstale":"2024-04-25T16:31:37.939780+0000","last_undegraded":"2024-04-25T16:31:37.939780+0000","last_fullsized":"2024-04-25T16:31:37.939780+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:33.279300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.002881+0000","last_change":"2024-04-25T16:30:14.147443+0000","last_active":"2024-04-25T16:31:09.002881+0000","last_peered":"2024-04-25T16:31:09.002881+0000","last_clean":"2024-04-25T16:31:09.002881+0000","last_became_active":"2024-04-25T16:30:14.146838+0000","last_became_peered":"2024-04-25T16:30:14.146838+0000","last_unstale":"2024-04-25T16:31:09.002881+0000","last_undegraded":"2024-04-25T16:31:09.002881+0000","last_fullsized":"2024-04-25T16:31:09.002881+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:51.532685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.015526+0000","last_change":"2024-04-25T16:31:09.015526+0000","last_active":"2024-04-25T16:31:09.015526+0000","last_peered":"2024-04-25T16:31:09.015526+0000","last_clean":"2024-04-25T16:31:09.015526+0000","last_became_active":"2024-04-25T16:31:09.007972+0000","last_became_peered":"2024-04-25T16:31:09.007972+0000","last_unstale":"2024-04-25T16:31:09.015526+0000","last_undegraded":"2024-04-25T16:31:09.015526+0000","last_fullsized":"2024-04-25T16:31:09.015526+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:56.464426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963806+0000","last_change":"2024-04-25T16:30:09.998486+0000","last_active":"2024-04-25T16:31:37.963806+0000","last_peered":"2024-04-25T16:31:37.963806+0000","last_clean":"2024-04-25T16:31:37.963806+0000","last_became_active":"2024-04-25T16:30:09.998154+0000","last_became_peered":"2024-04-25T16:30:09.998154+0000","last_unstale":"2024-04-25T16:31:37.963806+0000","last_undegraded":"2024-04-25T16:31:37.963806+0000","last_fullsized":"2024-04-25T16:31:37.963806+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:55.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.184576+0000","last_change":"2024-04-25T16:30:13.995864+0000","last_active":"2024-04-25T16:31:09.184576+0000","last_peered":"2024-04-25T16:31:09.184576+0000","last_clean":"2024-04-25T16:31:09.184576+0000","last_became_active":"2024-04-25T16:30:13.995326+0000","last_became_peered":"2024-04-25T16:30:13.995326+0000","last_unstale":"2024-04-25T16:31:09.184576+0000","last_undegraded":"2024-04-25T16:31:09.184576+0000","last_fullsized":"2024-04-25T16:31:09.184576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:11.718826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":104,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:46.191993+0000","last_change":"2024-04-25T16:30:08.984189+0000","last_active":"2024-04-25T16:31:46.191993+0000","last_peered":"2024-04-25T16:31:46.191993+0000","last_clean":"2024-04-25T16:31:46.191993+0000","last_became_active":"2024-04-25T16:30:08.983821+0000","last_became_peered":"2024-04-25T16:30:08.983821+0000","last_unstale":"2024-04-25T16:31:46.191993+0000","last_undegraded":"2024-04-25T16:31:46.191993+0000","last_fullsized":"2024-04-25T16:31:46.191993+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:51.315397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.963625+0000","last_change":"2024-04-25T16:30:09.998763+0000","last_active":"2024-04-25T16:31:37.963625+0000","last_peered":"2024-04-25T16:31:37.963625+0000","last_clean":"2024-04-25T16:31:37.963625+0000","last_became_active":"2024-04-25T16:30:09.998439+0000","last_became_peered":"2024-04-25T16:30:09.998439+0000","last_unstale":"2024-04-25T16:31:37.963625+0000","last_undegraded":"2024-04-25T16:31:37.963625+0000","last_fullsized":"2024-04-25T16:31:37.963625+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:14.279883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.472083+0000","last_change":"2024-04-25T16:30:14.005427+0000","last_active":"2024-04-25T16:31:09.472083+0000","last_peered":"2024-04-25T16:31:09.472083+0000","last_clean":"2024-04-25T16:31:09.472083+0000","last_became_active":"2024-04-25T16:30:14.005006+0000","last_became_peered":"2024-04-25T16:30:14.005006+0000","last_unstale":"2024-04-25T16:31:09.472083+0000","last_undegraded":"2024-04-25T16:31:09.472083+0000","last_fullsized":"2024-04-25T16:31:09.472083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:49.563365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.006221+0000","last_change":"2024-04-25T16:30:08.985557+0000","last_active":"2024-04-25T16:31:09.006221+0000","last_peered":"2024-04-25T16:31:09.006221+0000","last_clean":"2024-04-25T16:31:09.006221+0000","last_became_active":"2024-04-25T16:30:08.985215+0000","last_became_peered":"2024-04-25T16:30:08.985215+0000","last_unstale":"2024-04-25T16:31:09.006221+0000","last_undegraded":"2024-04-25T16:31:09.006221+0000","last_fullsized":"2024-04-25T16:31:09.006221+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:51.504089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.947056+0000","last_change":"2024-04-25T16:30:09.991336+0000","last_active":"2024-04-25T16:31:37.947056+0000","last_peered":"2024-04-25T16:31:37.947056+0000","last_clean":"2024-04-25T16:31:37.947056+0000","last_became_active":"2024-04-25T16:30:09.990819+0000","last_became_peered":"2024-04-25T16:30:09.990819+0000","last_unstale":"2024-04-25T16:31:37.947056+0000","last_undegraded":"2024-04-25T16:31:37.947056+0000","last_fullsized":"2024-04-25T16:31:37.947056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:45.988061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.004993+0000","last_change":"2024-04-25T16:30:13.998463+0000","last_active":"2024-04-25T16:31:09.004993+0000","last_peered":"2024-04-25T16:31:09.004993+0000","last_clean":"2024-04-25T16:31:09.004993+0000","last_became_active":"2024-04-25T16:30:13.997952+0000","last_became_peered":"2024-04-25T16:30:13.997952+0000","last_unstale":"2024-04-25T16:31:09.004993+0000","last_undegraded":"2024-04-25T16:31:09.004993+0000","last_fullsized":"2024-04-25T16:31:09.004993+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:58.507942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:09.005908+0000","last_change":"2024-04-25T16:30:08.987232+0000","last_active":"2024-04-25T16:31:09.005908+0000","last_peered":"2024-04-25T16:31:09.005908+0000","last_clean":"2024-04-25T16:31:09.005908+0000","last_became_active":"2024-04-25T16:30:08.986873+0000","last_became_peered":"2024-04-25T16:30:08.986873+0000","last_unstale":"2024-04-25T16:31:09.005908+0000","last_undegraded":"2024-04-25T16:31:09.005908+0000","last_fullsized":"2024-04-25T16:31:09.005908+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:40.746479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.956754+0000","last_change":"2024-04-25T16:30:09.988566+0000","last_active":"2024-04-25T16:31:37.956754+0000","last_peered":"2024-04-25T16:31:37.956754+0000","last_clean":"2024-04-25T16:31:37.956754+0000","last_became_active":"2024-04-25T16:30:09.988164+0000","last_became_peered":"2024-04-25T16:30:09.988164+0000","last_unstale":"2024-04-25T16:31:37.956754+0000","last_undegraded":"2024-04-25T16:31:37.956754+0000","last_fullsized":"2024-04-25T16:31:37.956754+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:40.476766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.950950+0000","last_change":"2024-04-25T16:30:09.996349+0000","last_active":"2024-04-25T16:31:37.950950+0000","last_peered":"2024-04-25T16:31:37.950950+0000","last_clean":"2024-04-25T16:31:37.950950+0000","last_became_active":"2024-04-25T16:30:09.994974+0000","last_became_peered":"2024-04-25T16:30:09.994974+0000","last_unstale":"2024-04-25T16:31:37.950950+0000","last_undegraded":"2024-04-25T16:31:37.950950+0000","last_fullsized":"2024-04-25T16:31:37.950950+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:46.929057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951802+0000","last_change":"2024-04-25T16:30:09.997837+0000","last_active":"2024-04-25T16:31:37.951802+0000","last_peered":"2024-04-25T16:31:37.951802+0000","last_clean":"2024-04-25T16:31:37.951802+0000","last_became_active":"2024-04-25T16:30:09.997361+0000","last_became_peered":"2024-04-25T16:30:09.997361+0000","last_unstale":"2024-04-25T16:31:37.951802+0000","last_undegraded":"2024-04-25T16:31:37.951802+0000","last_fullsized":"2024-04-25T16:31:37.951802+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:35.698528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.942275+0000","last_change":"2024-04-25T16:30:09.991794+0000","last_active":"2024-04-25T16:31:37.942275+0000","last_peered":"2024-04-25T16:31:37.942275+0000","last_clean":"2024-04-25T16:31:37.942275+0000","last_became_active":"2024-04-25T16:30:09.991440+0000","last_became_peered":"2024-04-25T16:30:09.991440+0000","last_unstale":"2024-04-25T16:31:37.942275+0000","last_undegraded":"2024-04-25T16:31:37.942275+0000","last_fullsized":"2024-04-25T16:31:37.942275+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:11.322254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.951162+0000","last_change":"2024-04-25T16:30:10.002364+0000","last_active":"2024-04-25T16:31:37.951162+0000","last_peered":"2024-04-25T16:31:37.951162+0000","last_clean":"2024-04-25T16:31:37.951162+0000","last_became_active":"2024-04-25T16:30:10.001957+0000","last_became_peered":"2024-04-25T16:30:10.001957+0000","last_unstale":"2024-04-25T16:31:37.951162+0000","last_undegraded":"2024-04-25T16:31:37.951162+0000","last_fullsized":"2024-04-25T16:31:37.951162+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:19.150175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.944711+0000","last_change":"2024-04-25T16:30:10.004150+0000","last_active":"2024-04-25T16:31:37.944711+0000","last_peered":"2024-04-25T16:31:37.944711+0000","last_clean":"2024-04-25T16:31:37.944711+0000","last_became_active":"2024-04-25T16:30:10.003733+0000","last_became_peered":"2024-04-25T16:30:10.003733+0000","last_unstale":"2024-04-25T16:31:37.944711+0000","last_undegraded":"2024-04-25T16:31:37.944711+0000","last_fullsized":"2024-04-25T16:31:37.944711+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:11.850366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.943924+0000","last_change":"2024-04-25T16:30:09.991056+0000","last_active":"2024-04-25T16:31:37.943924+0000","last_peered":"2024-04-25T16:31:37.943924+0000","last_clean":"2024-04-25T16:31:37.943924+0000","last_became_active":"2024-04-25T16:30:09.988708+0000","last_became_peered":"2024-04-25T16:30:09.988708+0000","last_unstale":"2024-04-25T16:31:37.943924+0000","last_undegraded":"2024-04-25T16:31:37.943924+0000","last_fullsized":"2024-04-25T16:31:37.943924+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:13.062269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.948518+0000","last_change":"2024-04-25T16:30:09.996571+0000","last_active":"2024-04-25T16:31:37.948518+0000","last_peered":"2024-04-25T16:31:37.948518+0000","last_clean":"2024-04-25T16:31:37.948518+0000","last_became_active":"2024-04-25T16:30:09.996206+0000","last_became_peered":"2024-04-25T16:30:09.996206+0000","last_unstale":"2024-04-25T16:31:37.948518+0000","last_undegraded":"2024-04-25T16:31:37.948518+0000","last_fullsized":"2024-04-25T16:31:37.948518+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:07.107949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.942401+0000","last_change":"2024-04-25T16:30:09.995067+0000","last_active":"2024-04-25T16:31:37.942401+0000","last_peered":"2024-04-25T16:31:37.942401+0000","last_clean":"2024-04-25T16:31:37.942401+0000","last_became_active":"2024-04-25T16:30:09.994678+0000","last_became_peered":"2024-04-25T16:30:09.994678+0000","last_unstale":"2024-04-25T16:31:37.942401+0000","last_undegraded":"2024-04-25T16:31:37.942401+0000","last_fullsized":"2024-04-25T16:31:37.942401+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:09.000869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.953243+0000","last_change":"2024-04-25T16:30:09.996821+0000","last_active":"2024-04-25T16:31:37.953243+0000","last_peered":"2024-04-25T16:31:37.953243+0000","last_clean":"2024-04-25T16:31:37.953243+0000","last_became_active":"2024-04-25T16:30:09.996464+0000","last_became_peered":"2024-04-25T16:30:09.996464+0000","last_unstale":"2024-04-25T16:31:37.953243+0000","last_undegraded":"2024-04-25T16:31:37.953243+0000","last_fullsized":"2024-04-25T16:31:37.953243+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:54.961544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.948787+0000","last_change":"2024-04-25T16:30:09.991733+0000","last_active":"2024-04-25T16:31:37.948787+0000","last_peered":"2024-04-25T16:31:37.948787+0000","last_clean":"2024-04-25T16:31:37.948787+0000","last_became_active":"2024-04-25T16:30:09.991401+0000","last_became_peered":"2024-04-25T16:30:09.991401+0000","last_unstale":"2024-04-25T16:31:37.948787+0000","last_undegraded":"2024-04-25T16:31:37.948787+0000","last_fullsized":"2024-04-25T16:31:37.948787+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:14.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.964113+0000","last_change":"2024-04-25T16:30:09.999087+0000","last_active":"2024-04-25T16:31:37.964113+0000","last_peered":"2024-04-25T16:31:37.964113+0000","last_clean":"2024-04-25T16:31:37.964113+0000","last_became_active":"2024-04-25T16:30:09.998725+0000","last_became_peered":"2024-04-25T16:30:09.998725+0000","last_unstale":"2024-04-25T16:31:37.964113+0000","last_undegraded":"2024-04-25T16:31:37.964113+0000","last_fullsized":"2024-04-25T16:31:37.964113+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:19.155171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.950902+0000","last_change":"2024-04-25T16:30:09.995880+0000","last_active":"2024-04-25T16:31:37.950902+0000","last_peered":"2024-04-25T16:31:37.950902+0000","last_clean":"2024-04-25T16:31:37.950902+0000","last_became_active":"2024-04-25T16:30:09.995514+0000","last_became_peered":"2024-04-25T16:30:09.995514+0000","last_unstale":"2024-04-25T16:31:37.950902+0000","last_undegraded":"2024-04-25T16:31:37.950902+0000","last_fullsized":"2024-04-25T16:31:37.950902+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-25T16:31:37.946875+0000","last_change":"2024-04-25T16:30:09.998326+0000","last_active":"2024-04-25T16:31:37.946875+0000","last_peered":"2024-04-25T16:31:37.946875+0000","last_clean":"2024-04-25T16:31:37.946875+0000","last_became_active":"2024-04-25T16:30:09.995803+0000","last_became_peered":"2024-04-25T16:30:09.995803+0000","last_unstale":"2024-04-25T16:31:37.946875+0000","last_undegraded":"2024-04-25T16:31:37.946875+0000","last_fullsized":"2024-04-25T16:31:37.946875+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:46.983947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":71091,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":734,"num_read_kb":445,"num_write":251,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":213273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":219,"ondisk_log_size":219,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645345,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":456380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841574,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36936,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708216,"statfs":{"total":95995035648,"available":95957213184,"internally_reserved":0,"allocated":1638400,"data_stored":1111429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037803,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":460406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234032,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":456688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430261,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28536,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716616,"statfs":{"total":95995035648,"available":95965814784,"internally_reserved":0,"allocated":1556480,"data_stored":1053858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593787,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":451814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":516449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":453270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":517201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36264,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708888,"statfs":{"total":95995035648,"available":95957901312,"internally_reserved":0,"allocated":950272,"data_stored":451522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953541,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36260,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708892,"statfs":{"total":95995035648,"available":95957905408,"internally_reserved":0,"allocated":946176,"data_stored":456420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790016,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36860,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708292,"statfs":{"total":95995035648,"available":95957291008,"internally_reserved":0,"allocated":1560576,"data_stored":1044812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":65357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":66109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":65873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:31:52.202 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T16:31:52.202 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T16:31:52.202 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T16:31:52.202 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T16:31:52.438 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:31:53.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:53 smithi098 ceph-mon[29162]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:31:53.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:53 smithi098 ceph-mon[29162]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:53 smithi173 ceph-mon[32556]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:31:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:53 smithi173 ceph-mon[32556]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:53.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:53 smithi203 ceph-mon[33115]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:31:53.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:53 smithi203 ceph-mon[33115]: pgmap v335: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:31:54.120 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:54 smithi098 ceph-mon[29162]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3289777664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 msgr2=0x7f8aa0107dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3289777664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 0x7f8aa0107dd0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8a9800b0a0 tx=0x7f8a9800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3289777664 shutdown_connections 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3289777664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8aa0109100 0x7f8aa0109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3289777664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 0x7f8aa0107dd0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3289777664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aa0102640 0x7f8aa0102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3289777664 >> 172.21.15.98:0/3289777664 conn(0x7f8aa00fe3f0 msgr2=0x7f8aa0100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3289777664 shutdown_connections 2024-04-25T16:31:54.124 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.120+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3289777664 wait complete. 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa76a2700 1 Processor -- start 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa76a2700 1 -- start start 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa76a2700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8aa0102640 0x7f8aa01a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa76a2700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 0x7f8aa01a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa76a2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aa0109100 0x7f8aa01a5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa76a2700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa011a7c0 con 0x7f8aa0102f90 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa76a2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa011a640 con 0x7f8aa0109100 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa76a2700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa011a940 con 0x7f8aa0102640 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.121+0000 7f8aa4c3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 0x7f8aa01a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa4c3d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 0x7f8aa01a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38244/0 (socket says 172.21.15.98:38244) 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa543e700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8aa0102640 0x7f8aa01a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa4c3d700 1 -- 172.21.15.98:0/3334904760 learned_addr learned my addr 172.21.15.98:0/3334904760 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa5c3f700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aa0109100 0x7f8aa01a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:54.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa4c3d700 1 -- 172.21.15.98:0/3334904760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8aa0102640 msgr2=0x7f8aa01a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa4c3d700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8aa0102640 0x7f8aa01a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa4c3d700 1 -- 172.21.15.98:0/3334904760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aa0109100 msgr2=0x7f8aa01a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa4c3d700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aa0109100 0x7f8aa01a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa4c3d700 1 -- 172.21.15.98:0/3334904760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa01a5fc0 con 0x7f8aa0102f90 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa5c3f700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aa0109100 0x7f8aa01a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa4c3d700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 0x7f8aa01a1690 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000d950 tx=0x7f8a9000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8a967fc700 1 -- 172.21.15.98:0/3334904760 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a90016070 con 0x7f8aa0102f90 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa01a62b0 con 0x7f8aa0102f90 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8a967fc700 1 -- 172.21.15.98:0/3334904760 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8a90009dd0 con 0x7f8aa0102f90 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8a967fc700 1 -- 172.21.15.98:0/3334904760 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9000f3c0 con 0x7f8aa0102f90 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.122+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa01adbd0 con 0x7f8aa0102f90 2024-04-25T16:31:54.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.123+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa0051190 con 0x7f8aa0102f90 2024-04-25T16:31:54.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.125+0000 7f8a967fc700 1 -- 172.21.15.98:0/3334904760 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8a90019070 con 0x7f8aa0102f90 2024-04-25T16:31:54.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.126+0000 7f8a967fc700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a8c0afb80 0x7f8a8c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:54.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.126+0000 7f8aa543e700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a8c0afb80 0x7f8a8c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:54.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.126+0000 7f8a967fc700 1 -- 172.21.15.98:0/3334904760 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f8a900deba0 con 0x7f8aa0102f90 2024-04-25T16:31:54.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.126+0000 7f8aa543e700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a8c0afb80 0x7f8a8c0b2090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8aa006bf10 tx=0x7f8a9c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:54.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.129+0000 7f8a967fc700 1 -- 172.21.15.98:0/3334904760 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a900a7420 con 0x7f8aa0102f90 2024-04-25T16:31:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:54 smithi173 ceph-mon[32556]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T16:31:54.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.375+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8aa01adda0 con 0x7f8aa0102f90 2024-04-25T16:31:54.375 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.375+0000 7f8a967fc700 1 -- 172.21.15.98:0/3334904760 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f8a900a6d20 con 0x7f8aa0102f90 2024-04-25T16:31:54.375 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:54.375 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T16:31:54.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a8c0afb80 msgr2=0x7f8a8c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:54.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a8c0afb80 0x7f8a8c0b2090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8aa006bf10 tx=0x7f8a9c00b410 comp rx=0 tx=0).stop 2024-04-25T16:31:54.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 msgr2=0x7f8aa01a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:54.377 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 0x7f8aa01a1690 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000d950 tx=0x7f8a9000dc60 comp rx=0 tx=0).stop 2024-04-25T16:31:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 shutdown_connections 2024-04-25T16:31:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8a8c0afb80 0x7f8a8c0b2090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8aa0102640 0x7f8aa01a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8aa0109100 0x7f8aa01a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 --2- 172.21.15.98:0/3334904760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8aa0102f90 0x7f8aa01a1690 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 >> 172.21.15.98:0/3334904760 conn(0x7f8aa00fe3f0 msgr2=0x7f8aa0105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 shutdown_connections 2024-04-25T16:31:54.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:54.377+0000 7f8aa76a2700 1 -- 172.21.15.98:0/3334904760 wait complete. 2024-04-25T16:31:54.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:54 smithi203 ceph-mon[33115]: pgmap v336: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T16:31:55.094 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T16:31:55.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 -- 172.21.15.98:0/2195651786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038106f30 msgr2=0x7f4038109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 --2- 172.21.15.98:0/2195651786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038106f30 0x7f4038109350 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f402c009900 tx=0x7f402c02a410 comp rx=0 tx=0).stop 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 -- 172.21.15.98:0/2195651786 shutdown_connections 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 --2- 172.21.15.98:0/2195651786 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4038109890 0x7f403810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 --2- 172.21.15.98:0/2195651786 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038106f30 0x7f4038109350 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 --2- 172.21.15.98:0/2195651786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40381045d0 0x7f40381069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 -- 172.21.15.98:0/2195651786 >> 172.21.15.98:0/2195651786 conn(0x7f40380fdbd0 msgr2=0x7f4038100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 -- 172.21.15.98:0/2195651786 shutdown_connections 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.209+0000 7f403d453640 1 -- 172.21.15.98:0/2195651786 wait complete. 2024-04-25T16:31:55.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f403d453640 1 Processor -- start 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f403d453640 1 -- start start 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f403d453640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40381045d0 0x7f403819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f403d453640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4038106f30 0x7f403819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f403d453640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038109890 0x7f40381a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f403d453640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f403810f4c0 con 0x7f4038109890 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f403d453640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f403810f340 con 0x7f4038106f30 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f403d453640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f403810f640 con 0x7f40381045d0 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f40377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038109890 0x7f40381a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f40377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038109890 0x7f40381a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:38264/0 (socket says 172.21.15.98:38264) 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.210+0000 7f40377fe640 1 -- 172.21.15.98:0/4078915186 learned_addr learned my addr 172.21.15.98:0/4078915186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f4036ffd640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40381045d0 0x7f403819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f40367fc640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4038106f30 0x7f403819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f40377fe640 1 -- 172.21.15.98:0/4078915186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40381045d0 msgr2=0x7f403819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f40377fe640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40381045d0 0x7f403819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f40377fe640 1 -- 172.21.15.98:0/4078915186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4038106f30 msgr2=0x7f403819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f40377fe640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4038106f30 0x7f403819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f40377fe640 1 -- 172.21.15.98:0/4078915186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40381a1c50 con 0x7f4038109890 2024-04-25T16:31:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f40377fe640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038109890 0x7f40381a1690 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f403000cad0 tx=0x7f403000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f401ffff640 1 -- 172.21.15.98:0/4078915186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403000ad60 con 0x7f4038109890 2024-04-25T16:31:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f401ffff640 1 -- 172.21.15.98:0/4078915186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4030003e20 con 0x7f4038109890 2024-04-25T16:31:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40381a1f40 con 0x7f4038109890 2024-04-25T16:31:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f401ffff640 1 -- 172.21.15.98:0/4078915186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403000f6b0 con 0x7f4038109890 2024-04-25T16:31:55.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.211+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40381a23a0 con 0x7f4038109890 2024-04-25T16:31:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.212+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f403806a190 con 0x7f4038109890 2024-04-25T16:31:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.214+0000 7f401ffff640 1 -- 172.21.15.98:0/4078915186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4030003990 con 0x7f4038109890 2024-04-25T16:31:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.214+0000 7f401ffff640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f40200afaf0 0x7f40200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.214+0000 7f4036ffd640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f40200afaf0 0x7f40200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.214+0000 7f401ffff640 1 -- 172.21.15.98:0/4078915186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f40300df280 con 0x7f4038109890 2024-04-25T16:31:55.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.215+0000 7f4036ffd640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f40200afaf0 0x7f40200b2000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f40380757f0 tx=0x7f402800a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:55.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.218+0000 7f401ffff640 1 -- 172.21.15.98:0/4078915186 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4030014070 con 0x7f4038109890 2024-04-25T16:31:55.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:55 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3334904760' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:31:55.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:55 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3334904760' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:31:55.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.397+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f403802dc10 con 0x7f4038109890 2024-04-25T16:31:55.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.398+0000 7f401ffff640 1 -- 172.21.15.98:0/4078915186 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+24609 (secure 0 0 0) 0x7f40300a7b00 con 0x7f4038109890 2024-04-25T16:31:55.398 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:55.398 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:31:08.997612+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:31:55.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f40200afaf0 msgr2=0x7f40200b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f40200afaf0 0x7f40200b2000 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f40380757f0 tx=0x7f402800a2d0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038109890 msgr2=0x7f40381a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038109890 0x7f40381a1690 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f403000cad0 tx=0x7f403000ced0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 shutdown_connections 2024-04-25T16:31:55.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f40200afaf0 0x7f40200b2000 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4038109890 0x7f40381a1690 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4038106f30 0x7f403819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.400+0000 7f403d453640 1 --2- 172.21.15.98:0/4078915186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f40381045d0 0x7f403819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.401+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 >> 172.21.15.98:0/4078915186 conn(0x7f40380fdbd0 msgr2=0x7f4038106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:55.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.401+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 shutdown_connections 2024-04-25T16:31:55.401 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.401+0000 7f403d453640 1 -- 172.21.15.98:0/4078915186 wait complete. 2024-04-25T16:31:55.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:55 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3334904760' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:31:55.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:31:55.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:31:55.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 -- 172.21.15.98:0/3734280723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc1029e0 msgr2=0x7fc4fc1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 --2- 172.21.15.98:0/3734280723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc1029e0 0x7fc4fc1036e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4009ca0 tx=0x7fc4f402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 -- 172.21.15.98:0/3734280723 shutdown_connections 2024-04-25T16:31:55.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 --2- 172.21.15.98:0/3734280723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4fc108920 0x7fc4fc10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 --2- 172.21.15.98:0/3734280723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4fc103c20 0x7fc4fc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 --2- 172.21.15.98:0/3734280723 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc1029e0 0x7fc4fc1036e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 -- 172.21.15.98:0/3734280723 >> 172.21.15.98:0/3734280723 conn(0x7fc4fc0fdbd0 msgr2=0x7fc4fc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:55.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 -- 172.21.15.98:0/3734280723 shutdown_connections 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.524+0000 7fc50440c640 1 -- 172.21.15.98:0/3734280723 wait complete. 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc50440c640 1 Processor -- start 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc50440c640 1 -- start start 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc50440c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4fc1029e0 0x7fc4fc19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc50440c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc103c20 0x7fc4fc19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc50440c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4fc108920 0x7fc4fc1a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc50440c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc4fc10f540 con 0x7fc4fc103c20 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc50440c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc4fc10f3c0 con 0x7fc4fc108920 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc50440c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc4fc10f6c0 con 0x7fc4fc1029e0 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc5019a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc103c20 0x7fc4fc19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc5019a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc103c20 0x7fc4fc19aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40668/0 (socket says 172.21.15.98:40668) 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc5019a7640 1 -- 172.21.15.98:0/4114420220 learned_addr learned my addr 172.21.15.98:0/4114420220 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.525+0000 7fc5021a8640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4fc1029e0 0x7fc4fc19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc5029a9640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4fc108920 0x7fc4fc1a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc5019a7640 1 -- 172.21.15.98:0/4114420220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4fc1029e0 msgr2=0x7fc4fc19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc5019a7640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4fc1029e0 0x7fc4fc19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc5019a7640 1 -- 172.21.15.98:0/4114420220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4fc108920 msgr2=0x7fc4fc1a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc5019a7640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4fc108920 0x7fc4fc1a19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc5019a7640 1 -- 172.21.15.98:0/4114420220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4fc1a1f90 con 0x7fc4fc103c20 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc5019a7640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc103c20 0x7fc4fc19aab0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00b4a0 tx=0x7fc4ec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc4f37fe640 1 -- 172.21.15.98:0/4114420220 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec014070 con 0x7fc4fc103c20 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc4f37fe640 1 -- 172.21.15.98:0/4114420220 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc4ec010aa0 con 0x7fc4fc103c20 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4fc1a2280 con 0x7fc4fc103c20 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc4f37fe640 1 -- 172.21.15.98:0/4114420220 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4ec017b70 con 0x7fc4fc103c20 2024-04-25T16:31:55.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.526+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4fc1a27d0 con 0x7fc4fc103c20 2024-04-25T16:31:55.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.527+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4fc107730 con 0x7fc4fc103c20 2024-04-25T16:31:55.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.528+0000 7fc4f37fe640 1 -- 172.21.15.98:0/4114420220 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc4ec017070 con 0x7fc4fc103c20 2024-04-25T16:31:55.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.529+0000 7fc4f37fe640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc4e80afaf0 0x7fc4e80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.529+0000 7fc5021a8640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc4e80afaf0 0x7fc4e80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.529+0000 7fc4f37fe640 1 -- 172.21.15.98:0/4114420220 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fc4ec0de670 con 0x7fc4fc103c20 2024-04-25T16:31:55.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.529+0000 7fc5021a8640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc4e80afaf0 0x7fc4e80b2000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4006010 tx=0x7fc4f40318e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:55.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.532+0000 7fc4f37fe640 1 -- 172.21.15.98:0/4114420220 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4ec0a6ef0 con 0x7fc4fc103c20 2024-04-25T16:31:55.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.709+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc4fc02dce0 con 0x7fc4fc103c20 2024-04-25T16:31:55.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.710+0000 7fc4f37fe640 1 -- 172.21.15.98:0/4114420220 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7fc4ec0a67f0 con 0x7fc4fc103c20 2024-04-25T16:31:55.709 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:31:55.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc4e80afaf0 msgr2=0x7fc4e80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc4e80afaf0 0x7fc4e80b2000 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc4f4006010 tx=0x7fc4f40318e0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc103c20 msgr2=0x7fc4fc19aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc103c20 0x7fc4fc19aab0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec00b4a0 tx=0x7fc4ec00b960 comp rx=0 tx=0).stop 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 shutdown_connections 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc4e80afaf0 0x7fc4e80b2000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4fc108920 0x7fc4fc1a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc4fc103c20 0x7fc4fc19aab0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 --2- 172.21.15.98:0/4114420220 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4fc1029e0 0x7fc4fc19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 >> 172.21.15.98:0/4114420220 conn(0x7fc4fc0fdbd0 msgr2=0x7fc4fc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 shutdown_connections 2024-04-25T16:31:55.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.712+0000 7fc50440c640 1 -- 172.21.15.98:0/4114420220 wait complete. 2024-04-25T16:31:55.723 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 -- 172.21.15.98:0/3935373496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 msgr2=0x7fcd441036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 --2- 172.21.15.98:0/3935373496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 0x7fcd441036e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009900 tx=0x7fcd3402a410 comp rx=0 tx=0).stop 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 -- 172.21.15.98:0/3935373496 shutdown_connections 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 --2- 172.21.15.98:0/3935373496 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd44108920 0x7fcd4410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 --2- 172.21.15.98:0/3935373496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd44103c20 0x7fcd44104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 --2- 172.21.15.98:0/3935373496 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 0x7fcd441036e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 -- 172.21.15.98:0/3935373496 >> 172.21.15.98:0/3935373496 conn(0x7fcd440fdbd0 msgr2=0x7fcd44100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 -- 172.21.15.98:0/3935373496 shutdown_connections 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.834+0000 7fcd49815640 1 -- 172.21.15.98:0/3935373496 wait complete. 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd49815640 1 Processor -- start 2024-04-25T16:31:55.836 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd49815640 1 -- start start 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd49815640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 0x7fcd441993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd49815640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd44103c20 0x7fcd441998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd49815640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd44108920 0x7fcd441a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd49815640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410e3c0 con 0x7fcd441029e0 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd49815640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410e240 con 0x7fcd44103c20 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd49815640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4410e540 con 0x7fcd44108920 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd42ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 0x7fcd441993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd42ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 0x7fcd441993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40672/0 (socket says 172.21.15.98:40672) 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd42ffd640 1 -- 172.21.15.98:0/830759380 learned_addr learned my addr 172.21.15.98:0/830759380 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd3bfff640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd44103c20 0x7fcd441998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd437fe640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd44108920 0x7fcd441a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd42ffd640 1 -- 172.21.15.98:0/830759380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd44108920 msgr2=0x7fcd441a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd42ffd640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd44108920 0x7fcd441a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd42ffd640 1 -- 172.21.15.98:0/830759380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd44103c20 msgr2=0x7fcd441998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd42ffd640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd44103c20 0x7fcd441998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.835+0000 7fcd42ffd640 1 -- 172.21.15.98:0/830759380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd441a0e30 con 0x7fcd441029e0 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.836+0000 7fcd42ffd640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 0x7fcd441993a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcd340057c0 tx=0x7fcd34003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.836+0000 7fcd48813640 1 -- 172.21.15.98:0/830759380 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34035070 con 0x7fcd441029e0 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.836+0000 7fcd48813640 1 -- 172.21.15.98:0/830759380 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcd34003e40 con 0x7fcd441029e0 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.836+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd44069e90 con 0x7fcd441029e0 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.836+0000 7fcd48813640 1 -- 172.21.15.98:0/830759380 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34038ae0 con 0x7fcd441029e0 2024-04-25T16:31:55.838 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.836+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd4406a380 con 0x7fcd441029e0 2024-04-25T16:31:55.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.837+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd4406c2a0 con 0x7fcd441029e0 2024-04-25T16:31:55.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.838+0000 7fcd48813640 1 -- 172.21.15.98:0/830759380 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcd34038ca0 con 0x7fcd441029e0 2024-04-25T16:31:55.839 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.839+0000 7fcd48813640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcd300afaf0 0x7fcd300b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:55.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.839+0000 7fcd3bfff640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcd300afaf0 0x7fcd300b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:55.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.839+0000 7fcd48813640 1 -- 172.21.15.98:0/830759380 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fcd34101260 con 0x7fcd441029e0 2024-04-25T16:31:55.840 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.839+0000 7fcd3bfff640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcd300afaf0 0x7fcd300b2000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0058c0 tx=0x7fcd2c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:55.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:55.844+0000 7fcd48813640 1 -- 172.21.15.98:0/830759380 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd340c9ae0 con 0x7fcd441029e0 2024-04-25T16:31:56.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.033+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcd44107320 con 0x7fcd441029e0 2024-04-25T16:31:56.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.034+0000 7fcd48813640 1 -- 172.21.15.98:0/830759380 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) ==== 104+0+11 (secure 0 0 0) 0x7fcd340c93e0 con 0x7fcd441029e0 2024-04-25T16:31:56.034 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:31:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcd300afaf0 msgr2=0x7fcd300b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcd300afaf0 0x7fcd300b2000 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c0058c0 tx=0x7fcd2c00a2d0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 msgr2=0x7fcd441993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 0x7fcd441993a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcd340057c0 tx=0x7fcd34003cd0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 shutdown_connections 2024-04-25T16:31:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fcd300afaf0 0x7fcd300b2000 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcd44108920 0x7fcd441a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd44103c20 0x7fcd441998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 --2- 172.21.15.98:0/830759380 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fcd441029e0 0x7fcd441993a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 >> 172.21.15.98:0/830759380 conn(0x7fcd440fdbd0 msgr2=0x7fcd44104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 shutdown_connections 2024-04-25T16:31:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.036+0000 7fcd49815640 1 -- 172.21.15.98:0/830759380 wait complete. 2024-04-25T16:31:56.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:31:56.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 -- 172.21.15.98:0/4012730617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00104480 msgr2=0x7f0e0010abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 --2- 172.21.15.98:0/4012730617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00104480 0x7f0e0010abc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0df8009ca0 tx=0x7f0df802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 -- 172.21.15.98:0/4012730617 shutdown_connections 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 --2- 172.21.15.98:0/4012730617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00104480 0x7f0e0010abc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 --2- 172.21.15.98:0/4012730617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e00103980 0x7f0e00103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 --2- 172.21.15.98:0/4012730617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e00102fa0 0x7f0e001033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 -- 172.21.15.98:0/4012730617 >> 172.21.15.98:0/4012730617 conn(0x7f0e000fd380 msgr2=0x7f0e000ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 -- 172.21.15.98:0/4012730617 shutdown_connections 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.162+0000 7f0e0669f640 1 -- 172.21.15.98:0/4012730617 wait complete. 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0e0669f640 1 Processor -- start 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0e0669f640 1 -- start start 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0e0669f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e00102fa0 0x7f0e00198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0e0669f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00103980 0x7f0e00199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0e0669f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e00104480 0x7f0e001a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0e0669f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0010e420 con 0x7f0e00103980 2024-04-25T16:31:56.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0e0669f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0010e2a0 con 0x7f0e00102fa0 2024-04-25T16:31:56.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0e0669f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0010e5a0 con 0x7f0e00104480 2024-04-25T16:31:56.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0dff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00103980 0x7f0e00199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0dff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00103980 0x7f0e00199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40688/0 (socket says 172.21.15.98:40688) 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.163+0000 7f0dff7fe640 1 -- 172.21.15.98:0/1493181672 learned_addr learned my addr 172.21.15.98:0/1493181672 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0e04c3c640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e00104480 0x7f0e001a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dfffff640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e00102fa0 0x7f0e00198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dff7fe640 1 -- 172.21.15.98:0/1493181672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e00104480 msgr2=0x7f0e001a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dff7fe640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e00104480 0x7f0e001a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dff7fe640 1 -- 172.21.15.98:0/1493181672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e00102fa0 msgr2=0x7f0e00198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dff7fe640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e00102fa0 0x7f0e00198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dff7fe640 1 -- 172.21.15.98:0/1493181672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e001a0b10 con 0x7f0e00103980 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dfffff640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e00102fa0 0x7f0e00198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:56.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0e04c3c640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e00104480 0x7f0e001a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:56.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dff7fe640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00103980 0x7f0e00199510 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0df400d950 tx=0x7f0df400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:56.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dfd7fa640 1 -- 172.21.15.98:0/1493181672 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df4017070 con 0x7f0e00103980 2024-04-25T16:31:56.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dfd7fa640 1 -- 172.21.15.98:0/1493181672 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0df4009c30 con 0x7f0e00103980 2024-04-25T16:31:56.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0dfd7fa640 1 -- 172.21.15.98:0/1493181672 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df400f620 con 0x7f0e00103980 2024-04-25T16:31:56.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e001a0e00 con 0x7f0e00103980 2024-04-25T16:31:56.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.164+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e001a1270 con 0x7f0e00103980 2024-04-25T16:31:56.169 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.165+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e00109950 con 0x7f0e00103980 2024-04-25T16:31:56.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.166+0000 7f0dfd7fa640 1 -- 172.21.15.98:0/1493181672 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0df4010460 con 0x7f0e00103980 2024-04-25T16:31:56.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.167+0000 7f0dfd7fa640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0de80afaf0 0x7f0de80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.167+0000 7f0dfffff640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0de80afaf0 0x7f0de80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.167+0000 7f0dfd7fa640 1 -- 172.21.15.98:0/1493181672 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f0df40dec40 con 0x7f0e00103980 2024-04-25T16:31:56.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.168+0000 7f0dfffff640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0de80afaf0 0x7f0de80b2000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0e0006b290 tx=0x7f0df0005e70 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:56.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.171+0000 7f0dfd7fa640 1 -- 172.21.15.98:0/1493181672 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0df40a74c0 con 0x7f0e00103980 2024-04-25T16:31:56.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.348+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0e0002dda0 con 0x7f0e00103980 2024-04-25T16:31:56.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.348+0000 7f0dfd7fa640 1 -- 172.21.15.98:0/1493181672 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) ==== 100+0+11 (secure 0 0 0) 0x7f0df4009de0 con 0x7f0e00103980 2024-04-25T16:31:56.347 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:31:56.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0de80afaf0 msgr2=0x7f0de80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0de80afaf0 0x7f0de80b2000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0e0006b290 tx=0x7f0df0005e70 comp rx=0 tx=0).stop 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00103980 msgr2=0x7f0e00199510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00103980 0x7f0e00199510 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0df400d950 tx=0x7f0df400de10 comp rx=0 tx=0).stop 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 shutdown_connections 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0de80afaf0 0x7f0de80b2000 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e00104480 0x7f0e001a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0e00103980 0x7f0e00199510 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 --2- 172.21.15.98:0/1493181672 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e00102fa0 0x7f0e00198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 >> 172.21.15.98:0/1493181672 conn(0x7f0e000fd380 msgr2=0x7f0e00106b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 shutdown_connections 2024-04-25T16:31:56.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.350+0000 7f0e0669f640 1 -- 172.21.15.98:0/1493181672 wait complete. 2024-04-25T16:31:56.360 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:56 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4078915186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:31:56.360 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:56 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4114420220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:31:56.360 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:56 smithi098 ceph-mon[29162]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:31:56.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:31:56.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:56 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4078915186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:31:56.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:56 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4114420220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:31:56.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:56 smithi203 ceph-mon[33115]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:31:56.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/2472177028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b381080b0 msgr2=0x7f8b3810a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/2472177028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b381080b0 0x7f8b3810a4d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009ca0 tx=0x7f8b3002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/2472177028 shutdown_connections 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/2472177028 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b381080b0 0x7f8b3810a4d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/2472177028 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b381033b0 0x7f8b381037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/2472177028 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b38102170 0x7f8b38102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/2472177028 >> 172.21.15.98:0/2472177028 conn(0x7f8b380fd340 msgr2=0x7f8b380ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/2472177028 shutdown_connections 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.476+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/2472177028 wait complete. 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3fa4a640 1 Processor -- start 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3fa4a640 1 -- start start 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3fa4a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b38102170 0x7f8b3819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3fa4a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b381033b0 0x7f8b3819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3fa4a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b381080b0 0x7f8b381a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3fa4a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3810d300 con 0x7f8b38102170 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3fa4a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3810d180 con 0x7f8b381033b0 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3fa4a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3810d480 con 0x7f8b381080b0 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3d7e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b38102170 0x7f8b3819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3d7e6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b38102170 0x7f8b3819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40700/0 (socket says 172.21.15.98:40700) 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3d7e6640 1 -- 172.21.15.98:0/545222561 learned_addr learned my addr 172.21.15.98:0/545222561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3dfe7640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b381080b0 0x7f8b381a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.477+0000 7f8b3cfe5640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b381033b0 0x7f8b3819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3d7e6640 1 -- 172.21.15.98:0/545222561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b381080b0 msgr2=0x7f8b381a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3d7e6640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b381080b0 0x7f8b381a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3d7e6640 1 -- 172.21.15.98:0/545222561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b381033b0 msgr2=0x7f8b3819a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3d7e6640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b381033b0 0x7f8b3819a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3d7e6640 1 -- 172.21.15.98:0/545222561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b381a1c50 con 0x7f8b38102170 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3d7e6640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b38102170 0x7f8b3819a110 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800d740 tx=0x7f8b2800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b2e7fc640 1 -- 172.21.15.98:0/545222561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28017070 con 0x7f8b38102170 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b2e7fc640 1 -- 172.21.15.98:0/545222561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b2800de50 con 0x7f8b38102170 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b2e7fc640 1 -- 172.21.15.98:0/545222561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28003bb0 con 0x7f8b38102170 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b381a1f40 con 0x7f8b38102170 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.478+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b381a23a0 con 0x7f8b38102170 2024-04-25T16:31:56.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.479+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b38069310 con 0x7f8b38102170 2024-04-25T16:31:56.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.481+0000 7f8b2e7fc640 1 -- 172.21.15.98:0/545222561 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8b28003d50 con 0x7f8b38102170 2024-04-25T16:31:56.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.481+0000 7f8b2e7fc640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8b240afaf0 0x7f8b240b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.481+0000 7f8b3cfe5640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8b240afaf0 0x7f8b240b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.481+0000 7f8b2e7fc640 1 -- 172.21.15.98:0/545222561 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f8b280decc0 con 0x7f8b38102170 2024-04-25T16:31:56.483 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.482+0000 7f8b3cfe5640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8b240afaf0 0x7f8b240b2000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8b3819b1b0 tx=0x7f8b3400b490 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:56.484 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.485+0000 7f8b2e7fc640 1 -- 172.21.15.98:0/545222561 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b38069310 con 0x7f8b38102170 2024-04-25T16:31:56.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:56 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4078915186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:31:56.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:56 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4114420220' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:31:56.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:56 smithi173 ceph-mon[32556]: pgmap v337: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:31:56.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.663+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8b3802dbd0 con 0x7f8b38102170 2024-04-25T16:31:56.663 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.664+0000 7f8b2e7fc640 1 -- 172.21.15.98:0/545222561 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) ==== 103+0+11 (secure 0 0 0) 0x7f8b280a6ef0 con 0x7f8b38102170 2024-04-25T16:31:56.663 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:31:56.665 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8b240afaf0 msgr2=0x7f8b240b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8b240afaf0 0x7f8b240b2000 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8b3819b1b0 tx=0x7f8b3400b490 comp rx=0 tx=0).stop 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b38102170 msgr2=0x7f8b3819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b38102170 0x7f8b3819a110 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800d740 tx=0x7f8b2800dc00 comp rx=0 tx=0).stop 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 shutdown_connections 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8b240afaf0 0x7f8b240b2000 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8b381080b0 0x7f8b381a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b381033b0 0x7f8b3819a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 --2- 172.21.15.98:0/545222561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8b38102170 0x7f8b3819a110 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 >> 172.21.15.98:0/545222561 conn(0x7f8b380fd340 msgr2=0x7f8b38104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:56.666 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 shutdown_connections 2024-04-25T16:31:56.667 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.666+0000 7f8b3fa4a640 1 -- 172.21.15.98:0/545222561 wait complete. 2024-04-25T16:31:56.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.785+0000 7f771be20640 1 -- 172.21.15.98:0/1113023598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 msgr2=0x7f77141033a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.785+0000 7f771be20640 1 --2- 172.21.15.98:0/1113023598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 0x7f77141033a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7710009900 tx=0x7f771002a410 comp rx=0 tx=0).stop 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.785+0000 7f771be20640 1 -- 172.21.15.98:0/1113023598 shutdown_connections 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 --2- 172.21.15.98:0/1113023598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7714104470 0x7f771410abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 --2- 172.21.15.98:0/1113023598 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7714103970 0x7f7714103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 --2- 172.21.15.98:0/1113023598 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 0x7f77141033a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 -- 172.21.15.98:0/1113023598 >> 172.21.15.98:0/1113023598 conn(0x7f77140fd370 msgr2=0x7f77140ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 -- 172.21.15.98:0/1113023598 shutdown_connections 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 -- 172.21.15.98:0/1113023598 wait complete. 2024-04-25T16:31:56.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 Processor -- start 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 -- start start 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 0x7f771419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7714103970 0x7f771419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7714104470 0x7f77141a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f771410e3c0 con 0x7f7714102f90 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f771410e240 con 0x7f7714104470 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.786+0000 7f771be20640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f771410e540 con 0x7f7714103970 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 0x7f771419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 0x7f771419a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40712/0 (socket says 172.21.15.98:40712) 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 -- 172.21.15.98:0/1715819326 learned_addr learned my addr 172.21.15.98:0/1715819326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:56.789 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f771a3bd640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7714104470 0x7f77141a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f77193bb640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7714103970 0x7f771419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 -- 172.21.15.98:0/1715819326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7714103970 msgr2=0x7f771419a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7714103970 0x7f771419a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 -- 172.21.15.98:0/1715819326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7714104470 msgr2=0x7f77141a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7714104470 0x7f77141a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 -- 172.21.15.98:0/1715819326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77141a1ce0 con 0x7f7714102f90 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f7719bbc640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 0x7f771419a1a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7710005ab0 tx=0x7f7710004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f770affd640 1 -- 172.21.15.98:0/1715819326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771003f070 con 0x7f7714102f90 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f770affd640 1 -- 172.21.15.98:0/1715819326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7710003c80 con 0x7f7714102f90 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77141a1f70 con 0x7f7714102f90 2024-04-25T16:31:56.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f770affd640 1 -- 172.21.15.98:0/1715819326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710035070 con 0x7f7714102f90 2024-04-25T16:31:56.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.787+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77141a2370 con 0x7f7714102f90 2024-04-25T16:31:56.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.788+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77141a25a0 con 0x7f7714102f90 2024-04-25T16:31:56.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.790+0000 7f770affd640 1 -- 172.21.15.98:0/1715819326 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f771003b610 con 0x7f7714102f90 2024-04-25T16:31:56.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.791+0000 7f770affd640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f77000afaf0 0x7f77000b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:56.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.791+0000 7f77193bb640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f77000afaf0 0x7f77000b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:56.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.791+0000 7f770affd640 1 -- 172.21.15.98:0/1715819326 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f77100fff40 con 0x7f7714102f90 2024-04-25T16:31:56.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.791+0000 7f77193bb640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f77000afaf0 0x7f77000b2000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f77040058c0 tx=0x7f7704005c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:56.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:56.795+0000 7f770affd640 1 -- 172.21.15.98:0/1715819326 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77100c87c0 con 0x7f7714102f90 2024-04-25T16:31:57.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.011+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77141a2770 con 0x7f7714102f90 2024-04-25T16:31:57.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.012+0000 7f770affd640 1 -- 172.21.15.98:0/1715819326 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7f77100c80c0 con 0x7f7714102f90 2024-04-25T16:31:57.011 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:57.012 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:30:38.051368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24484,24539,24457,24473,24494]},"id":1}]} 2024-04-25T16:31:57.012 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f77000afaf0 msgr2=0x7f77000b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f77000afaf0 0x7f77000b2000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f77040058c0 tx=0x7f7704005c80 comp rx=0 tx=0).stop 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 msgr2=0x7f771419a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 0x7f771419a1a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7710005ab0 tx=0x7f7710004270 comp rx=0 tx=0).stop 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 shutdown_connections 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f77000afaf0 0x7f77000b2000 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7714104470 0x7f77141a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7714103970 0x7f771419a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 --2- 172.21.15.98:0/1715819326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7714102f90 0x7f771419a1a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 >> 172.21.15.98:0/1715819326 conn(0x7f77140fd370 msgr2=0x7f7714109d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 shutdown_connections 2024-04-25T16:31:57.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.014+0000 7f771be20640 1 -- 172.21.15.98:0/1715819326 wait complete. 2024-04-25T16:31:57.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 -- 172.21.15.98:0/3772574682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178109790 msgr2=0x7f517810bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 --2- 172.21.15.98:0/3772574682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178109790 0x7f517810bca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5170009ca0 tx=0x7f517002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 -- 172.21.15.98:0/3772574682 shutdown_connections 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 --2- 172.21.15.98:0/3772574682 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178109790 0x7f517810bca0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 --2- 172.21.15.98:0/3772574682 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5178106c20 0x7f5178109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 --2- 172.21.15.98:0/3772574682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5178101e40 0x7f5178102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 -- 172.21.15.98:0/3772574682 >> 172.21.15.98:0/3772574682 conn(0x7f51780fd370 msgr2=0x7f51780ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 -- 172.21.15.98:0/3772574682 shutdown_connections 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 -- 172.21.15.98:0/3772574682 wait complete. 2024-04-25T16:31:57.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.134+0000 7f517f8f0640 1 Processor -- start 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 -- start start 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5178101e40 0x7f517819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178106c20 0x7f517819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5178109790 0x7f51781a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f517810f520 con 0x7f5178106c20 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f517810f3a0 con 0x7f5178101e40 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f517810f6a0 con 0x7f5178109790 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178106c20 0x7f517819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178106c20 0x7f517819a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40720/0 (socket says 172.21.15.98:40720) 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 -- 172.21.15.98:0/1687442131 learned_addr learned my addr 172.21.15.98:0/1687442131 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517de8d640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5178109790 0x7f51781a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517d68c640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5178101e40 0x7f517819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 -- 172.21.15.98:0/1687442131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5178109790 msgr2=0x7f51781a1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5178109790 0x7f51781a1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 -- 172.21.15.98:0/1687442131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5178101e40 msgr2=0x7f517819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5178101e40 0x7f517819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 -- 172.21.15.98:0/1687442131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51781a1bd0 con 0x7f5178106c20 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517ce8b640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178106c20 0x7f517819a5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f516800d770 tx=0x7f516800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f516e7fc640 1 -- 172.21.15.98:0/1687442131 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168017070 con 0x7f5178106c20 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f516e7fc640 1 -- 172.21.15.98:0/1687442131 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51680054b0 con 0x7f5178106c20 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51781a1ec0 con 0x7f5178106c20 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f516e7fc640 1 -- 172.21.15.98:0/1687442131 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168004c60 con 0x7f5178106c20 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.135+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51781a2330 con 0x7f5178106c20 2024-04-25T16:31:57.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.136+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51780fee90 con 0x7f5178106c20 2024-04-25T16:31:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.138+0000 7f516e7fc640 1 -- 172.21.15.98:0/1687442131 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5168004e00 con 0x7f5178106c20 2024-04-25T16:31:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.138+0000 7f516e7fc640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51640afaf0 0x7f51640b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.139+0000 7f517d68c640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51640afaf0 0x7f51640b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.139+0000 7f516e7fc640 1 -- 172.21.15.98:0/1687442131 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f51680deef0 con 0x7f5178106c20 2024-04-25T16:31:57.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.139+0000 7f517d68c640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51640afaf0 0x7f51640b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5174005fd0 tx=0x7f5174005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.141 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.142+0000 7f516e7fc640 1 -- 172.21.15.98:0/1687442131 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51680a7770 con 0x7f5178106c20 2024-04-25T16:31:57.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.320+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f517802ddc0 con 0x7f5178106c20 2024-04-25T16:31:57.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.321+0000 7f516e7fc640 1 -- 172.21.15.98:0/1687442131 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+24609 (secure 0 0 0) 0x7f51680a7070 con 0x7f5178106c20 2024-04-25T16:31:57.321 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:57.322 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:31:08.997612+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51640afaf0 msgr2=0x7f51640b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51640afaf0 0x7f51640b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5174005fd0 tx=0x7f5174005850 comp rx=0 tx=0).stop 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178106c20 msgr2=0x7f517819a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178106c20 0x7f517819a5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f516800d770 tx=0x7f516800dc30 comp rx=0 tx=0).stop 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 shutdown_connections 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51640afaf0 0x7f51640b2000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5178109790 0x7f51781a1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5178106c20 0x7f517819a5d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 --2- 172.21.15.98:0/1687442131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5178101e40 0x7f517819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 >> 172.21.15.98:0/1687442131 conn(0x7f51780fd370 msgr2=0x7f5178107bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 shutdown_connections 2024-04-25T16:31:57.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.324+0000 7f517f8f0640 1 -- 172.21.15.98:0/1687442131 wait complete. 2024-04-25T16:31:57.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:57 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/830759380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:57 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1493181672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:57 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/545222561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:57 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1715819326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:31:57.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:31:57.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:31:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:57 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/830759380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:57 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1493181672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:57 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/545222561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:57 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1715819326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:31:57.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 -- 172.21.15.98:0/2731924617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 msgr2=0x7fe9fc101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 --2- 172.21.15.98:0/2731924617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 0x7fe9fc101bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009900 tx=0x7fe9ec02a410 comp rx=0 tx=0).stop 2024-04-25T16:31:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 -- 172.21.15.98:0/2731924617 shutdown_connections 2024-04-25T16:31:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 --2- 172.21.15.98:0/2731924617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9fc102260 0x7fe9fc10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 --2- 172.21.15.98:0/2731924617 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 0x7fe9fc101bd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 --2- 172.21.15.98:0/2731924617 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc1050b0 0x7fe9fc1054c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 -- 172.21.15.98:0/2731924617 >> 172.21.15.98:0/2731924617 conn(0x7fe9fc0fd380 msgr2=0x7fe9fc0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 -- 172.21.15.98:0/2731924617 shutdown_connections 2024-04-25T16:31:57.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.447+0000 7fea00e84640 1 -- 172.21.15.98:0/2731924617 wait complete. 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fea00e84640 1 Processor -- start 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fea00e84640 1 -- start start 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fea00e84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 0x7fe9fc19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fea00e84640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9fc102260 0x7fe9fc19a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fea00e84640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc1050b0 0x7fe9fc1a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fea00e84640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10f4d0 con 0x7fe9fc101760 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fea00e84640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10f350 con 0x7fe9fc102260 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fea00e84640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc10f650 con 0x7fe9fc1050b0 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 0x7fe9fc19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fa59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 0x7fe9fc19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40738/0 (socket says 172.21.15.98:40738) 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fa59c640 1 -- 172.21.15.98:0/277388760 learned_addr learned my addr 172.21.15.98:0/277388760 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fad9d640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc1050b0 0x7fe9fc1a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9f9d9b640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9fc102260 0x7fe9fc19a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fa59c640 1 -- 172.21.15.98:0/277388760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc1050b0 msgr2=0x7fe9fc1a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fa59c640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc1050b0 0x7fe9fc1a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fa59c640 1 -- 172.21.15.98:0/277388760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9fc102260 msgr2=0x7fe9fc19a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fa59c640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9fc102260 0x7fe9fc19a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fa59c640 1 -- 172.21.15.98:0/277388760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc1a1cc0 con 0x7fe9fc101760 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.448+0000 7fe9fad9d640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc1050b0 0x7fe9fc1a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.449+0000 7fe9fa59c640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 0x7fe9fc19a180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0005fd0 tx=0x7fe9f000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.449+0000 7fe9eb7fe640 1 -- 172.21.15.98:0/277388760 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0017070 con 0x7fe9fc101760 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.449+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9fc1a1fb0 con 0x7fe9fc101760 2024-04-25T16:31:57.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.449+0000 7fe9eb7fe640 1 -- 172.21.15.98:0/277388760 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe9f0009c30 con 0x7fe9fc101760 2024-04-25T16:31:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.449+0000 7fe9eb7fe640 1 -- 172.21.15.98:0/277388760 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0003c90 con 0x7fe9fc101760 2024-04-25T16:31:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.449+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9fc1a2470 con 0x7fe9fc101760 2024-04-25T16:31:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.450+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9fc108a40 con 0x7fe9fc101760 2024-04-25T16:31:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.451+0000 7fe9eb7fe640 1 -- 172.21.15.98:0/277388760 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe9f0003e30 con 0x7fe9fc101760 2024-04-25T16:31:57.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.452+0000 7fe9eb7fe640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9e40afaf0 0x7fe9e40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.452+0000 7fe9f9d9b640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9e40afaf0 0x7fe9e40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.452+0000 7fe9eb7fe640 1 -- 172.21.15.98:0/277388760 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fe9f00df450 con 0x7fe9fc101760 2024-04-25T16:31:57.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.452+0000 7fe9f9d9b640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9e40afaf0 0x7fe9e40b2000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec006010 tx=0x7fe9ec000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.455+0000 7fe9eb7fe640 1 -- 172.21.15.98:0/277388760 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe9f0020030 con 0x7fe9fc101760 2024-04-25T16:31:57.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:57 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/830759380' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:57 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1493181672' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:57 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/545222561' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:31:57.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:57 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1715819326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:31:57.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.632+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe9fc02db50 con 0x7fe9fc101760 2024-04-25T16:31:57.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.633+0000 7fe9eb7fe640 1 -- 172.21.15.98:0/277388760 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7fe9f00a7cd0 con 0x7fe9fc101760 2024-04-25T16:31:57.632 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:31:57.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9e40afaf0 msgr2=0x7fe9e40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9e40afaf0 0x7fe9e40b2000 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec006010 tx=0x7fe9ec000bc0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 msgr2=0x7fe9fc19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 0x7fe9fc19a180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0005fd0 tx=0x7fe9f000dde0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 shutdown_connections 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe9e40afaf0 0x7fe9e40b2000 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe9fc1050b0 0x7fe9fc1a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe9fc102260 0x7fe9fc19a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 --2- 172.21.15.98:0/277388760 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe9fc101760 0x7fe9fc19a180 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 >> 172.21.15.98:0/277388760 conn(0x7fe9fc0fd380 msgr2=0x7fe9fc105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 shutdown_connections 2024-04-25T16:31:57.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.635+0000 7fea00e84640 1 -- 172.21.15.98:0/277388760 wait complete. 2024-04-25T16:31:57.646 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:31:57.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/491243857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a41043d0 msgr2=0x7fe7a4102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/491243857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a41043d0 0x7fe7a4102dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe794009900 tx=0x7fe79402a410 comp rx=0 tx=0).stop 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/491243857 shutdown_connections 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/491243857 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a4108a20 0x7fe7a410ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/491243857 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a41043d0 0x7fe7a4102dd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/491243857 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7a4103ef0 0x7fe7a41027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/491243857 >> 172.21.15.98:0/491243857 conn(0x7fe7a40fd350 msgr2=0x7fe7a40ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/491243857 shutdown_connections 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.758+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/491243857 wait complete. 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7ac2ee640 1 Processor -- start 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7ac2ee640 1 -- start start 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7ac2ee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7a4103ef0 0x7fe7a4199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7ac2ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a4108a20 0x7fe7a419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7ac2ee640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a419f1b0 0x7fe7a41a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7ac2ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a410e3e0 con 0x7fe7a4108a20 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7ac2ee640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a410e260 con 0x7fe7a419f1b0 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7ac2ee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a410e560 con 0x7fe7a4103ef0 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7a9889640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a4108a20 0x7fe7a419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7a9889640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a4108a20 0x7fe7a419a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40742/0 (socket says 172.21.15.98:40742) 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7a9889640 1 -- 172.21.15.98:0/620112982 learned_addr learned my addr 172.21.15.98:0/620112982 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.759+0000 7fe7aa08a640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7a4103ef0 0x7fe7a4199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7aa88b640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a419f1b0 0x7fe7a41a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7a9889640 1 -- 172.21.15.98:0/620112982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7a4103ef0 msgr2=0x7fe7a4199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7a9889640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7a4103ef0 0x7fe7a4199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7a9889640 1 -- 172.21.15.98:0/620112982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a419f1b0 msgr2=0x7fe7a41a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7a9889640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a419f1b0 0x7fe7a41a15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7a9889640 1 -- 172.21.15.98:0/620112982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a41a1c40 con 0x7fe7a4108a20 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7aa88b640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a419f1b0 0x7fe7a41a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7a9889640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a4108a20 0x7fe7a419a2c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe794009230 tx=0x7fe7940042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe79b7fe640 1 -- 172.21.15.98:0/620112982 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79403f030 con 0x7fe7a4108a20 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe79b7fe640 1 -- 172.21.15.98:0/620112982 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe794003c90 con 0x7fe7a4108a20 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a41a1ed0 con 0x7fe7a4108a20 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe79b7fe640 1 -- 172.21.15.98:0/620112982 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794035070 con 0x7fe7a4108a20 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.760+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a41a2330 con 0x7fe7a4108a20 2024-04-25T16:31:57.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.761+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7a41a2590 con 0x7fe7a4108a20 2024-04-25T16:31:57.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.762+0000 7fe79b7fe640 1 -- 172.21.15.98:0/620112982 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe794042030 con 0x7fe7a4108a20 2024-04-25T16:31:57.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.763+0000 7fe79b7fe640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe7900afba0 0x7fe7900b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:57.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.763+0000 7fe7aa08a640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe7900afba0 0x7fe7900b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:57.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.763+0000 7fe79b7fe640 1 -- 172.21.15.98:0/620112982 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fe7940ff920 con 0x7fe7a4108a20 2024-04-25T16:31:57.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.764+0000 7fe7aa08a640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe7900afba0 0x7fe7900b20b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe7a419b120 tx=0x7fe7a000b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:57.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.767+0000 7fe79b7fe640 1 -- 172.21.15.98:0/620112982 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe7940c81a0 con 0x7fe7a4108a20 2024-04-25T16:31:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.944+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe7a41a2920 con 0x7fe7a4108a20 2024-04-25T16:31:57.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.945+0000 7fe79b7fe640 1 -- 172.21.15.98:0/620112982 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) ==== 104+0+11 (secure 0 0 0) 0x7fe7940d0220 con 0x7fe7a4108a20 2024-04-25T16:31:57.944 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:31:57.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe7900afba0 msgr2=0x7fe7900b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe7900afba0 0x7fe7900b20b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe7a419b120 tx=0x7fe7a000b390 comp rx=0 tx=0).stop 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a4108a20 msgr2=0x7fe7a419a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a4108a20 0x7fe7a419a2c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe794009230 tx=0x7fe7940042f0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 shutdown_connections 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fe7900afba0 0x7fe7900b20b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7a419f1b0 0x7fe7a41a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fe7a4108a20 0x7fe7a419a2c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 --2- 172.21.15.98:0/620112982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe7a4103ef0 0x7fe7a4199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 >> 172.21.15.98:0/620112982 conn(0x7fe7a40fd350 msgr2=0x7fe7a4109cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 shutdown_connections 2024-04-25T16:31:57.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:57.947+0000 7fe7ac2ee640 1 -- 172.21.15.98:0/620112982 wait complete. 2024-04-25T16:31:57.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 -- 172.21.15.98:0/4025750708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4104d30 msgr2=0x7f21f4105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/4025750708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4104d30 0x7f21f4105140 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f21dc009900 tx=0x7f21dc02a410 comp rx=0 tx=0).stop 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 -- 172.21.15.98:0/4025750708 shutdown_connections 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/4025750708 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21f4109a30 0x7f21f410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/4025750708 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4104d30 0x7f21f4105140 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/4025750708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f4103af0 0x7f21f41047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 -- 172.21.15.98:0/4025750708 >> 172.21.15.98:0/4025750708 conn(0x7f21f40fece0 msgr2=0x7f21f4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 -- 172.21.15.98:0/4025750708 shutdown_connections 2024-04-25T16:31:58.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.073+0000 7f21f8e6e640 1 -- 172.21.15.98:0/4025750708 wait complete. 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f8e6e640 1 Processor -- start 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f8e6e640 1 -- start start 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f8e6e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4103af0 0x7f21f419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f8e6e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f4104d30 0x7f21f419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f8e6e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21f4109a30 0x7f21f41a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f8e6e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410f4a0 con 0x7f21f4103af0 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f8e6e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410f320 con 0x7f21f4109a30 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f8e6e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21f410f620 con 0x7f21f4104d30 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4103af0 0x7f21f419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4103af0 0x7f21f419a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40764/0 (socket says 172.21.15.98:40764) 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.074+0000 7f21f259c640 1 -- 172.21.15.98:0/3758913616 learned_addr learned my addr 172.21.15.98:0/3758913616 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21ebfff640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f4104d30 0x7f21f419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f2d9d640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21f4109a30 0x7f21f41a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f259c640 1 -- 172.21.15.98:0/3758913616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f4104d30 msgr2=0x7f21f419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f259c640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f4104d30 0x7f21f419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f259c640 1 -- 172.21.15.98:0/3758913616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21f4109a30 msgr2=0x7f21f41a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f259c640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21f4109a30 0x7f21f41a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f259c640 1 -- 172.21.15.98:0/3758913616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21f41a1bb0 con 0x7f21f4103af0 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f2d9d640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21f4109a30 0x7f21f41a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f259c640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4103af0 0x7f21f419a190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f21e400c970 tx=0x7f21e400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21eb7fe640 1 -- 172.21.15.98:0/3758913616 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e4007c70 con 0x7f21f4103af0 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21eb7fe640 1 -- 172.21.15.98:0/3758913616 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21e4007e10 con 0x7f21f4103af0 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21f41a1ea0 con 0x7f21f4103af0 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21eb7fe640 1 -- 172.21.15.98:0/3758913616 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21e40186f0 con 0x7f21f4103af0 2024-04-25T16:31:58.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.075+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21f41a2300 con 0x7f21f4103af0 2024-04-25T16:31:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.076+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21f406a190 con 0x7f21f4103af0 2024-04-25T16:31:58.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.078+0000 7f21eb7fe640 1 -- 172.21.15.98:0/3758913616 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f21e400fa90 con 0x7f21f4103af0 2024-04-25T16:31:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.078+0000 7f21eb7fe640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21e00afaf0 0x7f21e00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.078+0000 7f21ebfff640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21e00afaf0 0x7f21e00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.079+0000 7f21eb7fe640 1 -- 172.21.15.98:0/3758913616 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f21e40df1c0 con 0x7f21f4103af0 2024-04-25T16:31:58.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.079+0000 7f21ebfff640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21e00afaf0 0x7f21e00b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f21f419b230 tx=0x7f21dc000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:58.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.082+0000 7f21eb7fe640 1 -- 172.21.15.98:0/3758913616 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21e4053ff0 con 0x7f21f4103af0 2024-04-25T16:31:58.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:58 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1687442131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:31:58.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:58 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/277388760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:31:58.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:58 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/620112982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:31:58.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:58 smithi098 ceph-mon[29162]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:31:58.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.263+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f21f402dc30 con 0x7f21f4103af0 2024-04-25T16:31:58.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.263+0000 7f21eb7fe640 1 -- 172.21.15.98:0/3758913616 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) ==== 100+0+11 (secure 0 0 0) 0x7f21f406a190 con 0x7f21f4103af0 2024-04-25T16:31:58.263 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:31:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21e00afaf0 msgr2=0x7f21e00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21e00afaf0 0x7f21e00b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f21f419b230 tx=0x7f21dc000bc0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4103af0 msgr2=0x7f21f419a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4103af0 0x7f21f419a190 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f21e400c970 tx=0x7f21e400ce30 comp rx=0 tx=0).stop 2024-04-25T16:31:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 shutdown_connections 2024-04-25T16:31:58.265 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21e00afaf0 0x7f21e00b2000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21f4109a30 0x7f21f41a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21f4104d30 0x7f21f419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 --2- 172.21.15.98:0/3758913616 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21f4103af0 0x7f21f419a190 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.265+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 >> 172.21.15.98:0/3758913616 conn(0x7f21f40fece0 msgr2=0x7f21f4105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.266+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 shutdown_connections 2024-04-25T16:31:58.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.266+0000 7f21f8e6e640 1 -- 172.21.15.98:0/3758913616 wait complete. 2024-04-25T16:31:58.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:31:58.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 -- 172.21.15.98:0/1248947300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108108920 msgr2=0x7f110810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 --2- 172.21.15.98:0/1248947300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108108920 0x7f110810ad40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1100009ca0 tx=0x7f110002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 -- 172.21.15.98:0/1248947300 shutdown_connections 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 --2- 172.21.15.98:0/1248947300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108108920 0x7f110810ad40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 --2- 172.21.15.98:0/1248947300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1108103c20 0x7f1108104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 --2- 172.21.15.98:0/1248947300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11081029e0 0x7f11081036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 -- 172.21.15.98:0/1248947300 >> 172.21.15.98:0/1248947300 conn(0x7f11080fdbd0 msgr2=0x7f1108100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 -- 172.21.15.98:0/1248947300 shutdown_connections 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 -- 172.21.15.98:0/1248947300 wait complete. 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 Processor -- start 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.388+0000 7f111012d640 1 -- start start 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f111012d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11081029e0 0x7f1108199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f111012d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108103c20 0x7f1108199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f111012d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1108108920 0x7f11081a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f111012d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f110810e3f0 con 0x7f1108103c20 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f111012d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f110810e270 con 0x7f1108108920 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f111012d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f110810e570 con 0x7f11081029e0 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108103c20 0x7f1108199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108103c20 0x7f1108199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40790/0 (socket says 172.21.15.98:40790) 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 -- 172.21.15.98:0/3242498833 learned_addr learned my addr 172.21.15.98:0/3242498833 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110dec9640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11081029e0 0x7f1108199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 -- 172.21.15.98:0/3242498833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11081029e0 msgr2=0x7f1108199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11081029e0 0x7f1108199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 -- 172.21.15.98:0/3242498833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1108108920 msgr2=0x7f11081a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1108108920 0x7f11081a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 -- 172.21.15.98:0/3242498833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11081a0da0 con 0x7f1108103c20 2024-04-25T16:31:58.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f110d6c8640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108103c20 0x7f1108199850 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f10f800c890 tx=0x7f10f800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:58.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f10feffd640 1 -- 172.21.15.98:0/3242498833 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8004220 con 0x7f1108103c20 2024-04-25T16:31:58.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f10feffd640 1 -- 172.21.15.98:0/3242498833 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f10f8007e40 con 0x7f1108103c20 2024-04-25T16:31:58.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11081a1090 con 0x7f1108103c20 2024-04-25T16:31:58.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f10feffd640 1 -- 172.21.15.98:0/3242498833 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f8004a00 con 0x7f1108103c20 2024-04-25T16:31:58.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.389+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11081a1630 con 0x7f1108103c20 2024-04-25T16:31:58.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.390+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1108107730 con 0x7f1108103c20 2024-04-25T16:31:58.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.392+0000 7f10feffd640 1 -- 172.21.15.98:0/3242498833 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f10f80079b0 con 0x7f1108103c20 2024-04-25T16:31:58.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.392+0000 7f10feffd640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f10f40afb70 0x7f10f40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.392+0000 7f110dec9640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f10f40afb70 0x7f10f40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.392+0000 7f10feffd640 1 -- 172.21.15.98:0/3242498833 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f10f80df0e0 con 0x7f1108103c20 2024-04-25T16:31:58.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.393+0000 7f110dec9640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f10f40afb70 0x7f10f40b2080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f11040058c0 tx=0x7f1104005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:58.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.396+0000 7f10feffd640 1 -- 172.21.15.98:0/3242498833 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10f8014030 con 0x7f1108103c20 2024-04-25T16:31:58.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:58 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1687442131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:31:58.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:58 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/277388760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:31:58.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:58 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/620112982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:31:58.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:58 smithi203 ceph-mon[33115]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:31:58.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:58 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1687442131' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:31:58.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:58 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/277388760' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:31:58.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:58 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/620112982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:31:58.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:58 smithi173 ceph-mon[32556]: pgmap v338: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:31:58.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.574+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f110802dd90 con 0x7f1108103c20 2024-04-25T16:31:58.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.574+0000 7f10feffd640 1 -- 172.21.15.98:0/3242498833 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v83) ==== 103+0+11 (secure 0 0 0) 0x7f10f80a79e0 con 0x7f1108103c20 2024-04-25T16:31:58.574 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:31:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f10f40afb70 msgr2=0x7f10f40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f10f40afb70 0x7f10f40b2080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f11040058c0 tx=0x7f1104005850 comp rx=0 tx=0).stop 2024-04-25T16:31:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108103c20 msgr2=0x7f1108199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108103c20 0x7f1108199850 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f10f800c890 tx=0x7f10f800cd50 comp rx=0 tx=0).stop 2024-04-25T16:31:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 shutdown_connections 2024-04-25T16:31:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f10f40afb70 0x7f10f40b2080 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1108108920 0x7f11081a07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1108103c20 0x7f1108199850 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 --2- 172.21.15.98:0/3242498833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11081029e0 0x7f1108199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 >> 172.21.15.98:0/3242498833 conn(0x7f11080fdbd0 msgr2=0x7f1108104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.576+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 shutdown_connections 2024-04-25T16:31:58.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.577+0000 7f111012d640 1 -- 172.21.15.98:0/3242498833 wait complete. 2024-04-25T16:31:58.588 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 -- 172.21.15.98:0/2738211573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 msgr2=0x7f4d2810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 --2- 172.21.15.98:0/2738211573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 0x7f4d2810ad40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009900 tx=0x7f4d1802a410 comp rx=0 tx=0).stop 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 -- 172.21.15.98:0/2738211573 shutdown_connections 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 --2- 172.21.15.98:0/2738211573 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 0x7f4d2810ad40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 --2- 172.21.15.98:0/2738211573 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d28103c20 0x7f4d28104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 --2- 172.21.15.98:0/2738211573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d281029e0 0x7f4d281036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 -- 172.21.15.98:0/2738211573 >> 172.21.15.98:0/2738211573 conn(0x7f4d280fdbd0 msgr2=0x7f4d28100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 -- 172.21.15.98:0/2738211573 shutdown_connections 2024-04-25T16:31:58.696 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.696+0000 7f4d2d034640 1 -- 172.21.15.98:0/2738211573 wait complete. 2024-04-25T16:31:58.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2d034640 1 Processor -- start 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2d034640 1 -- start start 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2d034640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d281029e0 0x7f4d28075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2d034640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d28103c20 0x7f4d2819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2d034640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 0x7f4d281a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2d034640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810e3c0 con 0x7f4d28108920 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2d034640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810e240 con 0x7f4d28103c20 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2d034640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2810e540 con 0x7f4d281029e0 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 0x7f4d281a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2759e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 0x7f4d281a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40810/0 (socket says 172.21.15.98:40810) 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d2759e640 1 -- 172.21.15.98:0/689775653 learned_addr learned my addr 172.21.15.98:0/689775653 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.697+0000 7f4d26d9d640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d281029e0 0x7f4d28075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2659c640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d28103c20 0x7f4d2819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2759e640 1 -- 172.21.15.98:0/689775653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d281029e0 msgr2=0x7f4d28075860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2759e640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d281029e0 0x7f4d28075860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2759e640 1 -- 172.21.15.98:0/689775653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d28103c20 msgr2=0x7f4d2819ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2759e640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d28103c20 0x7f4d2819ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2759e640 1 -- 172.21.15.98:0/689775653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d281a1be0 con 0x7f4d28108920 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2659c640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d28103c20 0x7f4d2819ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2759e640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 0x7f4d281a13c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4d18005ab0 tx=0x7f4d180042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d1ffff640 1 -- 172.21.15.98:0/689775653 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1803f030 con 0x7f4d28108920 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d1ffff640 1 -- 172.21.15.98:0/689775653 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d18003c90 con 0x7f4d28108920 2024-04-25T16:31:58.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d281a1e70 con 0x7f4d28108920 2024-04-25T16:31:58.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d1ffff640 1 -- 172.21.15.98:0/689775653 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18035070 con 0x7f4d28108920 2024-04-25T16:31:58.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.698+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d281a2280 con 0x7f4d28108920 2024-04-25T16:31:58.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.699+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d28107730 con 0x7f4d28108920 2024-04-25T16:31:58.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.700+0000 7f4d1ffff640 1 -- 172.21.15.98:0/689775653 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4d18042030 con 0x7f4d28108920 2024-04-25T16:31:58.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.701+0000 7f4d1ffff640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4d140afaf0 0x7f4d140b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:58.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.701+0000 7f4d26d9d640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4d140afaf0 0x7f4d140b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:58.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.701+0000 7f4d1ffff640 1 -- 172.21.15.98:0/689775653 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f4d180ffa10 con 0x7f4d28108920 2024-04-25T16:31:58.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.701+0000 7f4d26d9d640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4d140afaf0 0x7f4d140b2000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4d281035e0 tx=0x7f4d20009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:58.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.705+0000 7f4d1ffff640 1 -- 172.21.15.98:0/689775653 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d180c8290 con 0x7f4d28108920 2024-04-25T16:31:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.918+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d2802db50 con 0x7f4d28108920 2024-04-25T16:31:58.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.919+0000 7f4d1ffff640 1 -- 172.21.15.98:0/689775653 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7f4d180c7b90 con 0x7f4d28108920 2024-04-25T16:31:58.919 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:58.919 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:30:38.051368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24484,24539,24457,24473,24494]},"id":1}]} 2024-04-25T16:31:58.919 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4d140afaf0 msgr2=0x7f4d140b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4d140afaf0 0x7f4d140b2000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4d281035e0 tx=0x7f4d20009680 comp rx=0 tx=0).stop 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 msgr2=0x7f4d281a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 0x7f4d281a13c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4d18005ab0 tx=0x7f4d180042f0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 shutdown_connections 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4d140afaf0 0x7f4d140b2000 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4d28108920 0x7f4d281a13c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d28103c20 0x7f4d2819ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 --2- 172.21.15.98:0/689775653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4d281029e0 0x7f4d28075860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 >> 172.21.15.98:0/689775653 conn(0x7f4d280fdbd0 msgr2=0x7f4d28104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:58.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 shutdown_connections 2024-04-25T16:31:58.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:58.921+0000 7f4d2d034640 1 -- 172.21.15.98:0/689775653 wait complete. 2024-04-25T16:31:58.933 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:31:59.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 -- 172.21.15.98:0/3419486291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724103ae0 msgr2=0x7ff7241047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 --2- 172.21.15.98:0/3419486291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724103ae0 0x7ff7241047e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff714009900 tx=0x7ff71402a410 comp rx=0 tx=0).stop 2024-04-25T16:31:59.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 -- 172.21.15.98:0/3419486291 shutdown_connections 2024-04-25T16:31:59.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 --2- 172.21.15.98:0/3419486291 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff724109a20 0x7ff72410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 --2- 172.21.15.98:0/3419486291 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff724104d20 0x7ff724105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 --2- 172.21.15.98:0/3419486291 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724103ae0 0x7ff7241047e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 -- 172.21.15.98:0/3419486291 >> 172.21.15.98:0/3419486291 conn(0x7ff7240fecd0 msgr2=0x7ff724101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 -- 172.21.15.98:0/3419486291 shutdown_connections 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.041+0000 7ff72958f640 1 -- 172.21.15.98:0/3419486291 wait complete. 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff72958f640 1 Processor -- start 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff72958f640 1 -- start start 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff72958f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff724103ae0 0x7ff72419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff72958f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff724104d20 0x7ff72419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff72958f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724109a20 0x7ff7241a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff72958f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff72410f490 con 0x7ff724109a20 2024-04-25T16:31:59.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff72958f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff72410f310 con 0x7ff724103ae0 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff72958f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff72410f610 con 0x7ff724104d20 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff7237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724109a20 0x7ff7241a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff7237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724109a20 0x7ff7241a15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40816/0 (socket says 172.21.15.98:40816) 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff7237fe640 1 -- 172.21.15.98:0/2831402038 learned_addr learned my addr 172.21.15.98:0/2831402038 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.042+0000 7ff71bfff640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff724104d20 0x7ff72419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff722ffd640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff724103ae0 0x7ff72419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff7237fe640 1 -- 172.21.15.98:0/2831402038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff724104d20 msgr2=0x7ff72419a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff7237fe640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff724104d20 0x7ff72419a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff7237fe640 1 -- 172.21.15.98:0/2831402038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff724103ae0 msgr2=0x7ff72419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff7237fe640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff724103ae0 0x7ff72419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff7237fe640 1 -- 172.21.15.98:0/2831402038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7241a1c10 con 0x7ff724109a20 2024-04-25T16:31:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff722ffd640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff724103ae0 0x7ff72419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff7237fe640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724109a20 0x7ff7241a15a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00eaa0 tx=0x7ff71c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff720ff9640 1 -- 172.21.15.98:0/2831402038 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c00cdb0 con 0x7ff724109a20 2024-04-25T16:31:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff720ff9640 1 -- 172.21.15.98:0/2831402038 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff71c005020 con 0x7ff724109a20 2024-04-25T16:31:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7241a1f00 con 0x7ff724109a20 2024-04-25T16:31:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff720ff9640 1 -- 172.21.15.98:0/2831402038 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff71c0116d0 con 0x7ff724109a20 2024-04-25T16:31:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.043+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7241a2360 con 0x7ff724109a20 2024-04-25T16:31:59.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.044+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff724108830 con 0x7ff724109a20 2024-04-25T16:31:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.045+0000 7ff720ff9640 1 -- 172.21.15.98:0/2831402038 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff71c0051d0 con 0x7ff724109a20 2024-04-25T16:31:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.046+0000 7ff720ff9640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7100afaf0 0x7ff7100b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.046+0000 7ff722ffd640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7100afaf0 0x7ff7100b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.047+0000 7ff720ff9640 1 -- 172.21.15.98:0/2831402038 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7ff71c0df4c0 con 0x7ff724109a20 2024-04-25T16:31:59.050 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.047+0000 7ff722ffd640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7100afaf0 0x7ff7100b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff7140057c0 tx=0x7ff714000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.050+0000 7ff720ff9640 1 -- 172.21.15.98:0/2831402038 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff71c016030 con 0x7ff724109a20 2024-04-25T16:31:59.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.228+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff72402dd20 con 0x7ff724109a20 2024-04-25T16:31:59.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.230+0000 7ff720ff9640 1 -- 172.21.15.98:0/2831402038 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+24609 (secure 0 0 0) 0x7ff71c0a7ce0 con 0x7ff724109a20 2024-04-25T16:31:59.229 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:59.229 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":83,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:31:08.997612+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:31:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7100afaf0 msgr2=0x7ff7100b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7100afaf0 0x7ff7100b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff7140057c0 tx=0x7ff714000bc0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724109a20 msgr2=0x7ff7241a15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724109a20 0x7ff7241a15a0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff71c00eaa0 tx=0x7ff71c00ef60 comp rx=0 tx=0).stop 2024-04-25T16:31:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 shutdown_connections 2024-04-25T16:31:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7100afaf0 0x7ff7100b2000 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff724109a20 0x7ff7241a15a0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff724104d20 0x7ff72419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 --2- 172.21.15.98:0/2831402038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff724103ae0 0x7ff72419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 >> 172.21.15.98:0/2831402038 conn(0x7ff7240fecd0 msgr2=0x7ff724105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 shutdown_connections 2024-04-25T16:31:59.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.232+0000 7ff72958f640 1 -- 172.21.15.98:0/2831402038 wait complete. 2024-04-25T16:31:59.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:59 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3758913616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:31:59.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:59 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3242498833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:31:59.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:31:59 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/689775653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:31:59.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:31:59.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.352+0000 7efc1b16c640 1 -- 172.21.15.98:0/3074861455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14109a40 msgr2=0x7efc1410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.352+0000 7efc1b16c640 1 --2- 172.21.15.98:0/3074861455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14109a40 0x7efc1410be60 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009ca0 tx=0x7efc0c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.352+0000 7efc1b16c640 1 -- 172.21.15.98:0/3074861455 shutdown_connections 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.352+0000 7efc1b16c640 1 --2- 172.21.15.98:0/3074861455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14109a40 0x7efc1410be60 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.352+0000 7efc1b16c640 1 --2- 172.21.15.98:0/3074861455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc14104d40 0x7efc14105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.352+0000 7efc1b16c640 1 --2- 172.21.15.98:0/3074861455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc14103b00 0x7efc14104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 -- 172.21.15.98:0/3074861455 >> 172.21.15.98:0/3074861455 conn(0x7efc140fed10 msgr2=0x7efc141011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 -- 172.21.15.98:0/3074861455 shutdown_connections 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 -- 172.21.15.98:0/3074861455 wait complete. 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 Processor -- start 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 -- start start 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14103b00 0x7efc1419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc14104d40 0x7efc1419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc14109a40 0x7efc141a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc1410f530 con 0x7efc14103b00 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc1410f3b0 con 0x7efc14104d40 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc1b16c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc1410f6b0 con 0x7efc14109a40 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.353+0000 7efc18f08640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14103b00 0x7efc1419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc18f08640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14103b00 0x7efc1419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40834/0 (socket says 172.21.15.98:40834) 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc18f08640 1 -- 172.21.15.98:0/736801208 learned_addr learned my addr 172.21.15.98:0/736801208 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc19709640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc14109a40 0x7efc141a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc13fff640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc14104d40 0x7efc1419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc18f08640 1 -- 172.21.15.98:0/736801208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc14109a40 msgr2=0x7efc141a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc18f08640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc14109a40 0x7efc141a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc18f08640 1 -- 172.21.15.98:0/736801208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc14104d40 msgr2=0x7efc1419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc18f08640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc14104d40 0x7efc1419a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc18f08640 1 -- 172.21.15.98:0/736801208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc141a1c30 con 0x7efc14103b00 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc19709640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc14109a40 0x7efc141a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:59.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc18f08640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14103b00 0x7efc1419a0f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efc0800d740 tx=0x7efc0800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc11ffb640 1 -- 172.21.15.98:0/736801208 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08017070 con 0x7efc14103b00 2024-04-25T16:31:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc11ffb640 1 -- 172.21.15.98:0/736801208 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc0800de50 con 0x7efc14103b00 2024-04-25T16:31:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc141a1f20 con 0x7efc14103b00 2024-04-25T16:31:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc11ffb640 1 -- 172.21.15.98:0/736801208 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc08003bb0 con 0x7efc14103b00 2024-04-25T16:31:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.354+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc141a2380 con 0x7efc14103b00 2024-04-25T16:31:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.356+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc1406a190 con 0x7efc14103b00 2024-04-25T16:31:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.357+0000 7efc11ffb640 1 -- 172.21.15.98:0/736801208 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7efc08003d50 con 0x7efc14103b00 2024-04-25T16:31:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.357+0000 7efc11ffb640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc040afaf0 0x7efc040b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.358+0000 7efc13fff640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc040afaf0 0x7efc040b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.358+0000 7efc11ffb640 1 -- 172.21.15.98:0/736801208 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7efc080dedf0 con 0x7efc14103b00 2024-04-25T16:31:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.358+0000 7efc13fff640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc040afaf0 0x7efc040b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efc1419b190 tx=0x7efc00005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.361+0000 7efc11ffb640 1 -- 172.21.15.98:0/736801208 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc080a7670 con 0x7efc14103b00 2024-04-25T16:31:59.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:59 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3758913616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:31:59.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:59 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3242498833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:31:59.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:31:59 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/689775653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:31:59.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:59 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3758913616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:31:59.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:59 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3242498833' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:31:59.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:31:59 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/689775653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:31:59.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.576+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc1402dc10 con 0x7efc14103b00 2024-04-25T16:31:59.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.577+0000 7efc11ffb640 1 -- 172.21.15.98:0/736801208 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7efc080a6f70 con 0x7efc14103b00 2024-04-25T16:31:59.577 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:31:59.577 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:30:38.051368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24484,24539,24457,24473,24494]},"id":1}]} 2024-04-25T16:31:59.577 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc040afaf0 msgr2=0x7efc040b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc040afaf0 0x7efc040b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efc1419b190 tx=0x7efc00005f90 comp rx=0 tx=0).stop 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14103b00 msgr2=0x7efc1419a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14103b00 0x7efc1419a0f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efc0800d740 tx=0x7efc0800dc00 comp rx=0 tx=0).stop 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 shutdown_connections 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc040afaf0 0x7efc040b2000 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc14109a40 0x7efc141a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc14104d40 0x7efc1419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 --2- 172.21.15.98:0/736801208 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc14103b00 0x7efc1419a0f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 >> 172.21.15.98:0/736801208 conn(0x7efc140fed10 msgr2=0x7efc14105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 shutdown_connections 2024-04-25T16:31:59.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.579+0000 7efc1b16c640 1 -- 172.21.15.98:0/736801208 wait complete. 2024-04-25T16:31:59.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T16:30:09.342794+0000', 'modified': '2024-04-25T16:30:38.051368+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24539, 'mds_1': 24494, 'mds_2': 24484, 'mds_3': 24473, 'mds_4': 24457}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14634': {'gid': 14634, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3308601909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3308601909}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3308601909}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14646': {'gid': 14646, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6841/373690634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6840', 'nonce': 373690634}, {'type': 'v1', 'addr': '172.21.15.98:6841', 'nonce': 373690634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24415': {'gid': 24415, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/1945729206', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 1945729206}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 1945729206}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6833/3037734321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 3037734321}, {'type': 'v1', 'addr': '172.21.15.173:6833', 'nonce': 3037734321}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/2230353021', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 2230353021}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 2230353021}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6837/998442662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 998442662}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 998442662}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.173:6835/960684117', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 960684117}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 960684117}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.203:6839/2982027693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 2982027693}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 2982027693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.173:6837/311453164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 311453164}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 311453164}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.173:6839/2050448663', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2050448663}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2050448663}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24539, 'qdb_cluster': [24484, 24539, 24457, 24473, 24494]} 2024-04-25T16:31:59.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T16:31:59.591 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T16:31:59.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 -- 172.21.15.98:0/3055584410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0109a30 msgr2=0x7f21d010be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 --2- 172.21.15.98:0/3055584410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0109a30 0x7f21d010be50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f21c8009ca0 tx=0x7f21c802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 -- 172.21.15.98:0/3055584410 shutdown_connections 2024-04-25T16:31:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 --2- 172.21.15.98:0/3055584410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0109a30 0x7f21d010be50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 --2- 172.21.15.98:0/3055584410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21d0104d30 0x7f21d0105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 --2- 172.21.15.98:0/3055584410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d0103af0 0x7f21d01047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 -- 172.21.15.98:0/3055584410 >> 172.21.15.98:0/3055584410 conn(0x7f21d00fece0 msgr2=0x7f21d0101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 -- 172.21.15.98:0/3055584410 shutdown_connections 2024-04-25T16:31:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 -- 172.21.15.98:0/3055584410 wait complete. 2024-04-25T16:31:59.703 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 Processor -- start 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.700+0000 7f21d62d8640 1 -- start start 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d62d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0103af0 0x7f21d019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d62d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d0104d30 0x7f21d019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d62d8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21d0109a30 0x7f21d01a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d62d8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f21d010f500 con 0x7f21d0103af0 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d62d8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f21d010f380 con 0x7f21d0109a30 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d62d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f21d010f680 con 0x7f21d0104d30 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0103af0 0x7f21d019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0103af0 0x7f21d019a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40860/0 (socket says 172.21.15.98:40860) 2024-04-25T16:31:59.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 -- 172.21.15.98:0/1218327460 learned_addr learned my addr 172.21.15.98:0/1218327460 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d4875640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21d0109a30 0x7f21d01a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 -- 172.21.15.98:0/1218327460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d0104d30 msgr2=0x7f21d019a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d0104d30 0x7f21d019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 -- 172.21.15.98:0/1218327460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21d0109a30 msgr2=0x7f21d01a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21d0109a30 0x7f21d01a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 -- 172.21.15.98:0/1218327460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d01a1b60 con 0x7f21d0103af0 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cffff640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0103af0 0x7f21d019a0d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f21c000c970 tx=0x7f21c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cd7fa640 1 -- 172.21.15.98:0/1218327460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0007c70 con 0x7f21d0103af0 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d62d8640 1 -- 172.21.15.98:0/1218327460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d01a1e50 con 0x7f21d0103af0 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cd7fa640 1 -- 172.21.15.98:0/1218327460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21c0007e10 con 0x7f21d0103af0 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21cd7fa640 1 -- 172.21.15.98:0/1218327460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c00186f0 con 0x7f21d0103af0 2024-04-25T16:31:59.705 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.701+0000 7f21d62d8640 1 -- 172.21.15.98:0/1218327460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d01a2310 con 0x7f21d0103af0 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21dd187740 1 Processor -- start 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21dd187740 1 -- start start 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21dd187740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599c2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21dd187740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb595f83b0 0x55eb599c28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21dd187740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21cd7fa640 1 -- 172.21.15.98:0/1218327460 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f21c000f450 con 0x7f21d0103af0 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21dd187740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb59927740 con 0x55eb59600590 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21dd187740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eb598c33a0 con 0x55eb59621270 2024-04-25T16:31:59.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.704+0000 7f21dd187740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eb59923700 con 0x55eb595f83b0 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40874/0 (socket says 172.21.15.98:40874) 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 -- 172.21.15.98:0/2976140054 learned_addr learned my addr 172.21.15.98:0/2976140054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cffff640 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb595f83b0 0x55eb599c28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21d4875640 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 -- 172.21.15.98:0/2976140054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb595f83b0 msgr2=0x55eb599c28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb595f83b0 0x55eb599c28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 -- 172.21.15.98:0/2976140054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 msgr2=0x55eb599c2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599c2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 -- 172.21.15.98:0/2976140054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb598df750 con 0x55eb59600590 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21d4875640 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599c2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cd7fa640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21b80afaf0 0x7f21b80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cf7fe640 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599c97f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f21c400a9c0 tx=0x7f21c400ae80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de19dd326fae3c68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21d4875640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21b80afaf0 0x7f21b80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21b67fc640 1 -- 172.21.15.98:0/2976140054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c400cde0 con 0x55eb59600590 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21cd7fa640 1 -- 172.21.15.98:0/1218327460 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f21c00df080 con 0x7f21d0103af0 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21b67fc640 1 -- 172.21.15.98:0/2976140054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21c400f8e0 con 0x55eb59600590 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.705+0000 7f21b67fc640 1 -- 172.21.15.98:0/2976140054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4018900 con 0x55eb59600590 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 -- 172.21.15.98:0/2976140054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 msgr2=0x55eb599c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21d4875640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21b80afaf0 0x7f21b80b2000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f21c8031920 tx=0x7f21c8009910 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599c97f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f21c400a9c0 tx=0x7f21c400ae80 comp rx=0 tx=0).stop 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 -- 172.21.15.98:0/2976140054 shutdown_connections 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599c97f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb595f83b0 0x55eb599c28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 --2- 172.21.15.98:0/2976140054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599c2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 -- 172.21.15.98:0/2976140054 >> 172.21.15.98:0/2976140054 conn(0x55eb596153c0 msgr2=0x55eb599bdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 -- 172.21.15.98:0/2976140054 shutdown_connections 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.706+0000 7f21dd187740 1 -- 172.21.15.98:0/2976140054 wait complete. 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 1 -- start start 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 1 Processor -- start 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599cc830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59621270 0x55eb599d3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb599ca240 con 0x55eb59600590 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eb598aefe0 con 0x55eb595f83b0 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eb599ca3c0 con 0x55eb59621270 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21dd187740 10 client.0 _ll_register_callbacks cb 0x55eb598c5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21cf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21cf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40880/0 (socket says 172.21.15.98:40880) 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21cf7fe640 1 -- 172.21.15.98:0/238319864 learned_addr learned my addr 172.21.15.98:0/238319864 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21cffff640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21d4875640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59621270 0x55eb599d3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21cf7fe640 1 -- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59621270 msgr2=0x55eb599d3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21cf7fe640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59621270 0x55eb599d3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.707+0000 7f21cf7fe640 1 -- 172.21.15.98:0/238319864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 msgr2=0x55eb599cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21cf7fe640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21cf7fe640 1 -- 172.21.15.98:0/238319864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb596392d0 con 0x55eb59600590 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21d4875640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59621270 0x55eb599d3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21cffff640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21cf7fe640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599cc830 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f21c400a990 tx=0x7f21c400f680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4dcbcc89c6ce859 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21af7fe640 1 -- 172.21.15.98:0/238319864 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c4029060 con 0x55eb59600590 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21af7fe640 1 -- 172.21.15.98:0/238319864 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f21c4018640 con 0x55eb59600590 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 1 -- 172.21.15.98:0/238319864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21c400f8e0 con 0x55eb59600590 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21af7fe640 1 -- 172.21.15.98:0/238319864 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21c4030050 con 0x55eb59600590 2024-04-25T16:31:59.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 10 client.15234 fetch_fsmap learned FSMap version 28 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 1 -- 172.21.15.98:0/238319864 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb597036b0 con 0x55eb59600590 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21af7fe640 1 -- 172.21.15.98:0/238319864 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21c400f8e0 con 0x55eb59600590 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21af7fe640 1 -- 172.21.15.98:0/238319864 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f21c40236d0 con 0x55eb59600590 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 10 client.15234 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 10 client.15234 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 20 client.15234 populate_metadata read hostname 'smithi098' 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55eb595dfe40 0x55eb599d9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 4 client.15234 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.708+0000 7f21dd187740 1 -- 172.21.15.98:0/238319864 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb59935350 con 0x55eb595dfe40 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.709+0000 7f21cffff640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55eb595dfe40 0x55eb599d9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.709+0000 7f21af7fe640 10 client.15234 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.709+0000 7f21cffff640 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55eb595dfe40 0x55eb599d9ca0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ec43d1ab0011a8 server_cookie=e31cd158e578882d in_seq=0 out_seq=0 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21af7fe640 1 -- 172.21.15.98:0/238319864 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eb59935350 con 0x55eb595dfe40 2024-04-25T16:31:59.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21af7fe640 10 client.15234 handle_command_reply: tid=0 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 client.15234 shutdown 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 -- 172.21.15.98:0/238319864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 msgr2=0x55eb599cc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599cc830 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f21c400a990 tx=0x7f21c400f680 comp rx=0 tx=0).stop 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 -- 172.21.15.98:0/238319864 shutdown_connections 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55eb595dfe40 0x55eb599d9ca0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59621270 0x55eb599d3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59600590 0x55eb599cc830 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21af7fe640 0 client.15234 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 --2- 172.21.15.98:0/238319864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.710+0000 7f21dd187740 1 -- 172.21.15.98:0/238319864 >> 172.21.15.98:0/238319864 conn(0x55eb596153c0 msgr2=0x55eb599bcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.711+0000 7f21dd187740 1 -- 172.21.15.98:0/238319864 shutdown_connections 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.711+0000 7f21dd187740 1 -- 172.21.15.98:0/238319864 wait complete. 2024-04-25T16:31:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.711+0000 7f21dd187740 20 client.15234 trim_cache size 0 max 16384 2024-04-25T16:31:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21dd187740 1 Processor -- start 2024-04-25T16:31:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21dd187740 1 -- start start 2024-04-25T16:31:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21dd187740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb595f83b0 0x55eb599c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21dd187740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 0x55eb599bddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21dd187740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21dd187740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb599d7fe0 con 0x55eb595f83b0 2024-04-25T16:31:59.734 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21dd187740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eb599d86b0 con 0x55eb59621270 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21dd187740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eb599d8d80 con 0x55eb59600590 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb595f83b0 0x55eb599c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb595f83b0 0x55eb599c6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40892/0 (socket says 172.21.15.98:40892) 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 -- 172.21.15.98:0/2339822100 learned_addr learned my addr 172.21.15.98:0/2339822100 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cf7fe640 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 0x55eb599bddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21d4875640 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 -- 172.21.15.98:0/2339822100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 msgr2=0x55eb599bddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 0x55eb599bddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 -- 172.21.15.98:0/2339822100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 msgr2=0x55eb599d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 -- 172.21.15.98:0/2339822100 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb599c9cf0 con 0x55eb595f83b0 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21d4875640 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:59.735 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.729+0000 7f21cffff640 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb595f83b0 0x55eb599c6d50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f21c00a3280 tx=0x7f21c00a6c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=704810ad555f5612 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21b5ffb640 1 -- 172.21.15.98:0/2339822100 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c00a6d10 con 0x55eb595f83b0 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21b5ffb640 1 -- 172.21.15.98:0/2339822100 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21c00a7900 con 0x55eb595f83b0 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21b5ffb640 1 -- 172.21.15.98:0/2339822100 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c00dc9a0 con 0x55eb595f83b0 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 -- 172.21.15.98:0/2339822100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb595f83b0 msgr2=0x55eb599c6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb595f83b0 0x55eb599c6d50 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f21c00a3280 tx=0x7f21c00a6c10 comp rx=0 tx=0).stop 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 -- 172.21.15.98:0/2339822100 shutdown_connections 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb59621270 0x55eb599d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 0x55eb599bddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 --2- 172.21.15.98:0/2339822100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb595f83b0 0x55eb599c6d50 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 -- 172.21.15.98:0/2339822100 >> 172.21.15.98:0/2339822100 conn(0x55eb596153c0 msgr2=0x55eb599bd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 -- 172.21.15.98:0/2339822100 shutdown_connections 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.730+0000 7f21dd187740 1 -- 172.21.15.98:0/2339822100 wait complete. 2024-04-25T16:31:59.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 1 -- start start 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 1 Processor -- start 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 0x55eb599cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59621270 0x55eb599d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eb599d9b20 con 0x55eb59621270 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eb599d9450 con 0x55eb595f83b0 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eb599da1f0 con 0x55eb59600590 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21dd187740 10 client.0 _ll_register_callbacks cb 0x7f21c00e2a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21d4875640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59621270 0x55eb599d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21d4875640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59621270 0x55eb599d24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40900/0 (socket says 172.21.15.98:40900) 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21d4875640 1 -- 172.21.15.98:0/1986997865 learned_addr learned my addr 172.21.15.98:0/1986997865 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.731+0000 7f21cffff640 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21cf7fe640 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 0x55eb599cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21d4875640 1 -- 172.21.15.98:0/1986997865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 msgr2=0x55eb599cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21d4875640 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 0x55eb599cf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21d4875640 1 -- 172.21.15.98:0/1986997865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 msgr2=0x55eb599cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21d4875640 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21d4875640 1 -- 172.21.15.98:0/1986997865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb599d2aa0 con 0x55eb59621270 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21d4875640 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59621270 0x55eb599d24e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f21c8035000 tx=0x7f21c80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d735e68a4a90f0c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21af7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21af7fe640 1 -- 172.21.15.98:0/1986997865 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c80305c0 con 0x55eb59621270 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21af7fe640 1 -- 172.21.15.98:0/1986997865 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f21c8030760 con 0x55eb59621270 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21dd187740 1 -- 172.21.15.98:0/1986997865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f21c00a7900 con 0x55eb59621270 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21af7fe640 1 -- 172.21.15.98:0/1986997865 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f21c8030960 con 0x55eb59621270 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21dd187740 10 client.15240 fetch_fsmap learned FSMap version 28 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21dd187740 1 -- 172.21.15.98:0/1986997865 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eb599d3c90 con 0x55eb59621270 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21af7fe640 1 -- 172.21.15.98:0/1986997865 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f21c00a7900 con 0x55eb59621270 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21af7fe640 1 -- 172.21.15.98:0/1986997865 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f21c8043040 con 0x55eb59621270 2024-04-25T16:31:59.738 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.732+0000 7f21dd187740 10 client.15240 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21dd187740 10 client.15240 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21dd187740 20 client.15240 populate_metadata read hostname 'smithi098' 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21dd187740 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55eb595b5280 0x55eb59a45cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21dd187740 4 client.15240 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21dd187740 1 -- 172.21.15.98:0/1986997865 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "status"}) -- 0x55eb5992f1c0 con 0x55eb595b5280 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21cffff640 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55eb595b5280 0x55eb59a45cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21cffff640 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55eb595b5280 0x55eb59a45cf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c9c013821b848903 server_cookie=75e6239558bd8ae5 in_seq=0 out_seq=0 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21af7fe640 10 client.15240 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21af7fe640 1 -- 172.21.15.98:0/1986997865 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55eb5992f1c0 con 0x55eb595b5280 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21af7fe640 10 client.15240 handle_command_reply: tid=0 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.733+0000 7f21dd187740 1 client.15240 shutdown 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 -- 172.21.15.98:0/1986997865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59621270 msgr2=0x55eb599d24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59621270 0x55eb599d24e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f21c8035000 tx=0x7f21c80095c0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 -- 172.21.15.98:0/1986997865 shutdown_connections 2024-04-25T16:31:59.739 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55eb595b5280 0x55eb59a45cf0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eb59621270 0x55eb599d24e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eb59600590 0x55eb599cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21af7fe640 0 client.15240 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 --2- 172.21.15.98:0/1986997865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eb595f83b0 0x55eb599cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 -- 172.21.15.98:0/1986997865 >> 172.21.15.98:0/1986997865 conn(0x55eb596153c0 msgr2=0x55eb599bc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 -- 172.21.15.98:0/1986997865 shutdown_connections 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 1 -- 172.21.15.98:0/1986997865 wait complete. 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21dd187740 20 client.15240 trim_cache size 0 max 16384 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:31:59.740 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 4, 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24457, 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 82.701610453000001, 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 83, 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 129.14233040600001 2024-04-25T16:31:59.741 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:31:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 -- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21b80afaf0 msgr2=0x7f21b80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21b80afaf0 0x7f21b80b2000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f21c8031920 tx=0x7f21c8009910 comp rx=0 tx=0).stop 2024-04-25T16:31:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 -- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0103af0 msgr2=0x7f21d019a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0103af0 0x7f21d019a0d0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f21c000c970 tx=0x7f21c000ce30 comp rx=0 tx=0).stop 2024-04-25T16:31:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 -- 172.21.15.98:0/1218327460 shutdown_connections 2024-04-25T16:31:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f21b80afaf0 0x7f21b80b2000 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21d0109a30 0x7f21d01a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21d0104d30 0x7f21d019a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 --2- 172.21.15.98:0/1218327460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f21d0103af0 0x7f21d019a0d0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 -- 172.21.15.98:0/1218327460 >> 172.21.15.98:0/1218327460 conn(0x7f21d00fece0 msgr2=0x7f21d0105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 -- 172.21.15.98:0/1218327460 shutdown_connections 2024-04-25T16:31:59.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.734+0000 7f21d62d8640 1 -- 172.21.15.98:0/1218327460 wait complete. 2024-04-25T16:31:59.745 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-25T16:31:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 -- 172.21.15.98:0/1918527745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b01045f0 msgr2=0x7f23b0106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 --2- 172.21.15.98:0/1918527745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b01045f0 0x7f23b0106a10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009900 tx=0x7f23a402a410 comp rx=0 tx=0).stop 2024-04-25T16:31:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 -- 172.21.15.98:0/1918527745 shutdown_connections 2024-04-25T16:31:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 --2- 172.21.15.98:0/1918527745 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b01098b0 0x7f23b010bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 --2- 172.21.15.98:0/1918527745 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23b0106f50 0x7f23b0109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 --2- 172.21.15.98:0/1918527745 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b01045f0 0x7f23b0106a10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 -- 172.21.15.98:0/1918527745 >> 172.21.15.98:0/1918527745 conn(0x7f23b00fdbf0 msgr2=0x7f23b01000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 -- 172.21.15.98:0/1918527745 shutdown_connections 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.853+0000 7f23b6467640 1 -- 172.21.15.98:0/1918527745 wait complete. 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23b6467640 1 Processor -- start 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23b6467640 1 -- start start 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23b6467640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b01045f0 0x7f23b019a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23b6467640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b0106f50 0x7f23b019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23b6467640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23b01098b0 0x7f23b01a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23b6467640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f23b010f490 con 0x7f23b0106f50 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23b6467640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f23b010f310 con 0x7f23b01098b0 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23b6467640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f23b010f610 con 0x7f23b01045f0 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b0106f50 0x7f23b019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b0106f50 0x7f23b019a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40924/0 (socket says 172.21.15.98:40924) 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.854+0000 7f23af7fe640 1 -- 172.21.15.98:0/2984196759 learned_addr learned my addr 172.21.15.98:0/2984196759 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23affff640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b01045f0 0x7f23b019a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23b4a04640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23b01098b0 0x7f23b01a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23af7fe640 1 -- 172.21.15.98:0/2984196759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b01045f0 msgr2=0x7f23b019a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23af7fe640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b01045f0 0x7f23b019a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23af7fe640 1 -- 172.21.15.98:0/2984196759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23b01098b0 msgr2=0x7f23b01a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23af7fe640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23b01098b0 0x7f23b01a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23af7fe640 1 -- 172.21.15.98:0/2984196759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b01a1bb0 con 0x7f23b0106f50 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23af7fe640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b0106f50 0x7f23b019a660 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f23a000e9f0 tx=0x7f23a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23ad7fa640 1 -- 172.21.15.98:0/2984196759 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a000cd80 con 0x7f23b0106f50 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23ad7fa640 1 -- 172.21.15.98:0/2984196759 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23a00108c0 con 0x7f23b0106f50 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23b6467640 1 -- 172.21.15.98:0/2984196759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b01a1ea0 con 0x7f23b0106f50 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23ad7fa640 1 -- 172.21.15.98:0/2984196759 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a00187c0 con 0x7f23b0106f50 2024-04-25T16:31:59.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.855+0000 7f23b6467640 1 -- 172.21.15.98:0/2984196759 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b01a2360 con 0x7f23b0106f50 2024-04-25T16:31:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.857+0000 7f23bd3f9740 1 Processor -- start 2024-04-25T16:31:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.857+0000 7f23ad7fa640 1 -- 172.21.15.98:0/2984196759 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f23a0010430 con 0x7f23b0106f50 2024-04-25T16:31:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23bd3f9740 1 -- start start 2024-04-25T16:31:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a74370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a748f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.864 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560be59d9740 con 0x560be56b2590 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560be59753a0 con 0x560be56d3270 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560be59d5700 con 0x560be56aa3b0 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40932/0 (socket says 172.21.15.98:40932) 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23af7fe640 1 -- 172.21.15.98:0/2655574805 learned_addr learned my addr 172.21.15.98:0/2655574805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23b4a04640 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a74370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23affff640 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23af7fe640 1 -- 172.21.15.98:0/2655574805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 msgr2=0x560be5a748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23af7fe640 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a748f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23ad7fa640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f23980afb70 0x7f23980b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23af7fe640 1 -- 172.21.15.98:0/2655574805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 msgr2=0x560be5a74370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23af7fe640 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a74370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23af7fe640 1 -- 172.21.15.98:0/2655574805 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560be5991750 con 0x560be56b2590 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.858+0000 7f23b4a04640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f23980afb70 0x7f23980b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.859+0000 7f23af7fe640 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7b7f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f23a00b6100 tx=0x7f23a00b8b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233c8631282a849b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.859+0000 7f23ad7fa640 1 -- 172.21.15.98:0/2984196759 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f23a00df330 con 0x7f23b0106f50 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.859+0000 7f23967fc640 1 -- 172.21.15.98:0/2655574805 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a0026030 con 0x560be56b2590 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.859+0000 7f23967fc640 1 -- 172.21.15.98:0/2655574805 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23a00b6130 con 0x560be56b2590 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.859+0000 7f23967fc640 1 -- 172.21.15.98:0/2655574805 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a00b6e60 con 0x560be56b2590 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 -- 172.21.15.98:0/2655574805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 msgr2=0x560be5a7b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7b7f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f23a00b6100 tx=0x7f23a00b8b60 comp rx=0 tx=0).stop 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23b4a04640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f23980afb70 0x7f23980b2080 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x560be5a75540 tx=0x7f23a8004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 -- 172.21.15.98:0/2655574805 shutdown_connections 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7b7f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/2655574805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a74370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 -- 172.21.15.98:0/2655574805 >> 172.21.15.98:0/2655574805 conn(0x560be56c73c0 msgr2=0x560be5a6fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 -- 172.21.15.98:0/2655574805 shutdown_connections 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 -- 172.21.15.98:0/2655574805 wait complete. 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 -- start start 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.860+0000 7f23bd3f9740 1 Processor -- start 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a7e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a856c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560be5a7c240 con 0x560be56b2590 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560be5960fe0 con 0x560be56d3270 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560be5a7c3c0 con 0x560be56aa3b0 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.867 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 10 client.0 _ll_register_callbacks cb 0x560be5977150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40942/0 (socket says 172.21.15.98:40942) 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 -- 172.21.15.98:0/3937151407 learned_addr learned my addr 172.21.15.98:0/3937151407 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23affff640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a7e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23b4a04640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a856c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 -- 172.21.15.98:0/3937151407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 msgr2=0x560be5a7e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a7e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 -- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 msgr2=0x560be5a856c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a856c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 -- 172.21.15.98:0/3937151407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560be56eb2d0 con 0x560be56b2590 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23b4a04640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a856c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23affff640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a7e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23af7fe640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7e760 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f23a00a6f70 tx=0x7f23a00e2be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856a9ec9a0155724 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f238f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f238f7fe640 1 -- 172.21.15.98:0/3937151407 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a0010720 con 0x560be56b2590 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f238f7fe640 1 -- 172.21.15.98:0/3937151407 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f23a00ab400 con 0x560be56b2590 2024-04-25T16:31:59.868 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3937151407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560be59e7350 con 0x560be56b2590 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f238f7fe640 1 -- 172.21.15.98:0/3937151407 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f23a00a7280 con 0x560be56b2590 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 10 client.15252 fetch_fsmap learned FSMap version 28 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.861+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3937151407 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560be57b56b0 con 0x560be56b2590 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f238f7fe640 1 -- 172.21.15.98:0/3937151407 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x560be59e7350 con 0x560be56b2590 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f238f7fe640 1 -- 172.21.15.98:0/3937151407 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f23a00ab0d0 con 0x560be56b2590 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f23bd3f9740 10 client.15252 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f23bd3f9740 10 client.15252 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f23bd3f9740 20 client.15252 populate_metadata read hostname 'smithi098' 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x560be5691e40 0x560be5a8b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f23bd3f9740 4 client.15252 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3937151407 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560be59e11c0 con 0x560be5691e40 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.862+0000 7f23affff640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x560be5691e40 0x560be5a8b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.863+0000 7f238f7fe640 10 client.15252 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.863+0000 7f23affff640 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x560be5691e40 0x560be5a8b990 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e60b9775eddee837 server_cookie=7dc8b4fd90a414b1 in_seq=0 out_seq=0 2024-04-25T16:31:59.869 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.863+0000 7f238f7fe640 1 -- 172.21.15.98:0/3937151407 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560be59e11c0 con 0x560be5691e40 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.863+0000 7f238f7fe640 10 client.15252 handle_command_reply: tid=0 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 client.15252 shutdown 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3937151407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 msgr2=0x560be5a7e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7e760 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f23a00a6f70 tx=0x7f23a00e2be0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3937151407 shutdown_connections 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x560be5691e40 0x560be5a8b990 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56d3270 0x560be5a856c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56b2590 0x560be5a7e760 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3937151407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56aa3b0 0x560be5a7e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3937151407 >> 172.21.15.98:0/3937151407 conn(0x560be56c73c0 msgr2=0x560be5a6ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f238f7fe640 0 client.15252 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3937151407 shutdown_connections 2024-04-25T16:31:59.870 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3937151407 wait complete. 2024-04-25T16:31:59.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.864+0000 7f23bd3f9740 20 client.15252 trim_cache size 0 max 16384 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.882+0000 7f23bd3f9740 1 Processor -- start 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.882+0000 7f23bd3f9740 1 -- start start 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.882+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a7d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.882+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 0x560be5a7da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.882+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 0x560be5a855e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.882+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560be5a89cd0 con 0x560be56aa3b0 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.882+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560be5a8a3a0 con 0x560be56b2590 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.882+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560be5a8aa70 con 0x560be56d3270 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a7d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a7d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40944/0 (socket says 172.21.15.98:40944) 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 -- 172.21.15.98:0/274075061 learned_addr learned my addr 172.21.15.98:0/274075061 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23b4a04640 1 --2- 172.21.15.98:0/274075061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 0x560be5a855e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 -- 172.21.15.98:0/274075061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 msgr2=0x560be5a855e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 --2- 172.21.15.98:0/274075061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 0x560be5a855e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 -- 172.21.15.98:0/274075061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 msgr2=0x560be5a7da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 --2- 172.21.15.98:0/274075061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 0x560be5a7da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 -- 172.21.15.98:0/274075061 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560be5a7ec30 con 0x560be56aa3b0 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f23affff640 1 --2- 172.21.15.98:0/274075061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a7d550 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f23b00757f0 tx=0x7f23a4004220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99515e58f4cf3989 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f2395ffb640 1 -- 172.21.15.98:0/274075061 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a404a070 con 0x560be56aa3b0 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f2395ffb640 1 -- 172.21.15.98:0/274075061 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23a4031cb0 con 0x560be56aa3b0 2024-04-25T16:31:59.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.883+0000 7f2395ffb640 1 -- 172.21.15.98:0/274075061 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4035040 con 0x560be56aa3b0 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 -- 172.21.15.98:0/274075061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 msgr2=0x560be5a7d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/274075061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a7d550 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f23b00757f0 tx=0x7f23a4004220 comp rx=0 tx=0).stop 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 -- 172.21.15.98:0/274075061 shutdown_connections 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/274075061 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 0x560be5a855e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/274075061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 0x560be5a7da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/274075061 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a7d550 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 -- 172.21.15.98:0/274075061 >> 172.21.15.98:0/274075061 conn(0x560be56c73c0 msgr2=0x560be5a78690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 -- 172.21.15.98:0/274075061 shutdown_connections 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 -- 172.21.15.98:0/274075061 wait complete. 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.884+0000 7f23bd3f9740 1 -- start start 2024-04-25T16:31:59.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23bd3f9740 1 Processor -- start 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a85dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 0x560be5a829a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23bd3f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 0x560be5a82fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560be5a8b810 con 0x560be56aa3b0 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560be5a8b140 con 0x560be56b2590 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23bd3f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560be5a8bee0 con 0x560be56d3270 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23bd3f9740 10 client.0 _ll_register_callbacks cb 0x7f23a8007520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a85dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a85dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40960/0 (socket says 172.21.15.98:40960) 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 -- 172.21.15.98:0/3068121868 learned_addr learned my addr 172.21.15.98:0/3068121868 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23af7fe640 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 0x560be5a829a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23b4a04640 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 0x560be5a82fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.890 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 -- 172.21.15.98:0/3068121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 msgr2=0x560be5a82fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 0x560be5a82fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 -- 172.21.15.98:0/3068121868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 msgr2=0x560be5a829a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 0x560be5a829a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 -- 172.21.15.98:0/3068121868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560be5af2800 con 0x560be56aa3b0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f23affff640 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a85dd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f23a4048040 tx=0x7f23a4031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee06337bfa2b783 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.885+0000 7f238f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f238f7fe640 1 -- 172.21.15.98:0/3068121868 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a404a040 con 0x560be56aa3b0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f238f7fe640 1 -- 172.21.15.98:0/3068121868 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f23a40457f0 con 0x560be56aa3b0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3068121868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23a4031cb0 con 0x560be56aa3b0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f238f7fe640 1 -- 172.21.15.98:0/3068121868 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f23a405b050 con 0x560be56aa3b0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 10 client.15258 fetch_fsmap learned FSMap version 28 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3068121868 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560be5af3a30 con 0x560be56aa3b0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f238f7fe640 1 -- 172.21.15.98:0/3068121868 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23a4031cb0 con 0x560be56aa3b0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f238f7fe640 1 -- 172.21.15.98:0/3068121868 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f23a4056030 con 0x560be56aa3b0 2024-04-25T16:31:59.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 10 client.15258 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 10 client.15258 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 20 client.15258 populate_metadata read hostname 'smithi098' 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x560be5667280 0x560be5af6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 4 client.15258 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.886+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3068121868 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "status"}) -- 0x560be595c9a0 con 0x560be5667280 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23af7fe640 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x560be5667280 0x560be5af6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f238f7fe640 10 client.15258 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23af7fe640 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x560be5667280 0x560be5af6ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=836cc173a470583d server_cookie=2b2449fa0a58ffd1 in_seq=0 out_seq=0 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f238f7fe640 1 -- 172.21.15.98:0/3068121868 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x560be595c9a0 con 0x560be5667280 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f238f7fe640 10 client.15258 handle_command_reply: tid=0 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 client.15258 shutdown 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3068121868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 msgr2=0x560be5a85dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a85dd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f23a4048040 tx=0x7f23a4031e60 comp rx=0 tx=0).stop 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3068121868 shutdown_connections 2024-04-25T16:31:59.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x560be5667280 0x560be5af6ed0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560be56d3270 0x560be5a82fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560be56b2590 0x560be5a829a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f238f7fe640 0 client.15258 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 --2- 172.21.15.98:0/3068121868 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560be56aa3b0 0x560be5a85dd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.887+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3068121868 >> 172.21.15.98:0/3068121868 conn(0x560be56c73c0 msgr2=0x560be5a7d550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3068121868 shutdown_connections 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23bd3f9740 1 -- 172.21.15.98:0/3068121868 wait complete. 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23bd3f9740 20 client.15258 trim_cache size 0 max 16384 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:31:59.893 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 3, 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24473, 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 83.861839544999995, 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 83, 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 134.907233096 2024-04-25T16:31:59.894 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:31:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 -- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f23980afb70 msgr2=0x7f23980b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f23980afb70 0x7f23980b2080 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x560be5a75540 tx=0x7f23a8004a40 comp rx=0 tx=0).stop 2024-04-25T16:31:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 -- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b0106f50 msgr2=0x7f23b019a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:31:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b0106f50 0x7f23b019a660 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f23a000e9f0 tx=0x7f23a000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 -- 172.21.15.98:0/2984196759 shutdown_connections 2024-04-25T16:31:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f23980afb70 0x7f23980b2080 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23b01098b0 0x7f23b01a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f23b0106f50 0x7f23b019a660 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 --2- 172.21.15.98:0/2984196759 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b01045f0 0x7f23b019a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:31:59.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 -- 172.21.15.98:0/2984196759 >> 172.21.15.98:0/2984196759 conn(0x7f23b00fdbf0 msgr2=0x7f23b0106610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:31:59.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 -- 172.21.15.98:0/2984196759 shutdown_connections 2024-04-25T16:31:59.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:31:59.888+0000 7f23b6467640 1 -- 172.21.15.98:0/2984196759 wait complete. 2024-04-25T16:31:59.899 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T16:32:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 -- 172.21.15.98:0/231325543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 msgr2=0x7f44841037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 --2- 172.21.15.98:0/231325543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 0x7f44841037e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f4474009900 tx=0x7f447402a410 comp rx=0 tx=0).stop 2024-04-25T16:32:00.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 -- 172.21.15.98:0/231325543 shutdown_connections 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 --2- 172.21.15.98:0/231325543 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44841080d0 0x7f448410a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 --2- 172.21.15.98:0/231325543 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 0x7f44841037e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 --2- 172.21.15.98:0/231325543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4484102190 0x7f4484102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 -- 172.21.15.98:0/231325543 >> 172.21.15.98:0/231325543 conn(0x7f44840fd380 msgr2=0x7f44840ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 -- 172.21.15.98:0/231325543 shutdown_connections 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.008+0000 7f448b6f2640 1 -- 172.21.15.98:0/231325543 wait complete. 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f448b6f2640 1 Processor -- start 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f448b6f2640 1 -- start start 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f448b6f2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4484102190 0x7f448419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f448b6f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 0x7f448419a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f448b6f2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44841080d0 0x7f44841a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f448b6f2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f448410f5c0 con 0x7f44841033d0 2024-04-25T16:32:00.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f448b6f2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f448410f440 con 0x7f4484102190 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f448b6f2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f448410f740 con 0x7f44841080d0 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f4488c8d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 0x7f448419a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f4488c8d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 0x7f448419a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40980/0 (socket says 172.21.15.98:40980) 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.009+0000 7f4488c8d640 1 -- 172.21.15.98:0/2049156232 learned_addr learned my addr 172.21.15.98:0/2049156232 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f4489c8f640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44841080d0 0x7f44841a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f448948e640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4484102190 0x7f448419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f4488c8d640 1 -- 172.21.15.98:0/2049156232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44841080d0 msgr2=0x7f44841a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f4488c8d640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44841080d0 0x7f44841a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f4488c8d640 1 -- 172.21.15.98:0/2049156232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4484102190 msgr2=0x7f448419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f4488c8d640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4484102190 0x7f448419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f4488c8d640 1 -- 172.21.15.98:0/2049156232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44841a1db0 con 0x7f44841033d0 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f4489c8f640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44841080d0 0x7f44841a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f4488c8d640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 0x7f448419a7b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4474009230 tx=0x7f44740042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f447a7fc640 1 -- 172.21.15.98:0/2049156232 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447403f030 con 0x7f44841033d0 2024-04-25T16:32:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f447a7fc640 1 -- 172.21.15.98:0/2049156232 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4474003c90 con 0x7f44841033d0 2024-04-25T16:32:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f447a7fc640 1 -- 172.21.15.98:0/2049156232 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474035070 con 0x7f44841033d0 2024-04-25T16:32:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f448b6f2640 1 -- 172.21.15.98:0/2049156232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44841a2040 con 0x7f44841033d0 2024-04-25T16:32:00.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.010+0000 7f448b6f2640 1 -- 172.21.15.98:0/2049156232 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44841a25b0 con 0x7f44841033d0 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f44925a4740 1 Processor -- start 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f447a7fc640 1 -- 172.21.15.98:0/2049156232 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4474042030 con 0x7f44841033d0 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f44925a4740 1 -- start start 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f44925a4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2962d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f44925a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2962d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f44925a4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f296347f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f44925a4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f29592740 con 0x562f2928c270 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f44925a4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562f2952e3a0 con 0x562f2926b590 2024-04-25T16:32:00.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f44925a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562f2958e700 con 0x562f292633b0 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f4489c8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2962d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f4489c8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2962d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40982/0 (socket says 172.21.15.98:40982) 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f4489c8f640 1 -- 172.21.15.98:0/3779463880 learned_addr learned my addr 172.21.15.98:0/3779463880 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.013+0000 7f4488c8d640 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f296347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f448948e640 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2962d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f447a7fc640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44700afaf0 0x7f44700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f4489c8f640 1 -- 172.21.15.98:0/3779463880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 msgr2=0x562f2962d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f4489c8f640 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2962d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f4489c8f640 1 -- 172.21.15.98:0/3779463880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 msgr2=0x562f296347f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f4489c8f640 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f296347f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f4489c8f640 1 -- 172.21.15.98:0/3779463880 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f2954a750 con 0x562f2928c270 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f448948e640 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2962d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f447a7fc640 1 -- 172.21.15.98:0/2049156232 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f44740ff740 con 0x7f44841033d0 2024-04-25T16:32:00.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f4489c8f640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44700afaf0 0x7f44700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f4489c8f640 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2962d370 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f447c00c450 tx=0x7f447c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3ca9ac1e5ece9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f446f7fe640 1 -- 172.21.15.98:0/3779463880 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c01f070 con 0x562f2928c270 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f446f7fe640 1 -- 172.21.15.98:0/3779463880 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f447c019de0 con 0x562f2928c270 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.014+0000 7f446f7fe640 1 -- 172.21.15.98:0/3779463880 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c00a8d0 con 0x562f2928c270 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 -- 172.21.15.98:0/3779463880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 msgr2=0x562f2962d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f4489c8f640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44700afaf0 0x7f44700b2000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f447c01c000 tx=0x7f447c01f000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2962d370 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f447c00c450 tx=0x7f447c00c910 comp rx=0 tx=0).stop 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 -- 172.21.15.98:0/3779463880 shutdown_connections 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f296347f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2962d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 --2- 172.21.15.98:0/3779463880 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2962d370 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 -- 172.21.15.98:0/3779463880 >> 172.21.15.98:0/3779463880 conn(0x562f292803c0 msgr2=0x562f29628d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 -- 172.21.15.98:0/3779463880 shutdown_connections 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 -- 172.21.15.98:0/3779463880 wait complete. 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.015+0000 7f44925a4740 1 -- start start 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f44925a4740 1 Processor -- start 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f44925a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f296372c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f44925a4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f29637800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f44925a4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f44925a4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f29635240 con 0x562f2928c270 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f44925a4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562f29519fe0 con 0x562f2926b590 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f44925a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562f296353c0 con 0x562f292633b0 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f44925a4740 10 client.0 _ll_register_callbacks cb 0x562f29530150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4489c8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4489c8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40992/0 (socket says 172.21.15.98:40992) 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4489c8f640 1 -- 172.21.15.98:0/511656165 learned_addr learned my addr 172.21.15.98:0/511656165 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4488c8d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f29637800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f448948e640 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f296372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4489c8f640 1 -- 172.21.15.98:0/511656165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 msgr2=0x562f296372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4489c8f640 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f296372c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4489c8f640 1 -- 172.21.15.98:0/511656165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 msgr2=0x562f29637800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4489c8f640 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f29637800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.016+0000 7f4489c8f640 1 -- 172.21.15.98:0/511656165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f292a42d0 con 0x562f2928c270 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f4489c8f640 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963e760 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f447c01bfd0 tx=0x7f447c0093d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff2fd8993fe5e19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f446cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f446cff9640 1 -- 172.21.15.98:0/511656165 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c029070 con 0x562f2928c270 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f446cff9640 1 -- 172.21.15.98:0/511656165 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f447c00ade0 con 0x562f2928c270 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 1 -- 172.21.15.98:0/511656165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f447c019de0 con 0x562f2928c270 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f446cff9640 1 -- 172.21.15.98:0/511656165 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f447c035050 con 0x562f2928c270 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 10 client.15270 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 1 -- 172.21.15.98:0/511656165 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562f2936e6b0 con 0x562f2928c270 2024-04-25T16:32:00.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f446cff9640 1 -- 172.21.15.98:0/511656165 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f447c019de0 con 0x562f2928c270 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f446cff9640 1 -- 172.21.15.98:0/511656165 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f447c015050 con 0x562f2928c270 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 10 client.15270 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 10 client.15270 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 20 client.15270 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562f2924ae40 0x562f29644c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 4 client.15270 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.017+0000 7f44925a4740 1 -- 172.21.15.98:0/511656165 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f295a0350 con 0x562f2924ae40 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.018+0000 7f448948e640 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562f2924ae40 0x562f29644c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.018+0000 7f446cff9640 10 client.15270 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.018+0000 7f448948e640 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562f2924ae40 0x562f29644c20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9e38a49bb89dd6ef server_cookie=a5190c2e8a7c077b in_seq=0 out_seq=0 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f446cff9640 1 -- 172.21.15.98:0/511656165 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562f295a0350 con 0x562f2924ae40 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f446cff9640 10 client.15270 handle_command_reply: tid=0 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 client.15270 shutdown 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 -- 172.21.15.98:0/511656165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 msgr2=0x562f2963e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963e760 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f447c01bfd0 tx=0x7f447c0093d0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 -- 172.21.15.98:0/511656165 shutdown_connections 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562f2924ae40 0x562f29644c20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963e760 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f29637800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f446cff9640 0 client.15270 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 --2- 172.21.15.98:0/511656165 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f296372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 -- 172.21.15.98:0/511656165 >> 172.21.15.98:0/511656165 conn(0x562f292803c0 msgr2=0x562f29627d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 -- 172.21.15.98:0/511656165 shutdown_connections 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 1 -- 172.21.15.98:0/511656165 wait complete. 2024-04-25T16:32:00.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.019+0000 7f44925a4740 20 client.15270 trim_cache size 0 max 16384 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.038+0000 7f44925a4740 1 Processor -- start 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.038+0000 7f44925a4740 1 -- start start 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.038+0000 7f44925a4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f292633b0 0x562f296309c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.038+0000 7f44925a4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f29637560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f44925a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f2928c270 0x562f2962c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f44925a4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f29642f60 con 0x562f292633b0 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f44925a4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562f29643630 con 0x562f2926b590 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f44925a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562f29643d00 con 0x562f2928c270 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f292633b0 0x562f296309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f292633b0 0x562f296309c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:40994/0 (socket says 172.21.15.98:40994) 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 -- 172.21.15.98:0/1304493631 learned_addr learned my addr 172.21.15.98:0/1304493631 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f4488c8d640 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f29637560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 -- 172.21.15.98:0/1304493631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f2928c270 msgr2=0x562f2962c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f4489c8f640 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f2928c270 0x562f2962c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f2928c270 0x562f2962c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 -- 172.21.15.98:0/1304493631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 msgr2=0x562f29637560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f29637560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 -- 172.21.15.98:0/1304493631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f29630790 con 0x562f292633b0 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f448948e640 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f292633b0 0x562f296309c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4480000c00 tx=0x7f4480016430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d24cb62a0b994f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f446effd640 1 -- 172.21.15.98:0/1304493631 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480016bf0 con 0x562f292633b0 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f446effd640 1 -- 172.21.15.98:0/1304493631 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4480016d90 con 0x562f292633b0 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.039+0000 7f446effd640 1 -- 172.21.15.98:0/1304493631 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480024820 con 0x562f292633b0 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 -- 172.21.15.98:0/1304493631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f292633b0 msgr2=0x562f296309c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f292633b0 0x562f296309c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4480000c00 tx=0x7f4480016430 comp rx=0 tx=0).stop 2024-04-25T16:32:00.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 -- 172.21.15.98:0/1304493631 shutdown_connections 2024-04-25T16:32:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f2928c270 0x562f2962c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f29637560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 --2- 172.21.15.98:0/1304493631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f292633b0 0x562f296309c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 -- 172.21.15.98:0/1304493631 >> 172.21.15.98:0/1304493631 conn(0x562f292803c0 msgr2=0x562f2962bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 -- 172.21.15.98:0/1304493631 shutdown_connections 2024-04-25T16:32:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.040+0000 7f44925a4740 1 -- 172.21.15.98:0/1304493631 wait complete. 2024-04-25T16:32:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 1 -- start start 2024-04-25T16:32:00.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 1 Processor -- start 2024-04-25T16:32:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2963bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f2963c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562f29644aa0 con 0x562f2928c270 2024-04-25T16:32:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562f296443d0 con 0x562f2926b590 2024-04-25T16:32:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562f29645170 con 0x562f292633b0 2024-04-25T16:32:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f44925a4740 10 client.0 _ll_register_callbacks cb 0x7f447c01d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:00.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f4489c8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f4489c8f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41010/0 (socket says 172.21.15.98:41010) 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f4489c8f640 1 -- 172.21.15.98:0/3430644778 learned_addr learned my addr 172.21.15.98:0/3430644778 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f4488c8d640 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f2963c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.041+0000 7f448948e640 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2963bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f4489c8f640 1 -- 172.21.15.98:0/3430644778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 msgr2=0x562f2963bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f4489c8f640 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2963bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f4489c8f640 1 -- 172.21.15.98:0/3430644778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 msgr2=0x562f2963c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f4489c8f640 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f2963c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f4489c8f640 1 -- 172.21.15.98:0/3430644778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f29632110 con 0x562f2928c270 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f4489c8f640 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963cfd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f447c00c950 tx=0x7f447c009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82503540b6ab28a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f446cff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f446cff9640 1 -- 172.21.15.98:0/3430644778 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c032020 con 0x562f2928c270 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f446cff9640 1 -- 172.21.15.98:0/3430644778 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f447c00a400 con 0x562f2928c270 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 1 -- 172.21.15.98:0/3430644778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4480016d90 con 0x562f2928c270 2024-04-25T16:32:00.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f446cff9640 1 -- 172.21.15.98:0/3430644778 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f447c029050 con 0x562f2928c270 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 10 client.15276 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 1 -- 172.21.15.98:0/3430644778 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562f296addf0 con 0x562f2928c270 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f446cff9640 1 -- 172.21.15.98:0/3430644778 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4480016d90 con 0x562f2928c270 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f446cff9640 1 -- 172.21.15.98:0/3430644778 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f447c015040 con 0x562f2928c270 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 10 client.15276 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 10 client.15276 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 20 client.15276 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562f29220280 0x562f296b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 4 client.15276 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.042+0000 7f44925a4740 1 -- 172.21.15.98:0/3430644778 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "status"}) -- 0x562f2959a1c0 con 0x562f29220280 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.043+0000 7f448948e640 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562f29220280 0x562f296b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.043+0000 7f448948e640 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562f29220280 0x562f296b1660 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fd43100d869d6252 server_cookie=2b4676b5111d9f02 in_seq=0 out_seq=0 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.043+0000 7f446cff9640 10 client.15276 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f446cff9640 1 -- 172.21.15.98:0/3430644778 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x562f2959a1c0 con 0x562f29220280 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f446cff9640 10 client.15276 handle_command_reply: tid=0 2024-04-25T16:32:00.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 client.15276 shutdown 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 -- 172.21.15.98:0/3430644778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 msgr2=0x562f2963cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963cfd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f447c00c950 tx=0x7f447c009ce0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 -- 172.21.15.98:0/3430644778 shutdown_connections 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562f29220280 0x562f296b1660 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562f2928c270 0x562f2963cfd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f446cff9640 0 client.15276 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562f2926b590 0x562f2963c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 --2- 172.21.15.98:0/3430644778 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562f292633b0 0x562f2963bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 -- 172.21.15.98:0/3430644778 >> 172.21.15.98:0/3430644778 conn(0x562f292803c0 msgr2=0x562f2962c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 -- 172.21.15.98:0/3430644778 shutdown_connections 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 1 -- 172.21.15.98:0/3430644778 wait complete. 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f44925a4740 20 client.15276 trim_cache size 0 max 16384 2024-04-25T16:32:00.049 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24484, 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 85.026625330000002, 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 83, 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 123.68958632 2024-04-25T16:32:00.050 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 -- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44700afaf0 msgr2=0x7f44700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44700afaf0 0x7f44700b2000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f447c01c000 tx=0x7f447c01f000 comp rx=0 tx=0).stop 2024-04-25T16:32:00.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 -- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 msgr2=0x7f448419a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 0x7f448419a7b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4474009230 tx=0x7f44740042f0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 -- 172.21.15.98:0/2049156232 shutdown_connections 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44700afaf0 0x7f44700b2000 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44841080d0 0x7f44841a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44841033d0 0x7f448419a7b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 --2- 172.21.15.98:0/2049156232 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4484102190 0x7f448419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.044+0000 7f448b6f2640 1 -- 172.21.15.98:0/2049156232 >> 172.21.15.98:0/2049156232 conn(0x7f44840fd380 msgr2=0x7f44841040b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.045+0000 7f448b6f2640 1 -- 172.21.15.98:0/2049156232 shutdown_connections 2024-04-25T16:32:00.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.045+0000 7f448b6f2640 1 -- 172.21.15.98:0/2049156232 wait complete. 2024-04-25T16:32:00.055 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 -- 172.21.15.98:0/4033167509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0108940 msgr2=0x7fdaf010ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/4033167509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0108940 0x7fdaf010ad60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009ca0 tx=0x7fdae802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 -- 172.21.15.98:0/4033167509 shutdown_connections 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/4033167509 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0108940 0x7fdaf010ad60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/4033167509 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaf0103c40 0x7fdaf0104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/4033167509 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaf0102a00 0x7fdaf0103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 -- 172.21.15.98:0/4033167509 >> 172.21.15.98:0/4033167509 conn(0x7fdaf00fdbf0 msgr2=0x7fdaf01000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 -- 172.21.15.98:0/4033167509 shutdown_connections 2024-04-25T16:32:00.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.163+0000 7fdaf4c51640 1 -- 172.21.15.98:0/4033167509 wait complete. 2024-04-25T16:32:00.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaf4c51640 1 Processor -- start 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaf4c51640 1 -- start start 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaf4c51640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0102a00 0x7fdaf0199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaf4c51640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaf0103c40 0x7fdaf01995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaf4c51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaf0108940 0x7fdaf01a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaf4c51640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf010e430 con 0x7fdaf0102a00 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaf4c51640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf010e2b0 con 0x7fdaf0103c40 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaf4c51640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdaf010e5b0 con 0x7fdaf0108940 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaee59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0102a00 0x7fdaf0199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaee59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0102a00 0x7fdaf0199060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41032/0 (socket says 172.21.15.98:41032) 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.164+0000 7fdaee59c640 1 -- 172.21.15.98:0/3801915231 learned_addr learned my addr 172.21.15.98:0/3801915231 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdae7fff640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaf0103c40 0x7fdaf01995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.166 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaf0108940 0x7fdaf01a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaee59c640 1 -- 172.21.15.98:0/3801915231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaf0108940 msgr2=0x7fdaf01a0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaee59c640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaf0108940 0x7fdaf01a0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaee59c640 1 -- 172.21.15.98:0/3801915231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaf0103c40 msgr2=0x7fdaf01995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaee59c640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaf0103c40 0x7fdaf01995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaee59c640 1 -- 172.21.15.98:0/3801915231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaf01a0af0 con 0x7fdaf0102a00 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaee59c640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0102a00 0x7fdaf0199060 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdae000c970 tx=0x7fdae000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdae77fe640 1 -- 172.21.15.98:0/3801915231 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae0007c70 con 0x7fdaf0102a00 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdae77fe640 1 -- 172.21.15.98:0/3801915231 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdae0007e10 con 0x7fdaf0102a00 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaf4c51640 1 -- 172.21.15.98:0/3801915231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaf01a0de0 con 0x7fdaf0102a00 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdae77fe640 1 -- 172.21.15.98:0/3801915231 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae00186f0 con 0x7fdaf0102a00 2024-04-25T16:32:00.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.165+0000 7fdaf4c51640 1 -- 172.21.15.98:0/3801915231 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaf01a1240 con 0x7fdaf0102a00 2024-04-25T16:32:00.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.168+0000 7fdafbbe3740 1 Processor -- start 2024-04-25T16:32:00.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.168+0000 7fdafbbe3740 1 -- start start 2024-04-25T16:32:00.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05376e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e05376e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537757f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0536d3740 con 0x55e0533cd270 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e05366f3a0 con 0x55e0533ac590 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e0536cf700 con 0x55e0533a43b0 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05376e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05376e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41048/0 (socket says 172.21.15.98:41048) 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 -- 172.21.15.98:0/3182991817 learned_addr learned my addr 172.21.15.98:0/3182991817 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdae77fe640 1 -- 172.21.15.98:0/3801915231 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdae00213f0 con 0x7fdaf0102a00 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdae7fff640 1 --2- 172.21.15.98:0/3182991817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537757f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 -- 172.21.15.98:0/3182991817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 msgr2=0x55e05376e8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/3182991817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e05376e8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 -- 172.21.15.98:0/3182991817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 msgr2=0x55e0537757f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/3182991817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537757f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 -- 172.21.15.98:0/3182991817 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e05368b750 con 0x55e0533cd270 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/3182991817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05376e370 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdae8031500 tx=0x7fdae8009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4b72a9737f4491 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdae4ff9640 1 -- 172.21.15.98:0/3182991817 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8031a80 con 0x55e0533cd270 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdae4ff9640 1 -- 172.21.15.98:0/3182991817 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdae8036040 con 0x55e0533cd270 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.169+0000 7fdae4ff9640 1 -- 172.21.15.98:0/3182991817 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae80044a0 con 0x55e0533cd270 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdae77fe640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdadc0afaf0 0x7fdadc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdaee59c640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdadc0afaf0 0x7fdadc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdae77fe640 1 -- 172.21.15.98:0/3801915231 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fdae00defc0 con 0x7fdaf0102a00 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdafbbe3740 1 -- 172.21.15.98:0/3182991817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 msgr2=0x55e05376e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdaee59c640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdadc0afaf0 0x7fdadc0b2000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdae00dcad0 tx=0x7fdae0075c90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/3182991817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05376e370 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdae8031500 tx=0x7fdae8009a20 comp rx=0 tx=0).stop 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdafbbe3740 1 -- 172.21.15.98:0/3182991817 shutdown_connections 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/3182991817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537757f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/3182991817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e05376e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/3182991817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05376e370 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.170+0000 7fdafbbe3740 1 -- 172.21.15.98:0/3182991817 >> 172.21.15.98:0/3182991817 conn(0x55e0533c13c0 msgr2=0x55e053769d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 -- 172.21.15.98:0/3182991817 shutdown_connections 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 -- 172.21.15.98:0/3182991817 wait complete. 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 -- start start 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 Processor -- start 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e053778270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537787b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0537761e0 con 0x55e0533cd270 2024-04-25T16:32:00.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e05365afe0 con 0x55e0533ac590 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e053776360 con 0x55e0533a43b0 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdafbbe3740 10 client.0 _ll_register_callbacks cb 0x55e053671150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdaeed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdaeed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41054/0 (socket says 172.21.15.98:41054) 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdaeed9d640 1 -- 172.21.15.98:0/404137693 learned_addr learned my addr 172.21.15.98:0/404137693 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.171+0000 7fdae7fff640 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537787b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e053778270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdae7fff640 1 -- 172.21.15.98:0/404137693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 msgr2=0x55e053778270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdae7fff640 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e053778270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdae7fff640 1 -- 172.21.15.98:0/404137693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 msgr2=0x55e05377f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdae7fff640 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdae7fff640 1 -- 172.21.15.98:0/404137693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0533e52d0 con 0x55e0533ac590 2024-04-25T16:32:00.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdae7fff640 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537787b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdaf019a100 tx=0x7fdad800bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a4f98f7d6497c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdad5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdad5ffb640 1 -- 172.21.15.98:0/404137693 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad801b470 con 0x55e0533ac590 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdafbbe3740 1 -- 172.21.15.98:0/404137693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae8036040 con 0x55e0533ac590 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdad5ffb640 1 -- 172.21.15.98:0/404137693 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fdad801b610 con 0x55e0533ac590 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdad5ffb640 1 -- 172.21.15.98:0/404137693 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdad8024050 con 0x55e0533ac590 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdafbbe3740 10 client.25010 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.172+0000 7fdafbbe3740 1 -- 172.21.15.98:0/404137693 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e0534af6b0 con 0x55e0533ac590 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdad5ffb640 1 -- 172.21.15.98:0/404137693 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdae8036040 con 0x55e0533ac590 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdad5ffb640 1 -- 172.21.15.98:0/404137693 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdad80128a0 con 0x55e0533ac590 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdafbbe3740 10 client.25010 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdafbbe3740 10 client.25010 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T16:32:00.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdafbbe3740 20 client.25010 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55e05338be40 0x55e0537860a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdafbbe3740 4 client.25010 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdafbbe3740 1 -- 172.21.15.98:0/404137693 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e0536e1350 con 0x55e05338be40 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.173+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55e05338be40 0x55e0537860a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.174+0000 7fdad5ffb640 10 client.25010 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.174+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55e05338be40 0x55e0537860a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=684f3df5339dfb0f server_cookie=ee4627e3c6b4b8ce in_seq=0 out_seq=0 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.174+0000 7fdad5ffb640 1 -- 172.21.15.98:0/404137693 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e0536e1350 con 0x55e05338be40 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.174+0000 7fdad5ffb640 10 client.25010 handle_command_reply: tid=0 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.174+0000 7fdafbbe3740 1 client.25010 shutdown 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 -- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 msgr2=0x55e0537787b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537787b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdaf019a100 tx=0x7fdad800bda0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 -- 172.21.15.98:0/404137693 shutdown_connections 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55e05338be40 0x55e0537860a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e0537787b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/404137693 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e053778270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdad5ffb640 0 client.25010 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:32:00.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 -- 172.21.15.98:0/404137693 >> 172.21.15.98:0/404137693 conn(0x55e0533c13c0 msgr2=0x55e053768d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 -- 172.21.15.98:0/404137693 shutdown_connections 2024-04-25T16:32:00.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 1 -- 172.21.15.98:0/404137693 wait complete. 2024-04-25T16:32:00.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.175+0000 7fdafbbe3740 20 client.25010 trim_cache size 0 max 16384 2024-04-25T16:32:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdafbbe3740 1 Processor -- start 2024-04-25T16:32:00.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdafbbe3740 1 -- start start 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533a43b0 0x55e053764a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533cd270 0x55e0537696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e0537843e0 con 0x55e0533a43b0 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e053784ab0 con 0x55e0533ac590 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e053785180 con 0x55e0533cd270 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533a43b0 0x55e053764a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533a43b0 0x55e053764a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41062/0 (socket says 172.21.15.98:41062) 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 -- 172.21.15.98:0/2521938664 learned_addr learned my addr 172.21.15.98:0/2521938664 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdae7fff640 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533cd270 0x55e0537696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 -- 172.21.15.98:0/2521938664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533cd270 msgr2=0x55e0537696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533cd270 0x55e0537696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 -- 172.21.15.98:0/2521938664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 msgr2=0x55e05377f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.194+0000 7fdaeed9d640 1 -- 172.21.15.98:0/2521938664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e05376f170 con 0x55e0533a43b0 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.195+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533a43b0 0x55e053764a20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdae802a900 tx=0x7fdae80095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac29635ff1d39789 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.195+0000 7fdad7fff640 1 -- 172.21.15.98:0/2521938664 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8003c20 con 0x55e0533a43b0 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.195+0000 7fdad7fff640 1 -- 172.21.15.98:0/2521938664 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdae8036040 con 0x55e0533a43b0 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.195+0000 7fdad7fff640 1 -- 172.21.15.98:0/2521938664 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8030890 con 0x55e0533a43b0 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2521938664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533a43b0 msgr2=0x55e053764a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533a43b0 0x55e053764a20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdae802a900 tx=0x7fdae80095e0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2521938664 shutdown_connections 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533cd270 0x55e0537696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2521938664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533a43b0 0x55e053764a20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2521938664 >> 172.21.15.98:0/2521938664 conn(0x55e0533c13c0 msgr2=0x55e0537768b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2521938664 shutdown_connections 2024-04-25T16:32:00.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2521938664 wait complete. 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- start start 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 Processor -- start 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e05377a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e053785f20 con 0x55e0533cd270 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e053785850 con 0x55e0533ac590 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e0537865f0 con 0x55e0533a43b0 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdafbbe3740 10 client.0 _ll_register_callbacks cb 0x7fdad8004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdaeed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdaeed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41074/0 (socket says 172.21.15.98:41074) 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.196+0000 7fdaeed9d640 1 -- 172.21.15.98:0/2650423704 learned_addr learned my addr 172.21.15.98:0/2650423704 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e05377a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdae7fff640 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdaeed9d640 1 -- 172.21.15.98:0/2650423704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 msgr2=0x55e05377a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e05377a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdaeed9d640 1 -- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 msgr2=0x55e05377ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdaeed9d640 1 -- 172.21.15.98:0/2650423704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e0537730f0 con 0x55e0533cd270 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdae7fff640 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdaeed9d640 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377dc50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009960 tx=0x7fdae80319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a202d7ff495ca015 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdad5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdad5ffb640 1 -- 172.21.15.98:0/2650423704 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdae8030830 con 0x55e0533cd270 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdad5ffb640 1 -- 172.21.15.98:0/2650423704 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fdae80309d0 con 0x55e0533cd270 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2650423704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdae8036040 con 0x55e0533cd270 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdad5ffb640 1 -- 172.21.15.98:0/2650423704 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdae8030bd0 con 0x55e0533cd270 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdafbbe3740 10 client.15294 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:00.203 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.197+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2650423704 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e05377f2c0 con 0x55e0533cd270 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdad5ffb640 1 -- 172.21.15.98:0/2650423704 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdae8036040 con 0x55e0533cd270 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdad5ffb640 1 -- 172.21.15.98:0/2650423704 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fdae8050070 con 0x55e0533cd270 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdafbbe3740 10 client.15294 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdafbbe3740 10 client.15294 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdafbbe3740 20 client.15294 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55e053361280 0x55e0537ef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdafbbe3740 4 client.15294 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2650423704 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "status"}) -- 0x55e0536db1c0 con 0x55e053361280 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdae7fff640 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55e053361280 0x55e0537ef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdad5ffb640 10 client.15294 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.198+0000 7fdae7fff640 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55e053361280 0x55e0537ef7a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ca819244d750f728 server_cookie=37e1e87ec146ee55 in_seq=0 out_seq=0 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdad5ffb640 1 -- 172.21.15.98:0/2650423704 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55e0536db1c0 con 0x55e053361280 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdad5ffb640 10 client.15294 handle_command_reply: tid=0 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 client.15294 shutdown 2024-04-25T16:32:00.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2650423704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 msgr2=0x55e05377dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377dc50 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fdae8009960 tx=0x7fdae80319a0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2650423704 shutdown_connections 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55e053361280 0x55e0537ef7a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e0533cd270 0x55e05377dc50 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdad5ffb640 0 client.15294 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e0533ac590 0x55e05377ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 --2- 172.21.15.98:0/2650423704 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e0533a43b0 0x55e05377a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2650423704 >> 172.21.15.98:0/2650423704 conn(0x55e0533c13c0 msgr2=0x55e0537696b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2650423704 shutdown_connections 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 1 -- 172.21.15.98:0/2650423704 wait complete. 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdafbbe3740 20 client.15294 trim_cache size 0 max 16384 2024-04-25T16:32:00.205 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24494, 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 86.192924050000002, 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 83, 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:32:00.206 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 126.721828997 2024-04-25T16:32:00.207 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:00.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdaf4c51640 1 -- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdadc0afaf0 msgr2=0x7fdadc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdadc0afaf0 0x7fdadc0b2000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fdae00dcad0 tx=0x7fdae0075c90 comp rx=0 tx=0).stop 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdaf4c51640 1 -- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0102a00 msgr2=0x7fdaf0199060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.199+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0102a00 0x7fdaf0199060 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdae000c970 tx=0x7fdae000ce30 comp rx=0 tx=0).stop 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.200+0000 7fdaf4c51640 1 -- 172.21.15.98:0/3801915231 shutdown_connections 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.200+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdadc0afaf0 0x7fdadc0b2000 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.200+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdaf0108940 0x7fdaf01a0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.200+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaf0103c40 0x7fdaf01995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.200+0000 7fdaf4c51640 1 --2- 172.21.15.98:0/3801915231 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdaf0102a00 0x7fdaf0199060 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.200+0000 7fdaf4c51640 1 -- 172.21.15.98:0/3801915231 >> 172.21.15.98:0/3801915231 conn(0x7fdaf00fdbf0 msgr2=0x7fdaf0104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.200+0000 7fdaf4c51640 1 -- 172.21.15.98:0/3801915231 shutdown_connections 2024-04-25T16:32:00.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.200+0000 7fdaf4c51640 1 -- 172.21.15.98:0/3801915231 wait complete. 2024-04-25T16:32:00.209 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:00 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2831402038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:00.209 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:00 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/736801208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:00.209 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:00 smithi098 ceph-mon[29162]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:32:00.210 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T16:32:00.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 -- 172.21.15.98:0/3009885703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654109a20 msgr2=0x7fa65410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/3009885703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654109a20 0x7fa65410be40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa64c009ca0 tx=0x7fa64c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 -- 172.21.15.98:0/3009885703 shutdown_connections 2024-04-25T16:32:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/3009885703 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654109a20 0x7fa65410be40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/3009885703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa654104d20 0x7fa654105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/3009885703 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa654103ae0 0x7fa6541047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 -- 172.21.15.98:0/3009885703 >> 172.21.15.98:0/3009885703 conn(0x7fa6540fecd0 msgr2=0x7fa654101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 -- 172.21.15.98:0/3009885703 shutdown_connections 2024-04-25T16:32:00.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.324+0000 7fa65bbb7640 1 -- 172.21.15.98:0/3009885703 wait complete. 2024-04-25T16:32:00.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65bbb7640 1 Processor -- start 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65bbb7640 1 -- start start 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65bbb7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654103ae0 0x7fa65419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65bbb7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa654104d20 0x7fa65419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65bbb7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa654109a20 0x7fa6541a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65bbb7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410f510 con 0x7fa654103ae0 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65bbb7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410f390 con 0x7fa654104d20 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65bbb7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410f690 con 0x7fa654109a20 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659953640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654103ae0 0x7fa65419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659953640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654103ae0 0x7fa65419a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41090/0 (socket says 172.21.15.98:41090) 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659953640 1 -- 172.21.15.98:0/110507915 learned_addr learned my addr 172.21.15.98:0/110507915 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659152640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa654104d20 0x7fa65419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65a154640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa654109a20 0x7fa6541a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659953640 1 -- 172.21.15.98:0/110507915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa654109a20 msgr2=0x7fa6541a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659953640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa654109a20 0x7fa6541a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659953640 1 -- 172.21.15.98:0/110507915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa654104d20 msgr2=0x7fa65419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659953640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa654104d20 0x7fa65419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa659953640 1 -- 172.21.15.98:0/110507915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6541a1c30 con 0x7fa654103ae0 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.325+0000 7fa65a154640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa654109a20 0x7fa6541a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.326+0000 7fa659953640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654103ae0 0x7fa65419a0f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa650005d00 tx=0x7fa65000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.326+0000 7fa64affd640 1 -- 172.21.15.98:0/110507915 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa650017070 con 0x7fa654103ae0 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.326+0000 7fa64affd640 1 -- 172.21.15.98:0/110507915 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa650009c30 con 0x7fa654103ae0 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.326+0000 7fa65bbb7640 1 -- 172.21.15.98:0/110507915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6541a1f20 con 0x7fa654103ae0 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.326+0000 7fa64affd640 1 -- 172.21.15.98:0/110507915 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65000f620 con 0x7fa654103ae0 2024-04-25T16:32:00.328 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.326+0000 7fa65bbb7640 1 -- 172.21.15.98:0/110507915 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6541a2330 con 0x7fa654103ae0 2024-04-25T16:32:00.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa662a6f740 1 Processor -- start 2024-04-25T16:32:00.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa662a6f740 1 -- start start 2024-04-25T16:32:00.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba68370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b69e3b0 0x56402ba688f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba6f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa662a6f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56402b9cd740 con 0x56402b6c7270 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa662a6f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56402b9693a0 con 0x56402b6a6590 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa662a6f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56402b9c9700 con 0x56402b69e3b0 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa65a154640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba68370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa65a154640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba68370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41104/0 (socket says 172.21.15.98:41104) 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa65a154640 1 -- 172.21.15.98:0/1824724542 learned_addr learned my addr 172.21.15.98:0/1824724542 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.329+0000 7fa64affd640 1 -- 172.21.15.98:0/110507915 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa65001a070 con 0x7fa654103ae0 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa659152640 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba6f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa65a154640 1 -- 172.21.15.98:0/1824724542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b69e3b0 msgr2=0x56402ba688f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa659953640 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b69e3b0 0x56402ba688f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa65a154640 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b69e3b0 0x56402ba688f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa65a154640 1 -- 172.21.15.98:0/1824724542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 msgr2=0x56402ba6f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa65a154640 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba6f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa65a154640 1 -- 172.21.15.98:0/1824724542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56402b985750 con 0x56402b6c7270 2024-04-25T16:32:00.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa659152640 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba6f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa659953640 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b69e3b0 0x56402ba688f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa65a154640 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba68370 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa64c005cc0 tx=0x7fa64c009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feaeaeac81e49abd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa64affd640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6400afaf0 0x7fa6400b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa63ffff640 1 -- 172.21.15.98:0/1824724542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c031a80 con 0x56402b6c7270 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa659953640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6400afaf0 0x7fa6400b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa63ffff640 1 -- 172.21.15.98:0/1824724542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa64c036070 con 0x56402b6c7270 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa63ffff640 1 -- 172.21.15.98:0/1824724542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c0044a0 con 0x56402b6c7270 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.330+0000 7fa64affd640 1 -- 172.21.15.98:0/110507915 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fa6500dea90 con 0x7fa654103ae0 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.331+0000 7fa662a6f740 1 -- 172.21.15.98:0/1824724542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 msgr2=0x56402ba68370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.331+0000 7fa659953640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6400afaf0 0x7fa6400b2000 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x56402ba69750 tx=0x7fa65000df10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.331+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba68370 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa64c005cc0 tx=0x7fa64c009a20 comp rx=0 tx=0).stop 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.331+0000 7fa662a6f740 1 -- 172.21.15.98:0/1824724542 shutdown_connections 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.331+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba6f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.331+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b69e3b0 0x56402ba688f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.331+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1824724542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba68370 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.331+0000 7fa662a6f740 1 -- 172.21.15.98:0/1824724542 >> 172.21.15.98:0/1824724542 conn(0x56402b6bb3c0 msgr2=0x56402ba63d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 -- 172.21.15.98:0/1824724542 shutdown_connections 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 -- 172.21.15.98:0/1824724542 wait complete. 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 -- start start 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 Processor -- start 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b69e3b0 0x56402ba72210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba72750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 0x56402ba796b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56402ba701e0 con 0x56402b69e3b0 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56402b954fe0 con 0x56402b6a6590 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56402ba70360 con 0x56402b6c7270 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa662a6f740 10 client.0 _ll_register_callbacks cb 0x56402b96b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa65a154640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b69e3b0 0x56402ba72210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa65a154640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b69e3b0 0x56402ba72210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41106/0 (socket says 172.21.15.98:41106) 2024-04-25T16:32:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.332+0000 7fa65a154640 1 -- 172.21.15.98:0/1496941779 learned_addr learned my addr 172.21.15.98:0/1496941779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa659152640 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba72750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa65a154640 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 0x56402ba796b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa65a154640 1 -- 172.21.15.98:0/1496941779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 msgr2=0x56402ba796b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa65a154640 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 0x56402ba796b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa65a154640 1 -- 172.21.15.98:0/1496941779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 msgr2=0x56402ba72750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa65a154640 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba72750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa65a154640 1 -- 172.21.15.98:0/1496941779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56402b6df2d0 con 0x56402b69e3b0 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa65a154640 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b69e3b0 0x56402ba72210 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa64c032ba0 tx=0x7fa64c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d394ac5c3d3091c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa63d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa63d7fa640 1 -- 172.21.15.98:0/1496941779 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c055060 con 0x56402b69e3b0 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa63d7fa640 1 -- 172.21.15.98:0/1496941779 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fa64c003b70 con 0x56402b69e3b0 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa662a6f740 1 -- 172.21.15.98:0/1496941779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa64c036070 con 0x56402b69e3b0 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa63d7fa640 1 -- 172.21.15.98:0/1496941779 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa64c003d70 con 0x56402b69e3b0 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa662a6f740 10 client.15306 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa662a6f740 1 -- 172.21.15.98:0/1496941779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56402b7a96b0 con 0x56402b69e3b0 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.333+0000 7fa63d7fa640 1 -- 172.21.15.98:0/1496941779 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa64c036070 con 0x56402b69e3b0 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa63d7fa640 1 -- 172.21.15.98:0/1496941779 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa64c0308a0 con 0x56402b69e3b0 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa662a6f740 10 client.15306 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa662a6f740 10 client.15306 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa662a6f740 20 client.15306 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56402b685e40 0x56402ba7f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa662a6f740 4 client.15306 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:00.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa662a6f740 1 -- 172.21.15.98:0/1496941779 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56402b9db350 con 0x56402b685e40 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa659152640 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56402b685e40 0x56402ba7f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa63d7fa640 10 client.15306 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.334+0000 7fa659152640 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56402b685e40 0x56402ba7f980 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e819809c246547 server_cookie=6d1bfec75d797455 in_seq=0 out_seq=0 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa63d7fa640 1 -- 172.21.15.98:0/1496941779 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56402b9db350 con 0x56402b685e40 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa63d7fa640 10 client.15306 handle_command_reply: tid=0 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 client.15306 shutdown 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 -- 172.21.15.98:0/1496941779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b69e3b0 msgr2=0x56402ba72210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b69e3b0 0x56402ba72210 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa64c032ba0 tx=0x7fa64c0095c0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 -- 172.21.15.98:0/1496941779 shutdown_connections 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56402b685e40 0x56402ba7f980 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 0x56402ba796b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b6a6590 0x56402ba72750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1496941779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b69e3b0 0x56402ba72210 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa63d7fa640 0 client.15306 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.335+0000 7fa662a6f740 1 -- 172.21.15.98:0/1496941779 >> 172.21.15.98:0/1496941779 conn(0x56402b6bb3c0 msgr2=0x56402ba62d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.336+0000 7fa662a6f740 1 -- 172.21.15.98:0/1496941779 shutdown_connections 2024-04-25T16:32:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.336+0000 7fa662a6f740 1 -- 172.21.15.98:0/1496941779 wait complete. 2024-04-25T16:32:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.336+0000 7fa662a6f740 20 client.15306 trim_cache size 0 max 16384 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.354+0000 7fa662a6f740 1 Processor -- start 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.354+0000 7fa662a6f740 1 -- start start 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.354+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba636b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.354+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6a6590 0x56402ba79640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.354+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 0x56402ba63bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa662a6f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56402ba7dcc0 con 0x56402b6a6590 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa662a6f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56402ba7e390 con 0x56402b69e3b0 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa662a6f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56402ba7ea60 con 0x56402b6c7270 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6a6590 0x56402ba79640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6a6590 0x56402ba79640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41122/0 (socket says 172.21.15.98:41122) 2024-04-25T16:32:00.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 -- 172.21.15.98:0/1920755326 learned_addr learned my addr 172.21.15.98:0/1920755326 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa65a154640 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 0x56402ba63bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa65a154640 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 -- 172.21.15.98:0/1920755326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 msgr2=0x56402ba63bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 0x56402ba63bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 -- 172.21.15.98:0/1920755326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 msgr2=0x56402ba636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba636b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 -- 172.21.15.98:0/1920755326 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56402ba6d0f0 con 0x56402b6a6590 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa65a154640 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa659152640 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6a6590 0x56402ba79640 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa65419b190 tx=0x7fa64400b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09130f616096f89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa63f7fe640 1 -- 172.21.15.98:0/1920755326 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64401f020 con 0x56402b6a6590 2024-04-25T16:32:00.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa63f7fe640 1 -- 172.21.15.98:0/1920755326 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa644003de0 con 0x56402b6a6590 2024-04-25T16:32:00.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.355+0000 7fa63f7fe640 1 -- 172.21.15.98:0/1920755326 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644012ab0 con 0x56402b6a6590 2024-04-25T16:32:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 -- 172.21.15.98:0/1920755326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6a6590 msgr2=0x56402ba79640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6a6590 0x56402ba79640 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa65419b190 tx=0x7fa64400b730 comp rx=0 tx=0).stop 2024-04-25T16:32:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 -- 172.21.15.98:0/1920755326 shutdown_connections 2024-04-25T16:32:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6c7270 0x56402ba63bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6a6590 0x56402ba79640 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 --2- 172.21.15.98:0/1920755326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 -- 172.21.15.98:0/1920755326 >> 172.21.15.98:0/1920755326 conn(0x56402b6bb3c0 msgr2=0x56402ba70850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.361 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 -- 172.21.15.98:0/1920755326 shutdown_connections 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 -- 172.21.15.98:0/1920755326 wait complete. 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.356+0000 7fa662a6f740 1 -- start start 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa662a6f740 1 Processor -- start 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba750d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6a6590 0x56402ba75610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa662a6f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba784b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa662a6f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56402ba7f800 con 0x56402b6c7270 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa662a6f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56402ba7f130 con 0x56402b69e3b0 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa662a6f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56402ba7fed0 con 0x56402b6a6590 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa662a6f740 10 client.0 _ll_register_callbacks cb 0x7fa64c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:00.362 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba784b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba784b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41134/0 (socket says 172.21.15.98:41134) 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 -- 172.21.15.98:0/3491757413 learned_addr learned my addr 172.21.15.98:0/3491757413 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa659152640 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6a6590 0x56402ba75610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba750d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 -- 172.21.15.98:0/3491757413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6a6590 msgr2=0x56402ba75610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6a6590 0x56402ba75610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 -- 172.21.15.98:0/3491757413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 msgr2=0x56402ba750d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba750d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.357+0000 7fa65a154640 1 -- 172.21.15.98:0/3491757413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56402ba78a70 con 0x56402b6c7270 2024-04-25T16:32:00.363 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa65a154640 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba784b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa64c005e30 tx=0x7fa64c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=498e8d7b359c289a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa63d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa63d7fa640 1 -- 172.21.15.98:0/3491757413 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64c064020 con 0x56402b6c7270 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa63d7fa640 1 -- 172.21.15.98:0/3491757413 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fa64c0309d0 con 0x56402b6c7270 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 1 -- 172.21.15.98:0/3491757413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa644003de0 con 0x56402b6c7270 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa63d7fa640 1 -- 172.21.15.98:0/3491757413 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa64c06f050 con 0x56402b6c7270 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 10 client.15312 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 1 -- 172.21.15.98:0/3491757413 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56402bae8890 con 0x56402b6c7270 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa63d7fa640 1 -- 172.21.15.98:0/3491757413 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa644003de0 con 0x56402b6c7270 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa63d7fa640 1 -- 172.21.15.98:0/3491757413 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa64c044430 con 0x56402b6c7270 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 10 client.15312 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 10 client.15312 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 20 client.15312 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56402b65b280 0x56402baec190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 4 client.15312 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:32:00.364 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.358+0000 7fa662a6f740 1 -- 172.21.15.98:0/3491757413 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "status"}) -- 0x56402b9d51c0 con 0x56402b65b280 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.359+0000 7fa659152640 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56402b65b280 0x56402baec190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.359+0000 7fa63d7fa640 10 client.15312 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.359+0000 7fa659152640 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56402b65b280 0x56402baec190 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bae52cf7cc44b496 server_cookie=52250911b259ee6a in_seq=0 out_seq=0 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.359+0000 7fa63d7fa640 1 -- 172.21.15.98:0/3491757413 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x56402b9d51c0 con 0x56402b65b280 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.359+0000 7fa63d7fa640 10 client.15312 handle_command_reply: tid=0 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.359+0000 7fa662a6f740 1 client.15312 shutdown 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 -- 172.21.15.98:0/3491757413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 msgr2=0x56402ba784b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba784b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa64c005e30 tx=0x7fa64c009900 comp rx=0 tx=0).stop 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 -- 172.21.15.98:0/3491757413 shutdown_connections 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56402b65b280 0x56402baec190 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56402b6c7270 0x56402ba784b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56402b6a6590 0x56402ba75610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 --2- 172.21.15.98:0/3491757413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56402b69e3b0 0x56402ba750d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa63d7fa640 0 client.15312 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 -- 172.21.15.98:0/3491757413 >> 172.21.15.98:0/3491757413 conn(0x56402b6bb3c0 msgr2=0x56402ba636b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.365 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 -- 172.21.15.98:0/3491757413 shutdown_connections 2024-04-25T16:32:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 1 -- 172.21.15.98:0/3491757413 wait complete. 2024-04-25T16:32:00.366 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa662a6f740 20 client.15312 trim_cache size 0 max 16384 2024-04-25T16:32:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:00.366 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:32:00.366 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-04-25T16:32:00.366 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24539, 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 110.02290078599999, 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 83, 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 115.744142825 2024-04-25T16:32:00.367 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 -- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6400afaf0 msgr2=0x7fa6400b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6400afaf0 0x7fa6400b2000 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x56402ba69750 tx=0x7fa65000df10 comp rx=0 tx=0).stop 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 -- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654103ae0 msgr2=0x7fa65419a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654103ae0 0x7fa65419a0f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa650005d00 tx=0x7fa65000dde0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 -- 172.21.15.98:0/110507915 shutdown_connections 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6400afaf0 0x7fa6400b2000 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa654109a20 0x7fa6541a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa654104d20 0x7fa65419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 --2- 172.21.15.98:0/110507915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa654103ae0 0x7fa65419a0f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 -- 172.21.15.98:0/110507915 >> 172.21.15.98:0/110507915 conn(0x7fa6540fecd0 msgr2=0x7fa654105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:00.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 -- 172.21.15.98:0/110507915 shutdown_connections 2024-04-25T16:32:00.369 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:00.360+0000 7fa65bbb7640 1 -- 172.21.15.98:0/110507915 wait complete. 2024-04-25T16:32:00.375 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T16:32:00.376 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T16:32:00.385 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T16:32:00.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T16:32:00.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:00 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2831402038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:00.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:00 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/736801208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:00.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:00 smithi203 ceph-mon[33115]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:32:00.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:00 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2831402038' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:00.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:00 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/736801208' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:00.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:00 smithi173 ceph-mon[32556]: pgmap v339: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:32:00.635 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:32:02.353 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:02 smithi098 ceph-mon[29162]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:32:02.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:02 smithi203 ceph-mon[33115]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:32:02.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1242545467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd8104210 msgr2=0x7f8bd8104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:02.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1242545467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd8104210 0x7f8bd8104620 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009990 tx=0x7f8bc8009ca0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1242545467 shutdown_connections 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1242545467 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bd8104bb0 0x7f8bd810b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1242545467 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bd81030e0 0x7f8bd81034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1242545467 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd8104210 0x7f8bd8104620 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1242545467 >> 172.21.15.98:0/1242545467 conn(0x7f8bd80fe740 msgr2=0x7f8bd8100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1242545467 shutdown_connections 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.466+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1242545467 wait complete. 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdf99b700 1 Processor -- start 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdf99b700 1 -- start start 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdf99b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bd81030e0 0x7f8bd8076340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:02.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdf99b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd8104210 0x7f8bd819f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdf99b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bd8104bb0 0x7f8bd81a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdf99b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd810ea10 con 0x7f8bd8104210 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdf99b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd810e890 con 0x7f8bd81030e0 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdf99b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd810eb90 con 0x7f8bd8104bb0 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdd737700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bd81030e0 0x7f8bd8076340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdd737700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bd81030e0 0x7f8bd8076340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:44618/0 (socket says 172.21.15.98:44618) 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdd737700 1 -- 172.21.15.98:0/1385719176 learned_addr learned my addr 172.21.15.98:0/1385719176 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bdcf36700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd8104210 0x7f8bd819f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.467+0000 7f8bddf38700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bd8104bb0 0x7f8bd81a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bdd737700 1 -- 172.21.15.98:0/1385719176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bd8104bb0 msgr2=0x7f8bd81a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bdd737700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bd8104bb0 0x7f8bd81a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bdd737700 1 -- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd8104210 msgr2=0x7f8bd819f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bdd737700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd8104210 0x7f8bd819f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bdd737700 1 -- 172.21.15.98:0/1385719176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd81a2250 con 0x7f8bd81030e0 2024-04-25T16:32:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bdd737700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bd81030e0 0x7f8bd8076340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400d950 tx=0x7f8bd400dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bce7fc700 1 -- 172.21.15.98:0/1385719176 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd4017070 con 0x7f8bd81030e0 2024-04-25T16:32:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd81a2540 con 0x7f8bd81030e0 2024-04-25T16:32:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bce7fc700 1 -- 172.21.15.98:0/1385719176 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8bd400f690 con 0x7f8bd81030e0 2024-04-25T16:32:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bce7fc700 1 -- 172.21.15.98:0/1385719176 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bd40039e0 con 0x7f8bd81030e0 2024-04-25T16:32:02.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.468+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd81a2ad0 con 0x7f8bd81030e0 2024-04-25T16:32:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.469+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd8051190 con 0x7f8bd81030e0 2024-04-25T16:32:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.471+0000 7f8bce7fc700 1 -- 172.21.15.98:0/1385719176 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8bd4003b80 con 0x7f8bd81030e0 2024-04-25T16:32:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.471+0000 7f8bce7fc700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8bc40afb80 0x7f8bc40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.471+0000 7f8bdcf36700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8bc40afb80 0x7f8bc40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.472+0000 7f8bce7fc700 1 -- 172.21.15.98:0/1385719176 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f8bd40df1c0 con 0x7f8bd81030e0 2024-04-25T16:32:02.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.472+0000 7f8bdcf36700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8bc40afb80 0x7f8bc40b2090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800b5c0 tx=0x7f8bc801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:02.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.475+0000 7f8bce7fc700 1 -- 172.21.15.98:0/1385719176 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8bd4020030 con 0x7f8bd81030e0 2024-04-25T16:32:02.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:02 smithi173 ceph-mon[32556]: pgmap v340: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:32:02.688 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.688+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f8bd8063ad0 con 0x7f8bc40afb80 2024-04-25T16:32:02.689 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.690+0000 7f8bce7fc700 1 -- 172.21.15.98:0/1385719176 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f8bd8063ad0 con 0x7f8bc40afb80 2024-04-25T16:32:02.689 INFO:teuthology.orchestra.run.smithi098.stdout:Backend: cephadm 2024-04-25T16:32:02.690 INFO:teuthology.orchestra.run.smithi098.stdout:Available: Yes 2024-04-25T16:32:02.690 INFO:teuthology.orchestra.run.smithi098.stdout:Paused: No 2024-04-25T16:32:02.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8bc40afb80 msgr2=0x7f8bc40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8bc40afb80 0x7f8bc40b2090 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800b5c0 tx=0x7f8bc801a040 comp rx=0 tx=0).stop 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bd81030e0 msgr2=0x7f8bd8076340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bd81030e0 0x7f8bd8076340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8bd400d950 tx=0x7f8bd400dc60 comp rx=0 tx=0).stop 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 shutdown_connections 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8bc40afb80 0x7f8bc40b2090 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8bd8104bb0 0x7f8bd81a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8bd81030e0 0x7f8bd8076340 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 --2- 172.21.15.98:0/1385719176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8bd8104210 0x7f8bd819f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 >> 172.21.15.98:0/1385719176 conn(0x7f8bd80fe740 msgr2=0x7f8bd8107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 shutdown_connections 2024-04-25T16:32:02.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:02.693+0000 7f8bdf99b700 1 -- 172.21.15.98:0/1385719176 wait complete. 2024-04-25T16:32:03.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:03 smithi203 ceph-mon[33115]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:03.476 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:03 smithi098 ceph-mon[29162]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:03.511 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T16:32:03.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:03 smithi173 ceph-mon[32556]: from='client.25034 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:03.756 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:32:04.138 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:04 smithi098 ceph-mon[29162]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:32:04.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:04 smithi203 ceph-mon[33115]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:32:04.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:04 smithi173 ceph-mon[32556]: pgmap v341: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 -- 172.21.15.98:0/1088180142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c1039c0 msgr2=0x7ff32c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 --2- 172.21.15.98:0/1088180142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c1039c0 0x7ff32c103dd0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff318009990 tx=0x7ff318009ca0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 -- 172.21.15.98:0/1088180142 shutdown_connections 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 --2- 172.21.15.98:0/1088180142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32c104360 0x7ff32c10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 --2- 172.21.15.98:0/1088180142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff32c102890 0x7ff32c102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 --2- 172.21.15.98:0/1088180142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c1039c0 0x7ff32c103dd0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 -- 172.21.15.98:0/1088180142 >> 172.21.15.98:0/1088180142 conn(0x7ff32c0fded0 msgr2=0x7ff32c100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 -- 172.21.15.98:0/1088180142 shutdown_connections 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.552+0000 7ff33302d700 1 -- 172.21.15.98:0/1088180142 wait complete. 2024-04-25T16:32:05.553 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff33302d700 1 Processor -- start 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff33302d700 1 -- start start 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff33302d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32c102890 0x7ff32c19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff33302d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff32c1039c0 0x7ff32c19ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff33302d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c104360 0x7ff32c1a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff33302d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c10fbd0 con 0x7ff32c104360 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff33302d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c10fa50 con 0x7ff32c1039c0 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff33302d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff32c10fd50 con 0x7ff32c102890 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff3315ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c104360 0x7ff32c1a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff3315ca700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c104360 0x7ff32c1a1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41912/0 (socket says 172.21.15.98:41912) 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff3315ca700 1 -- 172.21.15.98:0/528482397 learned_addr learned my addr 172.21.15.98:0/528482397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff330dc9700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32c102890 0x7ff32c19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.553+0000 7ff32bfff700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff32c1039c0 0x7ff32c19ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff3315ca700 1 -- 172.21.15.98:0/528482397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32c102890 msgr2=0x7ff32c19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:05.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff3315ca700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32c102890 0x7ff32c19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff3315ca700 1 -- 172.21.15.98:0/528482397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff32c1039c0 msgr2=0x7ff32c19ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:05.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff3315ca700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff32c1039c0 0x7ff32c19ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff3315ca700 1 -- 172.21.15.98:0/528482397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff32c1a21e0 con 0x7ff32c104360 2024-04-25T16:32:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff3315ca700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c104360 0x7ff32c1a1bf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff324007970 tx=0x7ff324007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff329ffb700 1 -- 172.21.15.98:0/528482397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32400d5e0 con 0x7ff32c104360 2024-04-25T16:32:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff32c1a24d0 con 0x7ff32c104360 2024-04-25T16:32:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff329ffb700 1 -- 172.21.15.98:0/528482397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff32400dc10 con 0x7ff32c104360 2024-04-25T16:32:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff329ffb700 1 -- 172.21.15.98:0/528482397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff324013cf0 con 0x7ff32c104360 2024-04-25T16:32:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.554+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff32c1a2a60 con 0x7ff32c104360 2024-04-25T16:32:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.556+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff32c051190 con 0x7ff32c104360 2024-04-25T16:32:05.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.557+0000 7ff329ffb700 1 -- 172.21.15.98:0/528482397 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff32401a020 con 0x7ff32c104360 2024-04-25T16:32:05.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.558+0000 7ff329ffb700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff31c0afc00 0x7ff31c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:05.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.558+0000 7ff330dc9700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff31c0afc00 0x7ff31c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:05.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.558+0000 7ff329ffb700 1 -- 172.21.15.98:0/528482397 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7ff3240de5a0 con 0x7ff32c104360 2024-04-25T16:32:05.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.558+0000 7ff330dc9700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff31c0afc00 0x7ff31c0b2110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff32c1038c0 tx=0x7ff32000a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:05.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.562+0000 7ff329ffb700 1 -- 172.21.15.98:0/528482397 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff324010070 con 0x7ff32c104360 2024-04-25T16:32:05.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.769+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7ff32c063ad0 con 0x7ff31c0afc00 2024-04-25T16:32:05.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.782+0000 7ff329ffb700 1 -- 172.21.15.98:0/528482397 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4092 (secure 0 0 0) 0x7ff32c063ad0 con 0x7ff31c0afc00 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.a smithi098 running (2m) 20s ago 2m 83.7M - 19.0.0-3291-g994e2f20 d53c96779a29 c85411a4bd2a 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.b smithi173 running (2m) 54s ago 2m 96.5M - 19.0.0-3291-g994e2f20 d53c96779a29 2d7b6bdfc8eb 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.c smithi203 running (2m) 54s ago 2m 83.6M - 19.0.0-3291-g994e2f20 d53c96779a29 684bc2cb2319 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.d smithi098 running (2m) 20s ago 2m 83.0M - 19.0.0-3291-g994e2f20 d53c96779a29 c6012484385a 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.e smithi173 running (2m) 54s ago 2m 84.4M - 19.0.0-3291-g994e2f20 d53c96779a29 ac8af86570cc 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.f smithi203 running (2m) 54s ago 2m 86.1M - 19.0.0-3291-g994e2f20 d53c96779a29 28934d9ccdf7 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.g smithi098 running (2m) 20s ago 2m 85.3M - 19.0.0-3291-g994e2f20 d53c96779a29 ca2f757698d5 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.h smithi173 running (2m) 54s ago 2m 84.9M - 19.0.0-3291-g994e2f20 d53c96779a29 e35367aa004f 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.i smithi203 running (2m) 54s ago 2m 84.0M - 19.0.0-3291-g994e2f20 d53c96779a29 33242d8aa263 2024-04-25T16:32:05.782 INFO:teuthology.orchestra.run.smithi098.stdout:mds.j smithi098 running (2m) 20s ago 2m 84.1M - 19.0.0-3291-g994e2f20 d53c96779a29 62399e7e2252 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:mds.k smithi173 running (2m) 54s ago 2m 83.5M - 19.0.0-3291-g994e2f20 d53c96779a29 c50286901662 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:mds.l smithi203 running (2m) 54s ago 2m 84.8M - 19.0.0-3291-g994e2f20 d53c96779a29 eb5325b0a0ff 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi098 *:9283,8765 running (9m) 20s ago 9m 584M - 19.0.0-3291-g994e2f20 d53c96779a29 4f5165254f85 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi173 *:8443,8765 running (7m) 54s ago 7m 520M - 19.0.0-3291-g994e2f20 d53c96779a29 4390a4506f4a 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.z smithi203 *:8443,8765 running (7m) 54s ago 7m 523M - 19.0.0-3291-g994e2f20 d53c96779a29 8c8b25a83d7d 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (9m) 20s ago 9m 119M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 14b5f450ea62 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi173 running (8m) 54s ago 8m 110M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 8af9d65e7b93 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi203 running (8m) 54s ago 8m 106M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 bb42c2be863d 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (7m) 20s ago 7m 123M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 0e442e05dd3f 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi173 running (7m) 54s ago 7m 121M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 9df6fa099810 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi203 running (6m) 54s ago 6m 117M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 d3661c45945a 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (6m) 20s ago 6m 122M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 793d66d8c165 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi173 running (5m) 54s ago 5m 118M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 60155c5098f3 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi203 running (5m) 54s ago 5m 122M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 b41248985744 2024-04-25T16:32:05.783 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi098 running (5m) 20s ago 4m 108M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 8cd85ba1e1a4 2024-04-25T16:32:05.784 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi173 running (4m) 54s ago 4m 112M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 8242f02e59a4 2024-04-25T16:32:05.784 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 smithi203 running (4m) 54s ago 4m 118M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 b4cbd7bc44c7 2024-04-25T16:32:05.784 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 smithi098 running (3m) 20s ago 3m 112M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 92514b8edd23 2024-04-25T16:32:05.784 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 smithi173 running (3m) 54s ago 3m 122M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 24c5a9729bb7 2024-04-25T16:32:05.784 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 smithi203 running (2m) 54s ago 2m 108M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 7ab3517cc00e 2024-04-25T16:32:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff31c0afc00 msgr2=0x7ff31c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:05.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff31c0afc00 0x7ff31c0b2110 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff32c1038c0 tx=0x7ff32000a3b0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c104360 msgr2=0x7ff32c1a1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c104360 0x7ff32c1a1bf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff324007970 tx=0x7ff324007c80 comp rx=0 tx=0).stop 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 shutdown_connections 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff31c0afc00 0x7ff31c0b2110 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff32c102890 0x7ff32c19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff32c104360 0x7ff32c1a1bf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 --2- 172.21.15.98:0/528482397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff32c1039c0 0x7ff32c19ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 >> 172.21.15.98:0/528482397 conn(0x7ff32c0fded0 msgr2=0x7ff32c106bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 shutdown_connections 2024-04-25T16:32:05.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:05.784+0000 7ff33302d700 1 -- 172.21.15.98:0/528482397 wait complete. 2024-04-25T16:32:06.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:06 smithi203 ceph-mon[33115]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:06.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:06 smithi203 ceph-mon[33115]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:32:06.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:06 smithi098 ceph-mon[29162]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:06.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:06 smithi098 ceph-mon[29162]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:32:06.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:06 smithi173 ceph-mon[32556]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:06.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:06 smithi173 ceph-mon[32556]: pgmap v342: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:32:06.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T16:32:06.820 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:32:08.368 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:08 smithi098 ceph-mon[29162]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:32:08.368 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-25T16:32:08.368 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-04-25T16:32:08.368 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-25T16:32:08.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:08 smithi203 ceph-mon[33115]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:32:08.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-25T16:32:08.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-04-25T16:32:08.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-25T16:32:08.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:08 smithi173 ceph-mon[32556]: pgmap v343: 129 pgs: 129 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:32:08.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-25T16:32:08.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]: dispatch 2024-04-25T16:32:08.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-04-25T16:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.587+0000 7f1f11d00700 1 -- 172.21.15.98:0/2400427614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c104d00 msgr2=0x7f1f0c107120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.587+0000 7f1f11d00700 1 --2- 172.21.15.98:0/2400427614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c104d00 0x7f1f0c107120 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1f00009990 tx=0x7f1f00009ca0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 -- 172.21.15.98:0/2400427614 shutdown_connections 2024-04-25T16:32:08.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 --2- 172.21.15.98:0/2400427614 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f0c109fc0 0x7f1f0c10c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 --2- 172.21.15.98:0/2400427614 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c104d00 0x7f1f0c107120 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 --2- 172.21.15.98:0/2400427614 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f0c107660 0x7f1f0c109a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 -- 172.21.15.98:0/2400427614 >> 172.21.15.98:0/2400427614 conn(0x7f1f0c0fe740 msgr2=0x7f1f0c100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 -- 172.21.15.98:0/2400427614 shutdown_connections 2024-04-25T16:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 -- 172.21.15.98:0/2400427614 wait complete. 2024-04-25T16:32:08.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 Processor -- start 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.588+0000 7f1f11d00700 1 -- start start 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f11d00700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f0c104d00 0x7f1f0c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f11d00700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f0c107660 0x7f1f0c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f11d00700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c109fc0 0x7f1f0c1a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f11d00700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c10fbe0 con 0x7f1f0c109fc0 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f11d00700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c10fa60 con 0x7f1f0c107660 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f11d00700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c10fd60 con 0x7f1f0c104d00 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c109fc0 0x7f1f0c1a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c109fc0 0x7f1f0c1a1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41934/0 (socket says 172.21.15.98:41934) 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 -- 172.21.15.98:0/4123948542 learned_addr learned my addr 172.21.15.98:0/4123948542 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:08.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0b7fe700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f0c104d00 0x7f1f0c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0affd700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f0c107660 0x7f1f0c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 -- 172.21.15.98:0/4123948542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f0c104d00 msgr2=0x7f1f0c19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f0c104d00 0x7f1f0c19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 -- 172.21.15.98:0/4123948542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f0c107660 msgr2=0x7f1f0c19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f0c107660 0x7f1f0c19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 -- 172.21.15.98:0/4123948542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0c1a2370 con 0x7f1f0c109fc0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0affd700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f0c107660 0x7f1f0c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f0bfff700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c109fc0 0x7f1f0c1a1cf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b970 tx=0x7f1f0400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f08ff9700 1 -- 172.21.15.98:0/4123948542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04019070 con 0x7f1f0c109fc0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f08ff9700 1 -- 172.21.15.98:0/4123948542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f0400cdc0 con 0x7f1f0c109fc0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.589+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0c102b50 con 0x7f1f0c109fc0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.590+0000 7f1f08ff9700 1 -- 172.21.15.98:0/4123948542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04014e30 con 0x7f1f0c109fc0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.590+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f0c1030c0 con 0x7f1f0c109fc0 2024-04-25T16:32:08.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.590+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f0c051190 con 0x7f1f0c109fc0 2024-04-25T16:32:08.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.593+0000 7f1f08ff9700 1 -- 172.21.15.98:0/4123948542 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1f0401c070 con 0x7f1f0c109fc0 2024-04-25T16:32:08.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.593+0000 7f1f08ff9700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ef40afb80 0x7f1ef40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:08.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.593+0000 7f1f0b7fe700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ef40afb80 0x7f1ef40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:08.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.594+0000 7f1f08ff9700 1 -- 172.21.15.98:0/4123948542 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f1f040de740 con 0x7f1f0c109fc0 2024-04-25T16:32:08.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.594+0000 7f1f0b7fe700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ef40afb80 0x7f1ef40b2090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000b5c0 tx=0x7f1f0001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:08.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.597+0000 7f1f08ff9700 1 -- 172.21.15.98:0/4123948542 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f040a6fc0 con 0x7f1f0c109fc0 2024-04-25T16:32:08.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.806+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f1f0c063ad0 con 0x7f1ef40afb80 2024-04-25T16:32:08.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.812+0000 7f1f08ff9700 1 -- 172.21.15.98:0/4123948542 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f1f0c063ad0 con 0x7f1ef40afb80 2024-04-25T16:32:08.812 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T16:32:08.812 INFO:teuthology.orchestra.run.smithi098.stdout:mds.all 12/12 57s ago 2m smithi098=a;smithi098=d;smithi098=g;smithi098=j;smithi173=b;smithi173=e;smithi173=h;smithi173=k;smithi203=c;smithi203=f;smithi203=i;smithi203=l;count:12 2024-04-25T16:32:08.812 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 3/3 57s ago 7m smithi098=x;smithi173=y;smithi203=z;count:3 2024-04-25T16:32:08.812 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 57s ago 8m smithi098:172.21.15.98=a;smithi173:172.21.15.173=b;smithi203:172.21.15.203=c;count:3 2024-04-25T16:32:08.812 INFO:teuthology.orchestra.run.smithi098.stdout:osd 12 57s ago - 2024-04-25T16:32:08.814 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ef40afb80 msgr2=0x7f1ef40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ef40afb80 0x7f1ef40b2090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000b5c0 tx=0x7f1f0001a040 comp rx=0 tx=0).stop 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c109fc0 msgr2=0x7f1f0c1a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c109fc0 0x7f1f0c1a1cf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400b970 tx=0x7f1f0400bc80 comp rx=0 tx=0).stop 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 shutdown_connections 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ef40afb80 0x7f1ef40b2090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f0c104d00 0x7f1f0c19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f0c109fc0 0x7f1f0c1a1cf0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 --2- 172.21.15.98:0/4123948542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f0c107660 0x7f1f0c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.814+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 >> 172.21.15.98:0/4123948542 conn(0x7f1f0c0fe740 msgr2=0x7f1f0c106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.815+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 shutdown_connections 2024-04-25T16:32:08.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:08.815+0000 7f1f11d00700 1 -- 172.21.15.98:0/4123948542 wait complete. 2024-04-25T16:32:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:09 smithi203 ceph-mon[33115]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-25T16:32:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-04-25T16:32:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-25T16:32:09.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:09 smithi203 ceph-mon[33115]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T16:32:09.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:09 smithi098 ceph-mon[29162]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:09.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-25T16:32:09.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-04-25T16:32:09.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-25T16:32:09.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:09 smithi098 ceph-mon[29162]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T16:32:09.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:09 smithi173 ceph-mon[32556]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:09.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-25T16:32:09.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 1]}]': finished 2024-04-25T16:32:09.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-04-25T16:32:09.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:09 smithi173 ceph-mon[32556]: osdmap e84: 12 total, 12 up, 12 in 2024-04-25T16:32:09.642 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T16:32:09.880 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:32:10.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:10 smithi098 ceph-mon[29162]: pgmap v345: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:32:10.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:10 smithi098 ceph-mon[29162]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T16:32:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:10 smithi203 ceph-mon[33115]: pgmap v345: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:32:10.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:10 smithi203 ceph-mon[33115]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T16:32:10.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:10 smithi173 ceph-mon[32556]: pgmap v345: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:32:10.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:10 smithi173 ceph-mon[32556]: osdmap e85: 12 total, 12 up, 12 in 2024-04-25T16:32:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 -- 172.21.15.98:0/752778454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6008102b70 msgr2=0x7f600810c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 --2- 172.21.15.98:0/752778454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6008102b70 0x7f600810c790 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f600000b0a0 tx=0x7f600000b3b0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 -- 172.21.15.98:0/752778454 shutdown_connections 2024-04-25T16:32:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 --2- 172.21.15.98:0/752778454 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6008102b70 0x7f600810c790 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 --2- 172.21.15.98:0/752778454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60081057d0 0x7f6008105be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 --2- 172.21.15.98:0/752778454 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6008102220 0x7f6008102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 -- 172.21.15.98:0/752778454 >> 172.21.15.98:0/752778454 conn(0x7f60080fdef0 msgr2=0x7f60081003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 -- 172.21.15.98:0/752778454 shutdown_connections 2024-04-25T16:32:11.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.709+0000 7f600cc9d700 1 -- 172.21.15.98:0/752778454 wait complete. 2024-04-25T16:32:11.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f600cc9d700 1 Processor -- start 2024-04-25T16:32:11.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f600cc9d700 1 -- start start 2024-04-25T16:32:11.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f600cc9d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6008102220 0x7f600819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f600cc9d700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6008102b70 0x7f600819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f600cc9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60081057d0 0x7f60081a3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f600cc9d700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f600810fb60 con 0x7f60081057d0 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f600cc9d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f600810f9e0 con 0x7f6008102220 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f600cc9d700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f600810fce0 con 0x7f6008102b70 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f6006d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60081057d0 0x7f60081a3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f6006d9d700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60081057d0 0x7f60081a3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41944/0 (socket says 172.21.15.98:41944) 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.710+0000 7f6006d9d700 1 -- 172.21.15.98:0/2119376845 learned_addr learned my addr 172.21.15.98:0/2119376845 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f600659c700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6008102220 0x7f600819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f6005d9b700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6008102b70 0x7f600819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f6006d9d700 1 -- 172.21.15.98:0/2119376845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6008102b70 msgr2=0x7f600819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f6006d9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6008102b70 0x7f600819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f6006d9d700 1 -- 172.21.15.98:0/2119376845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6008102220 msgr2=0x7f600819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f6006d9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6008102220 0x7f600819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f6006d9d700 1 -- 172.21.15.98:0/2119376845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60081a4440 con 0x7f60081057d0 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f6006d9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60081057d0 0x7f60081a3dc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6000009030 tx=0x7f6000009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f5ff77fe700 1 -- 172.21.15.98:0/2119376845 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600001f030 con 0x7f60081057d0 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f5ff77fe700 1 -- 172.21.15.98:0/2119376845 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6000007a00 con 0x7f60081057d0 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60081a46d0 con 0x7f60081057d0 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f5ff77fe700 1 -- 172.21.15.98:0/2119376845 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f600001b690 con 0x7f60081057d0 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.711+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60081a4b80 con 0x7f60081057d0 2024-04-25T16:32:11.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.712+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6008051190 con 0x7f60081057d0 2024-04-25T16:32:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.714+0000 7f5ff77fe700 1 -- 172.21.15.98:0/2119376845 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6000022030 con 0x7f60081057d0 2024-04-25T16:32:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.714+0000 7f5ff77fe700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff00afb80 0x7f5ff00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.714+0000 7f600659c700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff00afb80 0x7f5ff00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.715+0000 7f5ff77fe700 1 -- 172.21.15.98:0/2119376845 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f600000f020 con 0x7f60081057d0 2024-04-25T16:32:11.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.715+0000 7f600659c700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff00afb80 0x7f5ff00b2090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8005950 tx=0x7f5ff80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:11.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.718+0000 7f5ff77fe700 1 -- 172.21.15.98:0/2119376845 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6000029020 con 0x7f60081057d0 2024-04-25T16:32:11.918 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.919+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f6008104a50 con 0x7f5ff00afb80 2024-04-25T16:32:11.919 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.920+0000 7f5ff77fe700 1 -- 172.21.15.98:0/2119376845 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f6008104a50 con 0x7f5ff00afb80 2024-04-25T16:32:11.920 INFO:teuthology.orchestra.run.smithi098.stdout:HOST ADDR LABELS STATUS 2024-04-25T16:32:11.920 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 172.21.15.98 2024-04-25T16:32:11.920 INFO:teuthology.orchestra.run.smithi098.stdout:smithi173 172.21.15.173 2024-04-25T16:32:11.920 INFO:teuthology.orchestra.run.smithi098.stdout:smithi203 172.21.15.203 2024-04-25T16:32:11.920 INFO:teuthology.orchestra.run.smithi098.stdout:3 hosts in cluster 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff00afb80 msgr2=0x7f5ff00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff00afb80 0x7f5ff00b2090 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5ff8005950 tx=0x7f5ff80058e0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60081057d0 msgr2=0x7f60081a3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60081057d0 0x7f60081a3dc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f6000009030 tx=0x7f6000009c30 comp rx=0 tx=0).stop 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 shutdown_connections 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5ff00afb80 0x7f5ff00b2090 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f60081057d0 0x7f60081a3dc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6008102220 0x7f600819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 --2- 172.21.15.98:0/2119376845 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6008102b70 0x7f600819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:11.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 >> 172.21.15.98:0/2119376845 conn(0x7f60080fdef0 msgr2=0x7f6008103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 shutdown_connections 2024-04-25T16:32:11.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:11.922+0000 7f600cc9d700 1 -- 172.21.15.98:0/2119376845 wait complete. 2024-04-25T16:32:12.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:12 smithi098 ceph-mon[29162]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:12.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:12 smithi098 ceph-mon[29162]: pgmap v347: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-25T16:32:12.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:12 smithi203 ceph-mon[33115]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:12.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:12 smithi203 ceph-mon[33115]: pgmap v347: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-25T16:32:12.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:12 smithi173 ceph-mon[32556]: from='client.15336 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:12.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:12 smithi173 ceph-mon[32556]: pgmap v347: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-04-25T16:32:12.798 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T16:32:13.041 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:32:14.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:14 smithi098 ceph-mon[29162]: pgmap v348: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:14.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:14 smithi203 ceph-mon[33115]: pgmap v348: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:14.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:14 smithi173 ceph-mon[32556]: pgmap v348: 129 pgs: 1 peering, 128 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 -- 172.21.15.98:0/2493201860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e081048b0 msgr2=0x7f6e0810c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:14.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 --2- 172.21.15.98:0/2493201860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e081048b0 0x7f6e0810c5c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6e0000b0a0 tx=0x7f6e0000b3b0 comp rx=0 tx=0).stop 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 -- 172.21.15.98:0/2493201860 shutdown_connections 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 --2- 172.21.15.98:0/2493201860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e081048b0 0x7f6e0810c5c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 --2- 172.21.15.98:0/2493201860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e08105880 0x7f6e08105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 --2- 172.21.15.98:0/2493201860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e08103e30 0x7f6e08104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 -- 172.21.15.98:0/2493201860 >> 172.21.15.98:0/2493201860 conn(0x7f6e080ff8a0 msgr2=0x7f6e08101d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 -- 172.21.15.98:0/2493201860 shutdown_connections 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 -- 172.21.15.98:0/2493201860 wait complete. 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 Processor -- start 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.794+0000 7f6e10509700 1 -- start start 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e10509700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e08103e30 0x7f6e0819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e10509700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e081048b0 0x7f6e0819acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e10509700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e08105880 0x7f6e081a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e10509700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0810fc60 con 0x7f6e08105880 2024-04-25T16:32:14.796 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e10509700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0810fae0 con 0x7f6e08103e30 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e10509700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6e0810fde0 con 0x7f6e081048b0 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e08105880 0x7f6e081a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e08105880 0x7f6e081a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41972/0 (socket says 172.21.15.98:41972) 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 -- 172.21.15.98:0/978694542 learned_addr learned my addr 172.21.15.98:0/978694542 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0daa4700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e081048b0 0x7f6e0819acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0e2a5700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e08103e30 0x7f6e0819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 -- 172.21.15.98:0/978694542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e081048b0 msgr2=0x7f6e0819acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e081048b0 0x7f6e0819acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 -- 172.21.15.98:0/978694542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e08103e30 msgr2=0x7f6e0819a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e08103e30 0x7f6e0819a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 -- 172.21.15.98:0/978694542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e081a22d0 con 0x7f6e08105880 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.795+0000 7f6e0eaa6700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e08105880 0x7f6e081a1c50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009030 tx=0x7f6e000097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.796+0000 7f6dff7fe700 1 -- 172.21.15.98:0/978694542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0001f030 con 0x7f6e08105880 2024-04-25T16:32:14.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.796+0000 7f6dff7fe700 1 -- 172.21.15.98:0/978694542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e00005040 con 0x7f6e08105880 2024-04-25T16:32:14.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.796+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e081a2560 con 0x7f6e08105880 2024-04-25T16:32:14.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.796+0000 7f6dff7fe700 1 -- 172.21.15.98:0/978694542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e0001b690 con 0x7f6e08105880 2024-04-25T16:32:14.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.796+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e081a2a60 con 0x7f6e08105880 2024-04-25T16:32:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.797+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e08051190 con 0x7f6e08105880 2024-04-25T16:32:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.798+0000 7f6dff7fe700 1 -- 172.21.15.98:0/978694542 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6e00022030 con 0x7f6e08105880 2024-04-25T16:32:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.799+0000 7f6dff7fe700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6df40afc00 0x7f6df40b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.799+0000 7f6e0e2a5700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6df40afc00 0x7f6df40b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.799+0000 7f6dff7fe700 1 -- 172.21.15.98:0/978694542 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f6e0000f020 con 0x7f6e08105880 2024-04-25T16:32:14.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.800+0000 7f6e0e2a5700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6df40afc00 0x7f6df40b2110 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6e08105270 tx=0x7f6e04007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:14.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:14.803+0000 7f6dff7fe700 1 -- 172.21.15.98:0/978694542 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e00029020 con 0x7f6e08105880 2024-04-25T16:32:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.009+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f6e0802e870 con 0x7f6df40afc00 2024-04-25T16:32:15.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.011+0000 7f6dff7fe700 1 -- 172.21.15.98:0/978694542 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f6e0802e870 con 0x7f6df40afc00 2024-04-25T16:32:15.011 INFO:teuthology.orchestra.run.smithi098.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T16:32:15.011 INFO:teuthology.orchestra.run.smithi098.stdout:smithi098 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010H400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T16:32:15.011 INFO:teuthology.orchestra.run.smithi098.stdout:smithi173 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204019S400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T16:32:15.011 INFO:teuthology.orchestra.run.smithi098.stdout:smithi203 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GK400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6df40afc00 msgr2=0x7f6df40b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6df40afc00 0x7f6df40b2110 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6e08105270 tx=0x7f6e04007400 comp rx=0 tx=0).stop 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e08105880 msgr2=0x7f6e081a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e08105880 0x7f6e081a1c50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6e00009030 tx=0x7f6e000097f0 comp rx=0 tx=0).stop 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 shutdown_connections 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6df40afc00 0x7f6df40b2110 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6e08105880 0x7f6e081a1c50 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e08103e30 0x7f6e0819a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 --2- 172.21.15.98:0/978694542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e081048b0 0x7f6e0819acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 >> 172.21.15.98:0/978694542 conn(0x7f6e080ff8a0 msgr2=0x7f6e081085a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 shutdown_connections 2024-04-25T16:32:15.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:15.013+0000 7f6e10509700 1 -- 172.21.15.98:0/978694542 wait complete. 2024-04-25T16:32:15.723 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T16:32:15.733 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T16:32:15.733 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T16:32:15.987 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:32:16.375 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:16 smithi098 ceph-mon[29162]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:16.375 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:16 smithi098 ceph-mon[29162]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:16.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:16 smithi203 ceph-mon[33115]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:16.470 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:16 smithi203 ceph-mon[33115]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:16.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:16 smithi173 ceph-mon[32556]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:16.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:16 smithi173 ceph-mon[32556]: pgmap v349: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 -- 172.21.15.98:0/3836057087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954104210 msgr2=0x7fd954104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 --2- 172.21.15.98:0/3836057087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954104210 0x7fd954104620 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd948009990 tx=0x7fd948009ca0 comp rx=0 tx=0).stop 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 -- 172.21.15.98:0/3836057087 shutdown_connections 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 --2- 172.21.15.98:0/3836057087 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd954105ce0 0x7fd95410c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 --2- 172.21.15.98:0/3836057087 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954104210 0x7fd954104620 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 --2- 172.21.15.98:0/3836057087 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd954105340 0x7fd954105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 -- 172.21.15.98:0/3836057087 >> 172.21.15.98:0/3836057087 conn(0x7fd9540ff850 msgr2=0x7fd954101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 -- 172.21.15.98:0/3836057087 shutdown_connections 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.746+0000 7fd959235700 1 -- 172.21.15.98:0/3836057087 wait complete. 2024-04-25T16:32:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.747+0000 7fd959235700 1 Processor -- start 2024-04-25T16:32:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd959235700 1 -- start start 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd959235700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd954104210 0x7fd954077330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd959235700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd954105340 0x7fd9540759e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd959235700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954105ce0 0x7fd954075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd959235700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd95410fb10 con 0x7fd954105ce0 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd959235700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd95410f990 con 0x7fd954105340 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd959235700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd95410fc90 con 0x7fd954104210 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd95359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954105ce0 0x7fd954075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd95359e700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954105ce0 0x7fd954075f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52306/0 (socket says 172.21.15.98:52306) 2024-04-25T16:32:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd95359e700 1 -- 172.21.15.98:0/630588748 learned_addr learned my addr 172.21.15.98:0/630588748 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd95259c700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd954105340 0x7fd9540759e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd952d9d700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd954104210 0x7fd954077330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd95359e700 1 -- 172.21.15.98:0/630588748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd954104210 msgr2=0x7fd954077330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd95359e700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd954104210 0x7fd954077330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd95359e700 1 -- 172.21.15.98:0/630588748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd954105340 msgr2=0x7fd9540759e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.748+0000 7fd95359e700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd954105340 0x7fd9540759e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.749+0000 7fd95359e700 1 -- 172.21.15.98:0/630588748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd954076680 con 0x7fd954105ce0 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.749+0000 7fd95359e700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954105ce0 0x7fd954075f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd94c007ee0 tx=0x7fd94c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.749+0000 7fd93bfff700 1 -- 172.21.15.98:0/630588748 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94c004bb0 con 0x7fd954105ce0 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.749+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9541a47e0 con 0x7fd954105ce0 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.749+0000 7fd93bfff700 1 -- 172.21.15.98:0/630588748 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd94c004d50 con 0x7fd954105ce0 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.749+0000 7fd93bfff700 1 -- 172.21.15.98:0/630588748 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd94c0058b0 con 0x7fd954105ce0 2024-04-25T16:32:17.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.749+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9541a4cc0 con 0x7fd954105ce0 2024-04-25T16:32:17.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.750+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd954051190 con 0x7fd954105ce0 2024-04-25T16:32:17.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.751+0000 7fd93bfff700 1 -- 172.21.15.98:0/630588748 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd94c01a020 con 0x7fd954105ce0 2024-04-25T16:32:17.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.752+0000 7fd93bfff700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd93c0afb80 0x7fd93c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:17.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.752+0000 7fd952d9d700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd93c0afb80 0x7fd93c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:17.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.752+0000 7fd93bfff700 1 -- 172.21.15.98:0/630588748 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fd94c0ded10 con 0x7fd954105ce0 2024-04-25T16:32:17.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.753+0000 7fd952d9d700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd93c0afb80 0x7fd93c0b2090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd948006010 tx=0x7fd94801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:17.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:17.756+0000 7fd93bfff700 1 -- 172.21.15.98:0/630588748 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd94c010070 con 0x7fd954105ce0 2024-04-25T16:32:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.002+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fd95402e510 con 0x7fd954105ce0 2024-04-25T16:32:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.002+0000 7fd93bfff700 1 -- 172.21.15.98:0/630588748 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) ==== 76+0+3094 (secure 0 0 0) 0x7fd94c0a7540 con 0x7fd954105ce0 2024-04-25T16:32:18.002 INFO:teuthology.orchestra.run.smithi098.stdout:e28 2024-04-25T16:32:18.002 INFO:teuthology.orchestra.run.smithi098.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T16:32:18.002 INFO:teuthology.orchestra.run.smithi098.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:32:18.002 INFO:teuthology.orchestra.run.smithi098.stdout:legacy client fscid: 1 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:Filesystem 'cephfs' (1) 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:fs_name cephfs 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 28 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-04-25T16:30:09.342794+0000 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-04-25T16:30:38.051368+0000 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:tableserver 0 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:root 0 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:session_timeout 300 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:session_autoclose 300 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:max_file_size 1099511627776 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:max_xattr_size 65536 2024-04-25T16:32:18.003 INFO:teuthology.orchestra.run.smithi098.stdout:required_client_features {} 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure 0 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:last_failure_osd_epoch 0 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:max_mds 5 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:in 0,1,2,3,4 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:up {0=24539,1=24494,2=24484,3=24473,4=24457} 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:failed 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:damaged 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:stopped 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:data_pools [3,4] 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:metadata_pool 2 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:inline_data disabled 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:balancer 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:bal_rank_mask -1 2024-04-25T16:32:18.004 INFO:teuthology.orchestra.run.smithi098.stdout:standby_count_wanted 0 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:qdb_cluster leader: 24539 members: 24484,24539,24457,24473,24494 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.b{0:24539} state up:active seq 3 addr [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.f{0:24445} state up:standby-replay seq 1 addr [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.h{1:24494} state up:active seq 12 addr [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.k{1:24443} state up:standby-replay seq 1 addr [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.i{2:24484} state up:active seq 11 addr [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.l{2:24415} state up:standby-replay seq 1 addr [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.e{3:24473} state up:active seq 14 addr [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.a{3:14646} state up:standby-replay seq 1 addr [v2:172.21.15.98:6840/373690634,v1:172.21.15.98:6841/373690634] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.c{4:24457} state up:active seq 13 addr [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.g{4:14634} state up:standby-replay seq 1 addr [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:18.005 INFO:teuthology.orchestra.run.smithi098.stdout:Standby daemons: 2024-04-25T16:32:18.006 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:18.006 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.j{-1:14559} state up:standby seq 1 addr [v2:172.21.15.98:6834/441838145,v1:172.21.15.98:6835/441838145] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.006 INFO:teuthology.orchestra.run.smithi098.stdout:[mds.d{-1:14571} state up:standby seq 1 addr [v2:172.21.15.98:6836/1001942637,v1:172.21.15.98:6837/1001942637] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd93c0afb80 msgr2=0x7fd93c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd93c0afb80 0x7fd93c0b2090 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd948006010 tx=0x7fd94801a040 comp rx=0 tx=0).stop 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954105ce0 msgr2=0x7fd954075f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954105ce0 0x7fd954075f20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd94c007ee0 tx=0x7fd94c00d3b0 comp rx=0 tx=0).stop 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 shutdown_connections 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd93c0afb80 0x7fd93c0b2090 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd954104210 0x7fd954077330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd954105ce0 0x7fd954075f20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 --2- 172.21.15.98:0/630588748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd954105340 0x7fd9540759e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 >> 172.21.15.98:0/630588748 conn(0x7fd9540ff850 msgr2=0x7fd9541085e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 shutdown_connections 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:18.005+0000 7fd959235700 1 -- 172.21.15.98:0/630588748 wait complete. 2024-04-25T16:32:18.007 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-04-25T16:32:18.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:18 smithi098 ceph-mon[29162]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:18.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:18 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/630588748' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:32:18.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:18 smithi203 ceph-mon[33115]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:18.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:18 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/630588748' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:32:18.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:18 smithi173 ceph-mon[32556]: pgmap v350: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:32:18.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:18 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/630588748' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:32:18.701 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T16:32:18.939 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:32:20.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:20 smithi098 ceph-mon[29162]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:20.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:20 smithi203 ceph-mon[33115]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:20.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:20 smithi173 ceph-mon[32556]: pgmap v351: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 -- 172.21.15.98:0/284296534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638103100 msgr2=0x7f0638103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 --2- 172.21.15.98:0/284296534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638103100 0x7f0638103510 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f062c009990 tx=0x7f062c009ca0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 -- 172.21.15.98:0/284296534 shutdown_connections 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 --2- 172.21.15.98:0/284296534 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0638104bd0 0x7f063810b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 --2- 172.21.15.98:0/284296534 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638103100 0x7f0638103510 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 --2- 172.21.15.98:0/284296534 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0638104230 0x7f0638104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 -- 172.21.15.98:0/284296534 >> 172.21.15.98:0/284296534 conn(0x7f06380fe740 msgr2=0x7f0638100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 -- 172.21.15.98:0/284296534 shutdown_connections 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 -- 172.21.15.98:0/284296534 wait complete. 2024-04-25T16:32:20.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 Processor -- start 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 -- start start 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.740+0000 7f063ef0b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0638103100 0x7f063819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063ef0b700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0638104230 0x7f063819ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063ef0b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638104bd0 0x7f06381a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063ef0b700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f063810e9f0 con 0x7f0638104bd0 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063ef0b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f063810e870 con 0x7f0638103100 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063ef0b700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f063810eb70 con 0x7f0638104230 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638104bd0 0x7f06381a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638104bd0 0x7f06381a1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52330/0 (socket says 172.21.15.98:52330) 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 -- 172.21.15.98:0/3422072549 learned_addr learned my addr 172.21.15.98:0/3422072549 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f0637fff700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0638104230 0x7f063819ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:20.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063cca7700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0638103100 0x7f063819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 -- 172.21.15.98:0/3422072549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0638104230 msgr2=0x7f063819ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0638104230 0x7f063819ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 -- 172.21.15.98:0/3422072549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0638103100 msgr2=0x7f063819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0638103100 0x7f063819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 -- 172.21.15.98:0/3422072549 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06381a2260 con 0x7f0638104bd0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063cca7700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0638103100 0x7f063819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063d4a8700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638104bd0 0x7f06381a1be0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0630007ee0 tx=0x7f063000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f0635ffb700 1 -- 172.21.15.98:0/3422072549 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f063000dbd0 con 0x7f0638104bd0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06381a2550 con 0x7f0638104bd0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f0635ffb700 1 -- 172.21.15.98:0/3422072549 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f063000dd70 con 0x7f0638104bd0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.741+0000 7f0635ffb700 1 -- 172.21.15.98:0/3422072549 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06300058d0 con 0x7f0638104bd0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.742+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06381a2a40 con 0x7f0638104bd0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.743+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0638051190 con 0x7f0638104bd0 2024-04-25T16:32:20.755 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.744+0000 7f0635ffb700 1 -- 172.21.15.98:0/3422072549 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f063001a020 con 0x7f0638104bd0 2024-04-25T16:32:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.744+0000 7f0635ffb700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06280afb80 0x7f06280b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.745+0000 7f063cca7700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06280afb80 0x7f06280b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.745+0000 7f0635ffb700 1 -- 172.21.15.98:0/3422072549 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f06300de7b0 con 0x7f0638104bd0 2024-04-25T16:32:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.745+0000 7f063cca7700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06280afb80 0x7f06280b2090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f062c003820 tx=0x7f062c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:20.756 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.749+0000 7f0635ffb700 1 -- 172.21.15.98:0/3422072549 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0630010070 con 0x7f0638104bd0 2024-04-25T16:32:20.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.947+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f06381a2e60 con 0x7f0638104bd0 2024-04-25T16:32:20.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.948+0000 7f0635ffb700 1 -- 172.21.15.98:0/3422072549 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v85) ==== 56+0+5682 (secure 0 0 0) 0x7f06300a6fe0 con 0x7f0638104bd0 2024-04-25T16:32:20.947 INFO:teuthology.orchestra.run.smithi098.stdout:epoch 85 2024-04-25T16:32:20.947 INFO:teuthology.orchestra.run.smithi098.stdout:fsid c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:32:20.947 INFO:teuthology.orchestra.run.smithi098.stdout:created 2024-04-25T16:22:09.449249+0000 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:modified 2024-04-25T16:32:10.010488+0000 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:crush_version 27 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:full_ratio 0.95 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:backfillfull_ratio 0.9 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:nearfull_ratio 0.85 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:require_min_compat_client luminous 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:min_compat_client luminous 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:require_osd_release squid 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:stretch_mode_enabled false 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 77 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-25T16:32:20.948 INFO:teuthology.orchestra.run.smithi098.stdout:max_osd 12 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 up in weight 1 up_from 8 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6802/2011959581,v1:172.21.15.98:6803/2011959581] [v2:172.21.15.98:6804/2011959581,v1:172.21.15.98:6805/2011959581] exists,up 738a5e3f-5992-4c5f-befd-a6130241c120 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 up in weight 1 up_from 13 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6800/2144972305,v1:172.21.15.173:6801/2144972305] [v2:172.21.15.173:6802/2144972305,v1:172.21.15.173:6803/2144972305] exists,up 1d963fea-ee1e-4f05-af8e-31a28d82c49c 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 up in weight 1 up_from 18 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6800/3173578158,v1:172.21.15.203:6801/3173578158] [v2:172.21.15.203:6802/3173578158,v1:172.21.15.203:6803/3173578158] exists,up ea05c891-ec27-45e1-9226-2f3d5752a73f 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 up in weight 1 up_from 26 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6810/214861667,v1:172.21.15.98:6811/214861667] [v2:172.21.15.98:6812/214861667,v1:172.21.15.98:6813/214861667] exists,up fb6d74f9-c730-4a72-a11f-141d71e4ba71 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 up in weight 1 up_from 32 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6808/1101244004,v1:172.21.15.173:6809/1101244004] [v2:172.21.15.173:6810/1101244004,v1:172.21.15.173:6811/1101244004] exists,up 7375471d-1a21-428b-855f-8d9cd5f5e5fb 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 up in weight 1 up_from 37 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6808/4007044064,v1:172.21.15.203:6809/4007044064] [v2:172.21.15.203:6810/4007044064,v1:172.21.15.203:6811/4007044064] exists,up 61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 up in weight 1 up_from 43 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6818/3921819389,v1:172.21.15.98:6819/3921819389] [v2:172.21.15.98:6820/3921819389,v1:172.21.15.98:6821/3921819389] exists,up 89aa247d-c76d-4fa9-a368-bc45954b8b3f 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 up in weight 1 up_from 48 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6816/442659395,v1:172.21.15.173:6817/442659395] [v2:172.21.15.173:6818/442659395,v1:172.21.15.173:6819/442659395] exists,up c4a90448-6dcb-4dd1-aec5-cc423090bdf0 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.8 up in weight 1 up_from 54 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6816/3847547758,v1:172.21.15.203:6817/3847547758] [v2:172.21.15.203:6818/3847547758,v1:172.21.15.203:6819/3847547758] exists,up 921fce6a-8fac-4547-b46a-b177454e3b55 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.9 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.98:6826/3292339123,v1:172.21.15.98:6827/3292339123] [v2:172.21.15.98:6828/3292339123,v1:172.21.15.98:6829/3292339123] exists,up 53d9c2d6-bded-41da-a30b-7b4d2c44b55a 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.10 up in weight 1 up_from 66 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.173:6824/3946840322,v1:172.21.15.173:6825/3946840322] [v2:172.21.15.173:6826/3946840322,v1:172.21.15.173:6827/3946840322] exists,up 18d5642e-ef6c-41dd-8aa4-9fd5190d03f7 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:osd.11 up in weight 1 up_from 72 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.203:6824/1060778865,v1:172.21.15.203:6825/1060778865] [v2:172.21.15.203:6826/1060778865,v1:172.21.15.203:6827/1060778865] exists,up 09b2102c-57de-477e-9a48-008e84d76db9 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.6 [8,11] 2024-04-25T16:32:20.949 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.b [2,4] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.e [2,1] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 2.1d [8,11] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.12 [3,9] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.1b [4,8] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.36 [4,9] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.39 [5,6] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 3.3c [4,9] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.5 [5,10] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.6 [9,8] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.13 [9,8] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:pg_upmap_items 4.18 [9,8] 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1883089564 expires 2024-04-26T16:23:07.830236+0000 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/119631250 expires 2024-04-26T16:23:07.830236+0000 2024-04-25T16:32:20.950 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/217035597 expires 2024-04-26T16:23:07.830236+0000 2024-04-25T16:32:20.951 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/2573630616 expires 2024-04-26T16:23:07.830236+0000 2024-04-25T16:32:20.951 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/4161723298 expires 2024-04-26T16:22:41.266215+0000 2024-04-25T16:32:20.951 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3681298222 expires 2024-04-26T16:22:41.266215+0000 2024-04-25T16:32:20.951 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6801/4161723298 expires 2024-04-26T16:22:41.266215+0000 2024-04-25T16:32:20.951 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/1429360204 expires 2024-04-26T16:22:41.266215+0000 2024-04-25T16:32:20.951 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:0/3090764993 expires 2024-04-26T16:22:41.266215+0000 2024-04-25T16:32:20.951 INFO:teuthology.orchestra.run.smithi098.stdout:blocklist 172.21.15.98:6800/217035597 expires 2024-04-26T16:23:07.830236+0000 2024-04-25T16:32:20.951 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06280afb80 msgr2=0x7f06280b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06280afb80 0x7f06280b2090 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f062c003820 tx=0x7f062c005fb0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638104bd0 msgr2=0x7f06381a1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638104bd0 0x7f06381a1be0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0630007ee0 tx=0x7f063000d3b0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 shutdown_connections 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06280afb80 0x7f06280b2090 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0638104bd0 0x7f06381a1be0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0638103100 0x7f063819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 --2- 172.21.15.98:0/3422072549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0638104230 0x7f063819ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 >> 172.21.15.98:0/3422072549 conn(0x7f06380fe740 msgr2=0x7f0638100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 shutdown_connections 2024-04-25T16:32:20.952 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:20.951+0000 7f063ef0b700 1 -- 172.21.15.98:0/3422072549 wait complete. 2024-04-25T16:32:21.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:21 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3422072549' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:32:21.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:21 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3422072549' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:32:21.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:21 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3422072549' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:32:21.652 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T16:32:21.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 -- 172.21.15.98:0/3596039257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8103b10 msgr2=0x7fbba8104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/3596039257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8103b10 0x7fbba8104810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009900 tx=0x7fbba402a410 comp rx=0 tx=0).stop 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 -- 172.21.15.98:0/3596039257 shutdown_connections 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/3596039257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbba8109a50 0x7fbba810be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/3596039257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbba8104d50 0x7fbba8105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/3596039257 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8103b10 0x7fbba8104810 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 -- 172.21.15.98:0/3596039257 >> 172.21.15.98:0/3596039257 conn(0x7fbba80fed40 msgr2=0x7fbba81011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 -- 172.21.15.98:0/3596039257 shutdown_connections 2024-04-25T16:32:21.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.776+0000 7fbbb07f3640 1 -- 172.21.15.98:0/3596039257 wait complete. 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbb07f3640 1 Processor -- start 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbb07f3640 1 -- start start 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbb07f3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbba8103b10 0x7fbba819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbb07f3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbba8104d50 0x7fbba819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbb07f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8109a50 0x7fbba81a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbb07f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fbba810f4a0 con 0x7fbba8109a50 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbb07f3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fbba810f320 con 0x7fbba8103b10 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbb07f3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fbba810f620 con 0x7fbba8104d50 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbaed90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8109a50 0x7fbba81a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbaed90640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8109a50 0x7fbba81a1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52358/0 (socket says 172.21.15.98:52358) 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbaed90640 1 -- 172.21.15.98:0/2409982695 learned_addr learned my addr 172.21.15.98:0/2409982695 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbae58f640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbba8103b10 0x7fbba819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbadd8e640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbba8104d50 0x7fbba819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:21.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbaed90640 1 -- 172.21.15.98:0/2409982695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbba8104d50 msgr2=0x7fbba819a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbaed90640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbba8104d50 0x7fbba819a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbaed90640 1 -- 172.21.15.98:0/2409982695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbba8103b10 msgr2=0x7fbba819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbaed90640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbba8103b10 0x7fbba819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.777+0000 7fbbaed90640 1 -- 172.21.15.98:0/2409982695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba81a1ca0 con 0x7fbba8109a50 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.778+0000 7fbbaed90640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8109a50 0x7fbba81a1630 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbba000eaa0 tx=0x7fbba000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.778+0000 7fbb9f7fe640 1 -- 172.21.15.98:0/2409982695 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba000cdb0 con 0x7fbba8109a50 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.778+0000 7fbb9f7fe640 1 -- 172.21.15.98:0/2409982695 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbba0005020 con 0x7fbba8109a50 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.778+0000 7fbb9f7fe640 1 -- 172.21.15.98:0/2409982695 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba00116d0 con 0x7fbba8109a50 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.778+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba81a1f90 con 0x7fbba8109a50 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.778+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba81a23a0 con 0x7fbba8109a50 2024-04-25T16:32:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.779+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba8108860 con 0x7fbba8109a50 2024-04-25T16:32:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.780+0000 7fbb9f7fe640 1 -- 172.21.15.98:0/2409982695 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbba00051d0 con 0x7fbba8109a50 2024-04-25T16:32:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.781+0000 7fbb9f7fe640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbb940afaf0 0x7fbb940b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.781+0000 7fbbae58f640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbb940afaf0 0x7fbb940b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.781+0000 7fbb9f7fe640 1 -- 172.21.15.98:0/2409982695 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fbba00df3f0 con 0x7fbba8109a50 2024-04-25T16:32:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.781+0000 7fbbae58f640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbb940afaf0 0x7fbb940b2000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbba4005760 tx=0x7fbba4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:21.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.784+0000 7fbb9f7fe640 1 -- 172.21.15.98:0/2409982695 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbba0016030 con 0x7fbba8109a50 2024-04-25T16:32:21.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.962+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbba802dd20 con 0x7fbba8109a50 2024-04-25T16:32:21.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.964+0000 7fbb9f7fe640 1 -- 172.21.15.98:0/2409982695 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24806 (secure 0 0 0) 0x7fbba00a7610 con 0x7fbba8109a50 2024-04-25T16:32:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:21.964 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:32:10.010488+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbb940afaf0 msgr2=0x7fbb940b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbb940afaf0 0x7fbb940b2000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbba4005760 tx=0x7fbba4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8109a50 msgr2=0x7fbba81a1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8109a50 0x7fbba81a1630 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbba000eaa0 tx=0x7fbba000ef60 comp rx=0 tx=0).stop 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 shutdown_connections 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fbb940afaf0 0x7fbb940b2000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fbba8109a50 0x7fbba81a1630 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbba8104d50 0x7fbba819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 --2- 172.21.15.98:0/2409982695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbba8103b10 0x7fbba819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:21.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 >> 172.21.15.98:0/2409982695 conn(0x7fbba80fed40 msgr2=0x7fbba8105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:21.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 shutdown_connections 2024-04-25T16:32:21.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:21.966+0000 7fbbb07f3640 1 -- 172.21.15.98:0/2409982695 wait complete. 2024-04-25T16:32:21.978 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:32:21.979 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 -- 172.21.15.98:0/338386161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e741076a0 msgr2=0x7f9e74107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/338386161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e741076a0 0x7f9e74107ab0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c009ca0 tx=0x7f9e6c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 -- 172.21.15.98:0/338386161 shutdown_connections 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/338386161 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e741076a0 0x7f9e74107ab0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/338386161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e74104d40 0x7f9e74107160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/338386161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e741023e0 0x7f9e74104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 -- 172.21.15.98:0/338386161 >> 172.21.15.98:0/338386161 conn(0x7f9e740fdbf0 msgr2=0x7f9e741000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 -- 172.21.15.98:0/338386161 shutdown_connections 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.095+0000 7f9e7bee2640 1 -- 172.21.15.98:0/338386161 wait complete. 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7bee2640 1 Processor -- start 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7bee2640 1 -- start start 2024-04-25T16:32:22.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7bee2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e741023e0 0x7f9e741a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7bee2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74104d40 0x7f9e741a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7bee2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e741076a0 0x7f9e741a7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7bee2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9e74115b80 con 0x7f9e74104d40 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7bee2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9e74115a00 con 0x7f9e741023e0 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7bee2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9e74115d00 con 0x7f9e741076a0 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7947d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74104d40 0x7f9e741a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7947d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74104d40 0x7f9e741a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52370/0 (socket says 172.21.15.98:52370) 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7947d640 1 -- 172.21.15.98:0/3072445186 learned_addr learned my addr 172.21.15.98:0/3072445186 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e79c7e640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e741023e0 0x7f9e741a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7a47f640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e741076a0 0x7f9e741a7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7947d640 1 -- 172.21.15.98:0/3072445186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e741076a0 msgr2=0x7f9e741a7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7947d640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e741076a0 0x7f9e741a7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7947d640 1 -- 172.21.15.98:0/3072445186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e741023e0 msgr2=0x7f9e741a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7947d640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e741023e0 0x7f9e741a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7947d640 1 -- 172.21.15.98:0/3072445186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e741a82c0 con 0x7f9e74104d40 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.096+0000 7f9e7a47f640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e741076a0 0x7f9e741a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.097+0000 7f9e7947d640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74104d40 0x7f9e741a0cc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400b4a0 tx=0x7f9e6400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.097+0000 7f9e6affd640 1 -- 172.21.15.98:0/3072445186 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64014070 con 0x7f9e74104d40 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.097+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e741a85b0 con 0x7f9e74104d40 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.097+0000 7f9e6affd640 1 -- 172.21.15.98:0/3072445186 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9e64010aa0 con 0x7f9e74104d40 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.097+0000 7f9e6affd640 1 -- 172.21.15.98:0/3072445186 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64017b70 con 0x7f9e74104d40 2024-04-25T16:32:22.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.097+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e741a8ab0 con 0x7f9e74104d40 2024-04-25T16:32:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.098+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e740ff710 con 0x7f9e74104d40 2024-04-25T16:32:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.099+0000 7f9e6affd640 1 -- 172.21.15.98:0/3072445186 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9e64017070 con 0x7f9e74104d40 2024-04-25T16:32:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.100+0000 7f9e6affd640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9e600afaf0 0x7f9e600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.100+0000 7f9e79c7e640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9e600afaf0 0x7f9e600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.100+0000 7f9e6affd640 1 -- 172.21.15.98:0/3072445186 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f9e640de850 con 0x7f9e74104d40 2024-04-25T16:32:22.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.100+0000 7f9e79c7e640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9e600afaf0 0x7f9e600b2000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9e70005fd0 tx=0x7f9e70005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:22.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.103+0000 7f9e6affd640 1 -- 172.21.15.98:0/3072445186 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e640a7080 con 0x7f9e74104d40 2024-04-25T16:32:22.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:22 smithi098 ceph-mon[29162]: pgmap v352: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:22.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:22 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2409982695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:22.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.286+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9e740637a0 con 0x7f9e74104d40 2024-04-25T16:32:22.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.286+0000 7f9e6affd640 1 -- 172.21.15.98:0/3072445186 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f9e640af100 con 0x7f9e74104d40 2024-04-25T16:32:22.286 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9e600afaf0 msgr2=0x7f9e600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9e600afaf0 0x7f9e600b2000 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9e70005fd0 tx=0x7f9e70005850 comp rx=0 tx=0).stop 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74104d40 msgr2=0x7f9e741a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74104d40 0x7f9e741a0cc0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400b4a0 tx=0x7f9e6400b960 comp rx=0 tx=0).stop 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 shutdown_connections 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9e600afaf0 0x7f9e600b2000 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9e741076a0 0x7f9e741a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9e74104d40 0x7f9e741a0cc0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 --2- 172.21.15.98:0/3072445186 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9e741023e0 0x7f9e741a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 >> 172.21.15.98:0/3072445186 conn(0x7f9e740fdbf0 msgr2=0x7f9e74103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:22.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 shutdown_connections 2024-04-25T16:32:22.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.288+0000 7f9e7bee2640 1 -- 172.21.15.98:0/3072445186 wait complete. 2024-04-25T16:32:22.299 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:32:22.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 -- 172.21.15.98:0/3897236447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3448103c20 msgr2=0x7f3448104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 --2- 172.21.15.98:0/3897236447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3448103c20 0x7f3448104030 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3438009900 tx=0x7f343802a410 comp rx=0 tx=0).stop 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 -- 172.21.15.98:0/3897236447 shutdown_connections 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 --2- 172.21.15.98:0/3897236447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3448108920 0x7f344810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 --2- 172.21.15.98:0/3897236447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3448103c20 0x7f3448104030 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 --2- 172.21.15.98:0/3897236447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34481029e0 0x7f34481036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 -- 172.21.15.98:0/3897236447 >> 172.21.15.98:0/3897236447 conn(0x7f34480fdbd0 msgr2=0x7f3448100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 -- 172.21.15.98:0/3897236447 shutdown_connections 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 -- 172.21.15.98:0/3897236447 wait complete. 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.411+0000 7f344d89f640 1 Processor -- start 2024-04-25T16:32:22.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f344d89f640 1 -- start start 2024-04-25T16:32:22.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f344d89f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34481029e0 0x7f344819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f344d89f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3448103c20 0x7f344819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f344d89f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3448108920 0x7f34481a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f344d89f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f344810e390 con 0x7f34481029e0 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f344d89f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f344810e210 con 0x7f3448103c20 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f344d89f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f344810e510 con 0x7f3448108920 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34481029e0 0x7f344819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34481029e0 0x7f344819a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52384/0 (socket says 172.21.15.98:52384) 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 -- 172.21.15.98:0/1919856819 learned_addr learned my addr 172.21.15.98:0/1919856819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f34467fc640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3448103c20 0x7f344819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f34477fe640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3448108920 0x7f34481a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 -- 172.21.15.98:0/1919856819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3448108920 msgr2=0x7f34481a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3448108920 0x7f34481a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 -- 172.21.15.98:0/1919856819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3448103c20 msgr2=0x7f344819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3448103c20 0x7f344819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 -- 172.21.15.98:0/1919856819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34481a1f90 con 0x7f34481029e0 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f34477fe640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3448108920 0x7f34481a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f34467fc640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3448103c20 0x7f344819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.412+0000 7f3446ffd640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34481029e0 0x7f344819a450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c970 tx=0x7f343c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.413+0000 7f344c89d640 1 -- 172.21.15.98:0/1919856819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c004220 con 0x7f34481029e0 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.413+0000 7f344c89d640 1 -- 172.21.15.98:0/1919856819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f343c007e40 con 0x7f34481029e0 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.413+0000 7f344c89d640 1 -- 172.21.15.98:0/1919856819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c00f450 con 0x7f34481029e0 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.413+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34481a2280 con 0x7f34481029e0 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.413+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34481a27d0 con 0x7f34481029e0 2024-04-25T16:32:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.414+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344806c2a0 con 0x7f34481029e0 2024-04-25T16:32:22.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.415+0000 7f344c89d640 1 -- 172.21.15.98:0/1919856819 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f343c0079b0 con 0x7f34481029e0 2024-04-25T16:32:22.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.416+0000 7f344c89d640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34300afaf0 0x7f34300b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.416+0000 7f34467fc640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34300afaf0 0x7f34300b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.416+0000 7f344c89d640 1 -- 172.21.15.98:0/1919856819 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f343c0df120 con 0x7f34481029e0 2024-04-25T16:32:22.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.417+0000 7f34467fc640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34300afaf0 0x7f34300b2000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3438006010 tx=0x7f3438031420 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:22.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.420+0000 7f344c89d640 1 -- 172.21.15.98:0/1919856819 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f343c014030 con 0x7f34481029e0 2024-04-25T16:32:22.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:22 smithi203 ceph-mon[33115]: pgmap v352: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:22.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:22 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2409982695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:22.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:22 smithi173 ceph-mon[32556]: pgmap v352: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:22.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:22 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2409982695' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.597+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f34481a2a90 con 0x7f34481029e0 2024-04-25T16:32:22.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.597+0000 7f344c89d640 1 -- 172.21.15.98:0/1919856819 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f343c0a7950 con 0x7f34481029e0 2024-04-25T16:32:22.597 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34300afaf0 msgr2=0x7f34300b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34300afaf0 0x7f34300b2000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3438006010 tx=0x7f3438031420 comp rx=0 tx=0).stop 2024-04-25T16:32:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34481029e0 msgr2=0x7f344819a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34481029e0 0x7f344819a450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c970 tx=0x7f343c00ce30 comp rx=0 tx=0).stop 2024-04-25T16:32:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 shutdown_connections 2024-04-25T16:32:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f34300afaf0 0x7f34300b2000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3448108920 0x7f34481a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3448103c20 0x7f344819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 --2- 172.21.15.98:0/1919856819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f34481029e0 0x7f344819a450 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.599+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 >> 172.21.15.98:0/1919856819 conn(0x7f34480fdbd0 msgr2=0x7f3448109e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:22.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.600+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 shutdown_connections 2024-04-25T16:32:22.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.600+0000 7f344d89f640 1 -- 172.21.15.98:0/1919856819 wait complete. 2024-04-25T16:32:22.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:32:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.721+0000 7f9bc7861640 1 -- 172.21.15.98:0/1915587636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0102f50 msgr2=0x7f9bc0103360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.721+0000 7f9bc7861640 1 --2- 172.21.15.98:0/1915587636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0102f50 0x7f9bc0103360 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8009900 tx=0x7f9bb802a410 comp rx=0 tx=0).stop 2024-04-25T16:32:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.721+0000 7f9bc7861640 1 -- 172.21.15.98:0/1915587636 shutdown_connections 2024-04-25T16:32:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.721+0000 7f9bc7861640 1 --2- 172.21.15.98:0/1915587636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bc0104470 0x7f9bc010abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.721+0000 7f9bc7861640 1 --2- 172.21.15.98:0/1915587636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc0103930 0x7f9bc0103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.721+0000 7f9bc7861640 1 --2- 172.21.15.98:0/1915587636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0102f50 0x7f9bc0103360 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.721+0000 7f9bc7861640 1 -- 172.21.15.98:0/1915587636 >> 172.21.15.98:0/1915587636 conn(0x7f9bc00fd350 msgr2=0x7f9bc00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:22.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 -- 172.21.15.98:0/1915587636 shutdown_connections 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 -- 172.21.15.98:0/1915587636 wait complete. 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 Processor -- start 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 -- start start 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc0102f50 0x7f9bc019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bc0103930 0x7f9bc019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0104470 0x7f9bc01a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc010e3e0 con 0x7f9bc0104470 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc010e260 con 0x7f9bc0102f50 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.722+0000 7f9bc7861640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9bc010e560 con 0x7f9bc0103930 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0104470 0x7f9bc01a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0104470 0x7f9bc01a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52410/0 (socket says 172.21.15.98:52410) 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 -- 172.21.15.98:0/845865195 learned_addr learned my addr 172.21.15.98:0/845865195 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc4dfc640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bc0103930 0x7f9bc019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc55fd640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc0102f50 0x7f9bc019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 -- 172.21.15.98:0/845865195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bc0103930 msgr2=0x7f9bc019a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bc0103930 0x7f9bc019a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 -- 172.21.15.98:0/845865195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc0102f50 msgr2=0x7f9bc019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc0102f50 0x7f9bc019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 -- 172.21.15.98:0/845865195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bc01a1c00 con 0x7f9bc0104470 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc5dfe640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0104470 0x7f9bc01a1640 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000cce0 tx=0x7f9bb000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bbe7fc640 1 -- 172.21.15.98:0/845865195 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb000f450 con 0x7f9bc0104470 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bbe7fc640 1 -- 172.21.15.98:0/845865195 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9bb000fa80 con 0x7f9bc0104470 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bbe7fc640 1 -- 172.21.15.98:0/845865195 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0018ad0 con 0x7f9bc0104470 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bc01a1ef0 con 0x7f9bc0104470 2024-04-25T16:32:22.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.723+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bc01a2300 con 0x7f9bc0104470 2024-04-25T16:32:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.724+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bc01a25c0 con 0x7f9bc0104470 2024-04-25T16:32:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.726+0000 7f9bbe7fc640 1 -- 172.21.15.98:0/845865195 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9bb000f5f0 con 0x7f9bc0104470 2024-04-25T16:32:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.726+0000 7f9bbe7fc640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9ba80afaf0 0x7f9ba80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:22.726 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.726+0000 7f9bc55fd640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9ba80afaf0 0x7f9ba80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.727+0000 7f9bbe7fc640 1 -- 172.21.15.98:0/845865195 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f9bb00df780 con 0x7f9bc0104470 2024-04-25T16:32:22.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.727+0000 7f9bc55fd640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9ba80afaf0 0x7f9ba80b2000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9bb802a930 tx=0x7f9bb8031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:22.729 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.730+0000 7f9bbe7fc640 1 -- 172.21.15.98:0/845865195 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9bb0014030 con 0x7f9bc0104470 2024-04-25T16:32:22.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.905+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9bc01a2790 con 0x7f9bc0104470 2024-04-25T16:32:22.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.906+0000 7f9bbe7fc640 1 -- 172.21.15.98:0/845865195 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f9bb00a8040 con 0x7f9bc0104470 2024-04-25T16:32:22.905 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:32:22.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9ba80afaf0 msgr2=0x7f9ba80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9ba80afaf0 0x7f9ba80b2000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9bb802a930 tx=0x7f9bb8031400 comp rx=0 tx=0).stop 2024-04-25T16:32:22.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0104470 msgr2=0x7f9bc01a1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:22.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0104470 0x7f9bc01a1640 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000cce0 tx=0x7f9bb000a620 comp rx=0 tx=0).stop 2024-04-25T16:32:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 shutdown_connections 2024-04-25T16:32:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9ba80afaf0 0x7f9ba80b2000 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9bc0104470 0x7f9bc01a1640 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9bc0103930 0x7f9bc019a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 --2- 172.21.15.98:0/845865195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9bc0102f50 0x7f9bc019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 >> 172.21.15.98:0/845865195 conn(0x7f9bc00fd350 msgr2=0x7f9bc0109d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 shutdown_connections 2024-04-25T16:32:22.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:22.908+0000 7f9bc7861640 1 -- 172.21.15.98:0/845865195 wait complete. 2024-04-25T16:32:22.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 -- 172.21.15.98:0/2792905999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc188103c20 msgr2=0x7fc188104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 --2- 172.21.15.98:0/2792905999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc188103c20 0x7fc188104030 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009900 tx=0x7fc17c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 -- 172.21.15.98:0/2792905999 shutdown_connections 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 --2- 172.21.15.98:0/2792905999 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188108920 0x7fc18810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 --2- 172.21.15.98:0/2792905999 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc188103c20 0x7fc188104030 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 --2- 172.21.15.98:0/2792905999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1881029e0 0x7fc1881036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 -- 172.21.15.98:0/2792905999 >> 172.21.15.98:0/2792905999 conn(0x7fc1880fdbd0 msgr2=0x7fc188100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 -- 172.21.15.98:0/2792905999 shutdown_connections 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.031+0000 7fc18cc92640 1 -- 172.21.15.98:0/2792905999 wait complete. 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18cc92640 1 Processor -- start 2024-04-25T16:32:23.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18cc92640 1 -- start start 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18cc92640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1881029e0 0x7fc18806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18cc92640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188103c20 0x7fc18819f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18cc92640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc188108920 0x7fc1881a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18cc92640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc18810e390 con 0x7fc1881029e0 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18cc92640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc18810e210 con 0x7fc188108920 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18cc92640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc18810e510 con 0x7fc188103c20 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1881029e0 0x7fc18806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1881029e0 0x7fc18806a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52422/0 (socket says 172.21.15.98:52422) 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 -- 172.21.15.98:0/3653308244 learned_addr learned my addr 172.21.15.98:0/3653308244 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc186d9d640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc188108920 0x7fc1881a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc185d9b640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188103c20 0x7fc18819f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 -- 172.21.15.98:0/3653308244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188103c20 msgr2=0x7fc18819f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188103c20 0x7fc18819f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 -- 172.21.15.98:0/3653308244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc188108920 msgr2=0x7fc1881a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc188108920 0x7fc1881a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 -- 172.21.15.98:0/3653308244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1881a1ed0 con 0x7fc1881029e0 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.032+0000 7fc18659c640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1881029e0 0x7fc18806a900 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc17800c970 tx=0x7fc17800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.033+0000 7fc1777fe640 1 -- 172.21.15.98:0/3653308244 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178007c70 con 0x7fc1881029e0 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.033+0000 7fc1777fe640 1 -- 172.21.15.98:0/3653308244 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc178007e10 con 0x7fc1881029e0 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.033+0000 7fc1777fe640 1 -- 172.21.15.98:0/3653308244 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1780186f0 con 0x7fc1881029e0 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.033+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1881a21c0 con 0x7fc1881029e0 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.033+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1881a2710 con 0x7fc1881029e0 2024-04-25T16:32:23.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.034+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc188107730 con 0x7fc1881029e0 2024-04-25T16:32:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.035+0000 7fc1777fe640 1 -- 172.21.15.98:0/3653308244 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc17800fa90 con 0x7fc1881029e0 2024-04-25T16:32:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.036+0000 7fc1777fe640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1700afaf0 0x7fc1700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.036+0000 7fc185d9b640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1700afaf0 0x7fc1700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.036+0000 7fc1777fe640 1 -- 172.21.15.98:0/3653308244 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fc1780df040 con 0x7fc1881029e0 2024-04-25T16:32:23.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.036+0000 7fc185d9b640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1700afaf0 0x7fc1700b2000 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc18806a2a0 tx=0x7fc17c000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:23.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.039+0000 7fc1777fe640 1 -- 172.21.15.98:0/3653308244 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc178014030 con 0x7fc1881029e0 2024-04-25T16:32:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.219+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc18806c2a0 con 0x7fc1881029e0 2024-04-25T16:32:23.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.219+0000 7fc1777fe640 1 -- 172.21.15.98:0/3653308244 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7fc1780a7900 con 0x7fc1881029e0 2024-04-25T16:32:23.219 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1700afaf0 msgr2=0x7fc1700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1700afaf0 0x7fc1700b2000 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc18806a2a0 tx=0x7fc17c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1881029e0 msgr2=0x7fc18806a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1881029e0 0x7fc18806a900 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc17800c970 tx=0x7fc17800ce30 comp rx=0 tx=0).stop 2024-04-25T16:32:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 shutdown_connections 2024-04-25T16:32:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1700afaf0 0x7fc1700b2000 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc188108920 0x7fc1881a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc188103c20 0x7fc18819f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 --2- 172.21.15.98:0/3653308244 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc1881029e0 0x7fc18806a900 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.221+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 >> 172.21.15.98:0/3653308244 conn(0x7fc1880fdbd0 msgr2=0x7fc188104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.222+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 shutdown_connections 2024-04-25T16:32:23.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.222+0000 7fc18cc92640 1 -- 172.21.15.98:0/3653308244 wait complete. 2024-04-25T16:32:23.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:23 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3072445186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:23 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1919856819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.231 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:23 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/845865195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.233 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:32:23.340 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 -- 172.21.15.98:0/1388617607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588108900 msgr2=0x7f058810ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1388617607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588108900 0x7f058810ad20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0580009ca0 tx=0x7f058002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 -- 172.21.15.98:0/1388617607 shutdown_connections 2024-04-25T16:32:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1388617607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588108900 0x7f058810ad20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1388617607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0588103c00 0x7f0588104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1388617607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f05881029c0 0x7f05881036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 -- 172.21.15.98:0/1388617607 >> 172.21.15.98:0/1388617607 conn(0x7f05880fdbd0 msgr2=0x7f0588100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 -- 172.21.15.98:0/1388617607 shutdown_connections 2024-04-25T16:32:23.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.341+0000 7f058ce0c640 1 -- 172.21.15.98:0/1388617607 wait complete. 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058ce0c640 1 Processor -- start 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058ce0c640 1 -- start start 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058ce0c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05881029c0 0x7f058806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058ce0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588103c00 0x7f058819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058ce0c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0588108900 0x7f05881a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058ce0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f058810e460 con 0x7f0588103c00 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058ce0c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f058810e2e0 con 0x7f05881029c0 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058ce0c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f058810e5e0 con 0x7f0588108900 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0585d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588103c00 0x7f058819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0585d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588103c00 0x7f058819ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52454/0 (socket says 172.21.15.98:52454) 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0585d9b640 1 -- 172.21.15.98:0/1786105928 learned_addr learned my addr 172.21.15.98:0/1786105928 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f058659c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05881029c0 0x7f058806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.344 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0586d9d640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0588108900 0x7f05881a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0585d9b640 1 -- 172.21.15.98:0/1786105928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0588108900 msgr2=0x7f05881a1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0585d9b640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0588108900 0x7f05881a1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0585d9b640 1 -- 172.21.15.98:0/1786105928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05881029c0 msgr2=0x7f058806a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0585d9b640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05881029c0 0x7f058806a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.342+0000 7f0585d9b640 1 -- 172.21.15.98:0/1786105928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05881a1c80 con 0x7f0588103c00 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.343+0000 7f0585d9b640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588103c00 0x7f058819ef10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f057800b4a0 tx=0x7f057800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.343+0000 7f05777fe640 1 -- 172.21.15.98:0/1786105928 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578014070 con 0x7f0588103c00 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.343+0000 7f05777fe640 1 -- 172.21.15.98:0/1786105928 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0578010aa0 con 0x7f0588103c00 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.343+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05881a1f70 con 0x7f0588103c00 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.343+0000 7f05777fe640 1 -- 172.21.15.98:0/1786105928 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578017b70 con 0x7f0588103c00 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.343+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05881a23e0 con 0x7f0588103c00 2024-04-25T16:32:23.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.344+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0588107710 con 0x7f0588103c00 2024-04-25T16:32:23.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.345+0000 7f05777fe640 1 -- 172.21.15.98:0/1786105928 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0578017070 con 0x7f0588103c00 2024-04-25T16:32:23.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.346+0000 7f05777fe640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05700afaf0 0x7f05700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.346+0000 7f058659c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05700afaf0 0x7f05700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.346+0000 7f05777fe640 1 -- 172.21.15.98:0/1786105928 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f05780de610 con 0x7f0588103c00 2024-04-25T16:32:23.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.346+0000 7f058659c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05700afaf0 0x7f05700b2000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f057c005fd0 tx=0x7f057c00b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:23.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.350+0000 7f05777fe640 1 -- 172.21.15.98:0/1786105928 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f05780a67f0 con 0x7f0588103c00 2024-04-25T16:32:23.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:23 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3072445186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:23 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1919856819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:23 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/845865195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:23 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3072445186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:23 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1919856819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:23 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/845865195' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:23.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.566+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f058802db50 con 0x7f0588103c00 2024-04-25T16:32:23.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.567+0000 7f05777fe640 1 -- 172.21.15.98:0/1786105928 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7f05780aaaf0 con 0x7f0588103c00 2024-04-25T16:32:23.566 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:23.567 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:30:38.051368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24484,24539,24457,24473,24494]},"id":1}]} 2024-04-25T16:32:23.567 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-04-25T16:32:23.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05700afaf0 msgr2=0x7f05700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05700afaf0 0x7f05700b2000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f057c005fd0 tx=0x7f057c00b390 comp rx=0 tx=0).stop 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588103c00 msgr2=0x7f058819ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588103c00 0x7f058819ef10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f057800b4a0 tx=0x7f057800b960 comp rx=0 tx=0).stop 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 shutdown_connections 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05700afaf0 0x7f05700b2000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0588108900 0x7f05881a1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0588103c00 0x7f058819ef10 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 --2- 172.21.15.98:0/1786105928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05881029c0 0x7f058806a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 >> 172.21.15.98:0/1786105928 conn(0x7f05880fdbd0 msgr2=0x7f05881048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 shutdown_connections 2024-04-25T16:32:23.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.569+0000 7f058ce0c640 1 -- 172.21.15.98:0/1786105928 wait complete. 2024-04-25T16:32:23.580 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:32:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 -- 172.21.15.98:0/2779220268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df01029c0 msgr2=0x7f0df01036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 --2- 172.21.15.98:0/2779220268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df01029c0 0x7f0df01036c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009ca0 tx=0x7f0de802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 -- 172.21.15.98:0/2779220268 shutdown_connections 2024-04-25T16:32:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 --2- 172.21.15.98:0/2779220268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df0108900 0x7f0df010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 --2- 172.21.15.98:0/2779220268 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0df0103c00 0x7f0df0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 --2- 172.21.15.98:0/2779220268 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df01029c0 0x7f0df01036c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 -- 172.21.15.98:0/2779220268 >> 172.21.15.98:0/2779220268 conn(0x7f0df00fdbd0 msgr2=0x7f0df0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 -- 172.21.15.98:0/2779220268 shutdown_connections 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.688+0000 7f0df4996640 1 -- 172.21.15.98:0/2779220268 wait complete. 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0df4996640 1 Processor -- start 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0df4996640 1 -- start start 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0df4996640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df01029c0 0x7f0df006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0df4996640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df0103c00 0x7f0df019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0df4996640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0df0108900 0x7f0df01a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0df4996640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0df010e4a0 con 0x7f0df0103c00 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0df4996640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0df010e320 con 0x7f0df0108900 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0df4996640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0df010e620 con 0x7f0df01029c0 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0dedd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df0103c00 0x7f0df019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0dedd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df0103c00 0x7f0df019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52474/0 (socket says 172.21.15.98:52474) 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0dedd9b640 1 -- 172.21.15.98:0/65896197 learned_addr learned my addr 172.21.15.98:0/65896197 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0dee59c640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df01029c0 0x7f0df006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.689+0000 7f0deed9d640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0df0108900 0x7f0df01a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0dedd9b640 1 -- 172.21.15.98:0/65896197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df01029c0 msgr2=0x7f0df006a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0dedd9b640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df01029c0 0x7f0df006a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0dedd9b640 1 -- 172.21.15.98:0/65896197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0df0108900 msgr2=0x7f0df01a1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0dedd9b640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0df0108900 0x7f0df01a1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0dedd9b640 1 -- 172.21.15.98:0/65896197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df01a1d50 con 0x7f0df0103c00 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0dedd9b640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df0103c00 0x7f0df019efc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800b4a0 tx=0x7f0dd800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:23.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0de77fe640 1 -- 172.21.15.98:0/65896197 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8014070 con 0x7f0df0103c00 2024-04-25T16:32:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0de77fe640 1 -- 172.21.15.98:0/65896197 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0dd8010aa0 con 0x7f0df0103c00 2024-04-25T16:32:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df01a1fe0 con 0x7f0df0103c00 2024-04-25T16:32:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0de77fe640 1 -- 172.21.15.98:0/65896197 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8017b70 con 0x7f0df0103c00 2024-04-25T16:32:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.690+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df01a2520 con 0x7f0df0103c00 2024-04-25T16:32:23.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.691+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0df0107710 con 0x7f0df0103c00 2024-04-25T16:32:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.692+0000 7f0de77fe640 1 -- 172.21.15.98:0/65896197 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0dd8017070 con 0x7f0df0103c00 2024-04-25T16:32:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.693+0000 7f0de77fe640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0ddc0afaf0 0x7f0ddc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.693+0000 7f0dee59c640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0ddc0afaf0 0x7f0ddc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.693+0000 7f0de77fe640 1 -- 172.21.15.98:0/65896197 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f0dd80de620 con 0x7f0df0103c00 2024-04-25T16:32:23.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.693+0000 7f0dee59c640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0ddc0afaf0 0x7f0ddc0b2000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0de8006010 tx=0x7f0de80318e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:23.695 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.697+0000 7f0de77fe640 1 -- 172.21.15.98:0/65896197 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dd80a6800 con 0x7f0df0103c00 2024-04-25T16:32:23.874 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.875+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0df002db50 con 0x7f0df0103c00 2024-04-25T16:32:23.876 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.877+0000 7f0de77fe640 1 -- 172.21.15.98:0/65896197 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24806 (secure 0 0 0) 0x7f0dd80aab00 con 0x7f0df0103c00 2024-04-25T16:32:23.876 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:23.876 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:32:10.010488+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:32:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0ddc0afaf0 msgr2=0x7f0ddc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0ddc0afaf0 0x7f0ddc0b2000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0de8006010 tx=0x7f0de80318e0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df0103c00 msgr2=0x7f0df019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:23.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df0103c00 0x7f0df019efc0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f0dd800b4a0 tx=0x7f0dd800b960 comp rx=0 tx=0).stop 2024-04-25T16:32:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 shutdown_connections 2024-04-25T16:32:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0ddc0afaf0 0x7f0ddc0b2000 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0df0108900 0x7f0df01a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0df0103c00 0x7f0df019efc0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 --2- 172.21.15.98:0/65896197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0df01029c0 0x7f0df006a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 >> 172.21.15.98:0/65896197 conn(0x7f0df00fdbd0 msgr2=0x7f0df01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 shutdown_connections 2024-04-25T16:32:23.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:23.879+0000 7f0df4996640 1 -- 172.21.15.98:0/65896197 wait complete. 2024-04-25T16:32:23.890 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:32:23.891 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.001+0000 7f6870b1f640 1 -- 172.21.15.98:0/1995818502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c1097a0 msgr2=0x7f686c10bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.001+0000 7f6870b1f640 1 --2- 172.21.15.98:0/1995818502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c1097a0 0x7f686c10bcb0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f6864009ca0 tx=0x7f686402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 -- 172.21.15.98:0/1995818502 shutdown_connections 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 --2- 172.21.15.98:0/1995818502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c1097a0 0x7f686c10bcb0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 --2- 172.21.15.98:0/1995818502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f686c106c30 0x7f686c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 --2- 172.21.15.98:0/1995818502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f686c101e50 0x7f686c102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 -- 172.21.15.98:0/1995818502 >> 172.21.15.98:0/1995818502 conn(0x7f686c0fd380 msgr2=0x7f686c0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 -- 172.21.15.98:0/1995818502 shutdown_connections 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 -- 172.21.15.98:0/1995818502 wait complete. 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 Processor -- start 2024-04-25T16:32:24.004 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.002+0000 7f6870b1f640 1 -- start start 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6870b1f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f686c101e50 0x7f686c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6870b1f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c106c30 0x7f686c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6870b1f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f686c1097a0 0x7f686c1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6870b1f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10f510 con 0x7f686c106c30 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6870b1f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10f390 con 0x7f686c101e50 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6870b1f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f686c10f690 con 0x7f686c1097a0 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6869d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c106c30 0x7f686c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6869d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c106c30 0x7f686c19a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52488/0 (socket says 172.21.15.98:52488) 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f6869d9b640 1 -- 172.21.15.98:0/2055745970 learned_addr learned my addr 172.21.15.98:0/2055745970 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f686a59c640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f686c101e50 0x7f686c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f686ad9d640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f686c1097a0 0x7f686c1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f686a59c640 1 -- 172.21.15.98:0/2055745970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f686c1097a0 msgr2=0x7f686c1a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f686a59c640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f686c1097a0 0x7f686c1a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.005 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f686a59c640 1 -- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c106c30 msgr2=0x7f686c19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f686a59c640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c106c30 0x7f686c19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f686a59c640 1 -- 172.21.15.98:0/2055745970 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f686c1a1b80 con 0x7f686c101e50 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.003+0000 7f686a59c640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f686c101e50 0x7f686c19a0f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f685c00c940 tx=0x7f685c00ce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.004+0000 7f685b7fe640 1 -- 172.21.15.98:0/2055745970 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c007bb0 con 0x7f686c101e50 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.004+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686c1a1e70 con 0x7f686c101e50 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.004+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f686c1a2330 con 0x7f686c101e50 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.004+0000 7f685b7fe640 1 -- 172.21.15.98:0/2055745970 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f685c007d50 con 0x7f686c101e50 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.004+0000 7f685b7fe640 1 -- 172.21.15.98:0/2055745970 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c004d40 con 0x7f686c101e50 2024-04-25T16:32:24.006 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.005+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f684c0052c0 con 0x7f686c101e50 2024-04-25T16:32:24.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.007+0000 7f685b7fe640 1 -- 172.21.15.98:0/2055745970 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f685c004ee0 con 0x7f686c101e50 2024-04-25T16:32:24.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.007+0000 7f685b7fe640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f68540afaf0 0x7f68540b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.007+0000 7f6869d9b640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f68540afaf0 0x7f68540b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.007+0000 7f685b7fe640 1 -- 172.21.15.98:0/2055745970 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f685c0df120 con 0x7f686c101e50 2024-04-25T16:32:24.007 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.008+0000 7f6869d9b640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f68540afaf0 0x7f68540b2000 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6860005fd0 tx=0x7f686000b480 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:24.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.011+0000 7f685b7fe640 1 -- 172.21.15.98:0/2055745970 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f685c014030 con 0x7f686c101e50 2024-04-25T16:32:24.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.190+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f684c005880 con 0x7f686c101e50 2024-04-25T16:32:24.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.191+0000 7f685b7fe640 1 -- 172.21.15.98:0/2055745970 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f685c0a78f0 con 0x7f686c101e50 2024-04-25T16:32:24.191 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:32:24.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f68540afaf0 msgr2=0x7f68540b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f68540afaf0 0x7f68540b2000 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6860005fd0 tx=0x7f686000b480 comp rx=0 tx=0).stop 2024-04-25T16:32:24.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f686c101e50 msgr2=0x7f686c19a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f686c101e50 0x7f686c19a0f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f685c00c940 tx=0x7f685c00ce00 comp rx=0 tx=0).stop 2024-04-25T16:32:24.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 shutdown_connections 2024-04-25T16:32:24.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f68540afaf0 0x7f68540b2000 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f686c1097a0 0x7f686c1a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f686c106c30 0x7f686c19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 --2- 172.21.15.98:0/2055745970 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f686c101e50 0x7f686c19a0f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 >> 172.21.15.98:0/2055745970 conn(0x7f686c0fd380 msgr2=0x7f686c107ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:24.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.193+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 shutdown_connections 2024-04-25T16:32:24.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.194+0000 7f6870b1f640 1 -- 172.21.15.98:0/2055745970 wait complete. 2024-04-25T16:32:24.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:24 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3653308244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:24.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:24 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1786105928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:24.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:24 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/65896197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:24.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:24 smithi098 ceph-mon[29162]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 10 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:24.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:32:24.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 -- 172.21.15.98:0/1916351385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 msgr2=0x7f38d410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 --2- 172.21.15.98:0/1916351385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 0x7f38d410be50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f38cc009ca0 tx=0x7f38cc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 -- 172.21.15.98:0/1916351385 shutdown_connections 2024-04-25T16:32:24.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 --2- 172.21.15.98:0/1916351385 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 0x7f38d410be50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 --2- 172.21.15.98:0/1916351385 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38d4104d30 0x7f38d4105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 --2- 172.21.15.98:0/1916351385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d4103af0 0x7f38d41047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 -- 172.21.15.98:0/1916351385 >> 172.21.15.98:0/1916351385 conn(0x7f38d40fece0 msgr2=0x7f38d4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:24.316 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 -- 172.21.15.98:0/1916351385 shutdown_connections 2024-04-25T16:32:24.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.316+0000 7f38d91c0640 1 -- 172.21.15.98:0/1916351385 wait complete. 2024-04-25T16:32:24.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d91c0640 1 Processor -- start 2024-04-25T16:32:24.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d91c0640 1 -- start start 2024-04-25T16:32:24.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d91c0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d4103af0 0x7f38d419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d91c0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38d4104d30 0x7f38d419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d91c0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 0x7f38d41a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d91c0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f38d410f500 con 0x7f38d4109a30 2024-04-25T16:32:24.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d91c0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f38d410f380 con 0x7f38d4103af0 2024-04-25T16:32:24.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d91c0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f38d410f680 con 0x7f38d4104d30 2024-04-25T16:32:24.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.317+0000 7f38d359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 0x7f38d41a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d359e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 0x7f38d41a15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52494/0 (socket says 172.21.15.98:52494) 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d359e640 1 -- 172.21.15.98:0/2412372881 learned_addr learned my addr 172.21.15.98:0/2412372881 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38ca59c640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38d4104d30 0x7f38d419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d2d9d640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d4103af0 0x7f38d419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d359e640 1 -- 172.21.15.98:0/2412372881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38d4104d30 msgr2=0x7f38d419a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d359e640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38d4104d30 0x7f38d419a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d359e640 1 -- 172.21.15.98:0/2412372881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d4103af0 msgr2=0x7f38d419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d359e640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d4103af0 0x7f38d419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d359e640 1 -- 172.21.15.98:0/2412372881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38d41a1c10 con 0x7f38d4109a30 2024-04-25T16:32:24.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d359e640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 0x7f38d41a15a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f38cc031040 tx=0x7f38cc003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:24.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d0d99640 1 -- 172.21.15.98:0/2412372881 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc004240 con 0x7f38d4109a30 2024-04-25T16:32:24.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d0d99640 1 -- 172.21.15.98:0/2412372881 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f38cc003e50 con 0x7f38d4109a30 2024-04-25T16:32:24.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d0d99640 1 -- 172.21.15.98:0/2412372881 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc03c410 con 0x7f38d4109a30 2024-04-25T16:32:24.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38d41a1ea0 con 0x7f38d4109a30 2024-04-25T16:32:24.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.318+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38d41a2300 con 0x7f38d4109a30 2024-04-25T16:32:24.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.319+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d406a160 con 0x7f38d4109a30 2024-04-25T16:32:24.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.322+0000 7f38d0d99640 1 -- 172.21.15.98:0/2412372881 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f38cc0306e0 con 0x7f38d4109a30 2024-04-25T16:32:24.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.322+0000 7f38d0d99640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f38b40afaf0 0x7f38b40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.322+0000 7f38d2d9d640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f38b40afaf0 0x7f38b40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.322+0000 7f38d0d99640 1 -- 172.21.15.98:0/2412372881 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f38cc100880 con 0x7f38d4109a30 2024-04-25T16:32:24.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.323+0000 7f38d2d9d640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f38b40afaf0 0x7f38b40b2000 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f38bc005db0 tx=0x7f38bc005d40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:24.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.326+0000 7f38d0d99640 1 -- 172.21.15.98:0/2412372881 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38cc0098b0 con 0x7f38d4109a30 2024-04-25T16:32:24.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:24 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3653308244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:24.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:24 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1786105928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:24.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:24 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/65896197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:24.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:24 smithi203 ceph-mon[33115]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 10 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:24.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.505+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f38d402dbd0 con 0x7f38d4109a30 2024-04-25T16:32:24.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.505+0000 7f38d0d99640 1 -- 172.21.15.98:0/2412372881 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f38cc0c90b0 con 0x7f38d4109a30 2024-04-25T16:32:24.505 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f38b40afaf0 msgr2=0x7f38b40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f38b40afaf0 0x7f38b40b2000 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f38bc005db0 tx=0x7f38bc005d40 comp rx=0 tx=0).stop 2024-04-25T16:32:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 msgr2=0x7f38d41a15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 0x7f38d41a15a0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f38cc031040 tx=0x7f38cc003ce0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 shutdown_connections 2024-04-25T16:32:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f38b40afaf0 0x7f38b40b2000 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f38d4109a30 0x7f38d41a15a0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f38d4104d30 0x7f38d419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 --2- 172.21.15.98:0/2412372881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38d4103af0 0x7f38d419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.507+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 >> 172.21.15.98:0/2412372881 conn(0x7f38d40fece0 msgr2=0x7f38d4105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.508+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 shutdown_connections 2024-04-25T16:32:24.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.508+0000 7f38d91c0640 1 -- 172.21.15.98:0/2412372881 wait complete. 2024-04-25T16:32:24.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:24 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3653308244' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:24.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:24 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1786105928' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:24.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:24 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/65896197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:24.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:24 smithi173 ceph-mon[32556]: pgmap v353: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 10 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:32:24.518 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:32:24.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 -- 172.21.15.98:0/4210028373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c1033b0 msgr2=0x7f8f4c1037c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 --2- 172.21.15.98:0/4210028373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c1033b0 0x7f8f4c1037c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009900 tx=0x7f8f3802a410 comp rx=0 tx=0).stop 2024-04-25T16:32:24.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 -- 172.21.15.98:0/4210028373 shutdown_connections 2024-04-25T16:32:24.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 --2- 172.21.15.98:0/4210028373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f4c1080b0 0x7f8f4c10a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 --2- 172.21.15.98:0/4210028373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c1033b0 0x7f8f4c1037c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 --2- 172.21.15.98:0/4210028373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f4c102170 0x7f8f4c102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 -- 172.21.15.98:0/4210028373 >> 172.21.15.98:0/4210028373 conn(0x7f8f4c0fd340 msgr2=0x7f8f4c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:24.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 -- 172.21.15.98:0/4210028373 shutdown_connections 2024-04-25T16:32:24.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.629+0000 7f8f5325a640 1 -- 172.21.15.98:0/4210028373 wait complete. 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 Processor -- start 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 -- start start 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c102170 0x7f8f4c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f4c1033b0 0x7f8f4c19a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f4c1080b0 0x7f8f4c1a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4c10f4c0 con 0x7f8f4c102170 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4c10f340 con 0x7f8f4c1033b0 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4c10f640 con 0x7f8f4c1080b0 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c102170 0x7f8f4c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c102170 0x7f8f4c19a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52510/0 (socket says 172.21.15.98:52510) 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 -- 172.21.15.98:0/2022197892 learned_addr learned my addr 172.21.15.98:0/2022197892 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f4bfff640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f4c1033b0 0x7f8f4c19a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 -- 172.21.15.98:0/2022197892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f4c1080b0 msgr2=0x7f8f4c1a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f4c1080b0 0x7f8f4c1a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 -- 172.21.15.98:0/2022197892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f4c1033b0 msgr2=0x7f8f4c19a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f4c1033b0 0x7f8f4c19a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 -- 172.21.15.98:0/2022197892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4c1a1c90 con 0x7f8f4c102170 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f50ff6640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c102170 0x7f8f4c19a150 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8f40005d00 tx=0x7f8f4000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f49ffb640 1 -- 172.21.15.98:0/2022197892 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40017070 con 0x7f8f4c102170 2024-04-25T16:32:24.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f49ffb640 1 -- 172.21.15.98:0/2022197892 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8f40009c30 con 0x7f8f4c102170 2024-04-25T16:32:24.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.630+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f4c1a1f80 con 0x7f8f4c102170 2024-04-25T16:32:24.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.631+0000 7f8f49ffb640 1 -- 172.21.15.98:0/2022197892 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4000f620 con 0x7f8f4c102170 2024-04-25T16:32:24.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.631+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f4c1a23e0 con 0x7f8f4c102170 2024-04-25T16:32:24.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.632+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f4c106ec0 con 0x7f8f4c102170 2024-04-25T16:32:24.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.633+0000 7f8f49ffb640 1 -- 172.21.15.98:0/2022197892 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8f4001a070 con 0x7f8f4c102170 2024-04-25T16:32:24.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.634+0000 7f8f49ffb640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f3c0afaf0 0x7f8f3c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.634+0000 7f8f4bfff640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f3c0afaf0 0x7f8f3c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.634+0000 7f8f49ffb640 1 -- 172.21.15.98:0/2022197892 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f8f400dea20 con 0x7f8f4c102170 2024-04-25T16:32:24.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.635+0000 7f8f4bfff640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f3c0afaf0 0x7f8f3c0b2000 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c19b1f0 tx=0x7f8f38000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:24.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.638+0000 7f8f49ffb640 1 -- 172.21.15.98:0/2022197892 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f40009de0 con 0x7f8f4c102170 2024-04-25T16:32:24.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.815+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8f4c02dda0 con 0x7f8f4c102170 2024-04-25T16:32:24.815 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.816+0000 7f8f49ffb640 1 -- 172.21.15.98:0/2022197892 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f8f400a7230 con 0x7f8f4c102170 2024-04-25T16:32:24.816 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:32:24.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f3c0afaf0 msgr2=0x7f8f3c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f3c0afaf0 0x7f8f3c0b2000 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c19b1f0 tx=0x7f8f38000bc0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c102170 msgr2=0x7f8f4c19a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c102170 0x7f8f4c19a150 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f8f40005d00 tx=0x7f8f4000dde0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 shutdown_connections 2024-04-25T16:32:24.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f3c0afaf0 0x7f8f3c0b2000 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f4c1080b0 0x7f8f4c1a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f4c1033b0 0x7f8f4c19a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 --2- 172.21.15.98:0/2022197892 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f4c102170 0x7f8f4c19a150 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 >> 172.21.15.98:0/2022197892 conn(0x7f8f4c0fd340 msgr2=0x7f8f4c104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:24.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 shutdown_connections 2024-04-25T16:32:24.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.818+0000 7f8f5325a640 1 -- 172.21.15.98:0/2022197892 wait complete. 2024-04-25T16:32:24.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 -- 172.21.15.98:0/835261419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0104480 msgr2=0x7effc010abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 --2- 172.21.15.98:0/835261419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0104480 0x7effc010abc0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7effb8009ca0 tx=0x7effb802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 -- 172.21.15.98:0/835261419 shutdown_connections 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 --2- 172.21.15.98:0/835261419 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0104480 0x7effc010abc0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 --2- 172.21.15.98:0/835261419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc0103940 0x7effc0103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 --2- 172.21.15.98:0/835261419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc0102f60 0x7effc0103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 -- 172.21.15.98:0/835261419 >> 172.21.15.98:0/835261419 conn(0x7effc00fd380 msgr2=0x7effc00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 -- 172.21.15.98:0/835261419 shutdown_connections 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.941+0000 7effc6cee640 1 -- 172.21.15.98:0/835261419 wait complete. 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc6cee640 1 Processor -- start 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc6cee640 1 -- start start 2024-04-25T16:32:24.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc6cee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0102f60 0x7effc0102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc6cee640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc0103940 0x7effc01a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc6cee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc0104480 0x7effc01a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc6cee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7effc010e450 con 0x7effc0102f60 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc6cee640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7effc010e2d0 con 0x7effc0103940 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc6cee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7effc010e5d0 con 0x7effc0104480 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0102f60 0x7effc0102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0102f60 0x7effc0102200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52538/0 (socket says 172.21.15.98:52538) 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 -- 172.21.15.98:0/1400296525 learned_addr learned my addr 172.21.15.98:0/1400296525 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc528b640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc0104480 0x7effc01a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 -- 172.21.15.98:0/1400296525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc0104480 msgr2=0x7effc01a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc0104480 0x7effc01a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effbffff640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc0103940 0x7effc01a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 -- 172.21.15.98:0/1400296525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc0103940 msgr2=0x7effc01a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc0103940 0x7effc01a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:24.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 -- 172.21.15.98:0/1400296525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc01a3d60 con 0x7effc0102f60 2024-04-25T16:32:24.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.942+0000 7effc4a8a640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0102f60 0x7effc0102200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7effb400c8c0 tx=0x7effb400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:24.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.943+0000 7effbdffb640 1 -- 172.21.15.98:0/1400296525 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb4007bb0 con 0x7effc0102f60 2024-04-25T16:32:24.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.943+0000 7effbdffb640 1 -- 172.21.15.98:0/1400296525 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7effb4007d50 con 0x7effc0102f60 2024-04-25T16:32:24.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.943+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc01a3ff0 con 0x7effc0102f60 2024-04-25T16:32:24.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.943+0000 7effbdffb640 1 -- 172.21.15.98:0/1400296525 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb40186a0 con 0x7effc0102f60 2024-04-25T16:32:24.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.943+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc01a4500 con 0x7effc0102f60 2024-04-25T16:32:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.944+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc0109910 con 0x7effc0102f60 2024-04-25T16:32:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.945+0000 7effbdffb640 1 -- 172.21.15.98:0/1400296525 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7effb400fa00 con 0x7effc0102f60 2024-04-25T16:32:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.946+0000 7effbdffb640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7effb00afb70 0x7effb00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.946+0000 7effbffff640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7effb00afb70 0x7effb00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.946+0000 7effbdffb640 1 -- 172.21.15.98:0/1400296525 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7effb40df0c0 con 0x7effc0102f60 2024-04-25T16:32:24.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.946+0000 7effbffff640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7effb00afb70 0x7effb00b2080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7effc0101ba0 tx=0x7effac005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:24.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:24.950+0000 7effbdffb640 1 -- 172.21.15.98:0/1400296525 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7effb4014030 con 0x7effc0102f60 2024-04-25T16:32:25.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.133+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7effc00637d0 con 0x7effc0102f60 2024-04-25T16:32:25.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.133+0000 7effbdffb640 1 -- 172.21.15.98:0/1400296525 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7effb40a78f0 con 0x7effc0102f60 2024-04-25T16:32:25.133 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:25.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7effb00afb70 msgr2=0x7effb00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7effb00afb70 0x7effb00b2080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7effc0101ba0 tx=0x7effac005f90 comp rx=0 tx=0).stop 2024-04-25T16:32:25.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0102f60 msgr2=0x7effc0102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0102f60 0x7effc0102200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7effb400c8c0 tx=0x7effb400cd80 comp rx=0 tx=0).stop 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 shutdown_connections 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7effb00afb70 0x7effb00b2080 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc0104480 0x7effc01a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc0103940 0x7effc01a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 --2- 172.21.15.98:0/1400296525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7effc0102f60 0x7effc0102200 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 >> 172.21.15.98:0/1400296525 conn(0x7effc00fd380 msgr2=0x7effc0106ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 shutdown_connections 2024-04-25T16:32:25.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.136+0000 7effc6cee640 1 -- 172.21.15.98:0/1400296525 wait complete. 2024-04-25T16:32:25.146 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:25 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2055745970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.146 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:25 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2412372881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.146 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:25 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2022197892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:32:25.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.255+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/1498330405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8108940 msgr2=0x7f7cc810ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.255+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/1498330405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8108940 0x7f7cc810ad60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009ca0 tx=0x7f7cc002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/1498330405 shutdown_connections 2024-04-25T16:32:25.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/1498330405 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8108940 0x7f7cc810ad60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/1498330405 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc8103c40 0x7f7cc8104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/1498330405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cc8102a00 0x7f7cc8103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/1498330405 >> 172.21.15.98:0/1498330405 conn(0x7f7cc80fdbf0 msgr2=0x7f7cc81000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/1498330405 shutdown_connections 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/1498330405 wait complete. 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 Processor -- start 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.256+0000 7f7ccc8e7640 1 -- start start 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7ccc8e7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8102a00 0x7f7cc819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7ccc8e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc8103c40 0x7f7cc819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7ccc8e7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cc8108940 0x7f7cc81a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7ccc8e7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc810f540 con 0x7f7cc8102a00 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7ccc8e7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc810f3c0 con 0x7f7cc8108940 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7ccc8e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc810f6c0 con 0x7f7cc8103c40 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8102a00 0x7f7cc819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8102a00 0x7f7cc819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:52554/0 (socket says 172.21.15.98:52554) 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 -- 172.21.15.98:0/4109057455 learned_addr learned my addr 172.21.15.98:0/4109057455 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:25.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc5d9b640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc8103c40 0x7f7cc819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc6d9d640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cc8108940 0x7f7cc81a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 -- 172.21.15.98:0/4109057455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc8103c40 msgr2=0x7f7cc819a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc8103c40 0x7f7cc819a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 -- 172.21.15.98:0/4109057455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cc8108940 msgr2=0x7f7cc81a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cc8108940 0x7f7cc81a1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 -- 172.21.15.98:0/4109057455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc81a1c00 con 0x7f7cc8102a00 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cc659c640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8102a00 0x7f7cc819a0c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800c970 tx=0x7f7cb800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cb77fe640 1 -- 172.21.15.98:0/4109057455 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8004220 con 0x7f7cc8102a00 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cb77fe640 1 -- 172.21.15.98:0/4109057455 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7cb8007e40 con 0x7f7cc8102a00 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc81a1ef0 con 0x7f7cc8102a00 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7cb77fe640 1 -- 172.21.15.98:0/4109057455 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb800f450 con 0x7f7cc8102a00 2024-04-25T16:32:25.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.257+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc81a2350 con 0x7f7cc8102a00 2024-04-25T16:32:25.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.258+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc806a1c0 con 0x7f7cc8102a00 2024-04-25T16:32:25.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.260+0000 7f7cb77fe640 1 -- 172.21.15.98:0/4109057455 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7cb80079b0 con 0x7f7cc8102a00 2024-04-25T16:32:25.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.260+0000 7f7cb77fe640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7cb00afaf0 0x7f7cb00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.260+0000 7f7cc5d9b640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7cb00afaf0 0x7f7cb00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.260+0000 7f7cb77fe640 1 -- 172.21.15.98:0/4109057455 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7cb80df120 con 0x7f7cc8102a00 2024-04-25T16:32:25.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.261+0000 7f7cc5d9b640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7cb00afaf0 0x7f7cb00b2000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005fd0 tx=0x7f7cbc005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.264+0000 7f7cb77fe640 1 -- 172.21.15.98:0/4109057455 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7cc806a1c0 con 0x7f7cc8102a00 2024-04-25T16:32:25.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:25 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2055745970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:25 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2412372881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:25 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2022197892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.415 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:25 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2055745970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.415 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:25 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2412372881' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.415 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:25 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2022197892' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:25.478 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.479+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cc802dc10 con 0x7f7cc8102a00 2024-04-25T16:32:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.480+0000 7f7cb77fe640 1 -- 172.21.15.98:0/4109057455 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11935 (secure 0 0 0) 0x7f7cb8014030 con 0x7f7cc8102a00 2024-04-25T16:32:25.479 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:25.479 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:30:38.051368+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24484,24539,24457,24473,24494]},"id":1}]} 2024-04-25T16:32:25.479 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 28 2024-04-25T16:32:25.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7cb00afaf0 msgr2=0x7f7cb00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7cb00afaf0 0x7f7cb00b2000 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc005fd0 tx=0x7f7cbc005e20 comp rx=0 tx=0).stop 2024-04-25T16:32:25.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8102a00 msgr2=0x7f7cc819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8102a00 0x7f7cc819a0c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7cb800c970 tx=0x7f7cb800ce30 comp rx=0 tx=0).stop 2024-04-25T16:32:25.481 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 shutdown_connections 2024-04-25T16:32:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7cb00afaf0 0x7f7cb00b2000 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7cc8108940 0x7f7cc81a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7cc8103c40 0x7f7cc819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 --2- 172.21.15.98:0/4109057455 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7cc8102a00 0x7f7cc819a0c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 >> 172.21.15.98:0/4109057455 conn(0x7f7cc80fdbf0 msgr2=0x7f7cc8104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 shutdown_connections 2024-04-25T16:32:25.482 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.482+0000 7f7ccc8e7640 1 -- 172.21.15.98:0/4109057455 wait complete. 2024-04-25T16:32:25.493 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:32:25.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 -- 172.21.15.98:0/1043745373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20104d50 msgr2=0x7f4e20105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1043745373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20104d50 0x7f4e20105160 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009900 tx=0x7f4e1c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:25.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 -- 172.21.15.98:0/1043745373 shutdown_connections 2024-04-25T16:32:25.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1043745373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e20109a50 0x7f4e2010be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1043745373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20104d50 0x7f4e20105160 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1043745373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e20103b10 0x7f4e20104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 -- 172.21.15.98:0/1043745373 >> 172.21.15.98:0/1043745373 conn(0x7f4e200fed40 msgr2=0x7f4e201011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 -- 172.21.15.98:0/1043745373 shutdown_connections 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.602+0000 7f4e27a53640 1 -- 172.21.15.98:0/1043745373 wait complete. 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e27a53640 1 Processor -- start 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e27a53640 1 -- start start 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e27a53640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e20103b10 0x7f4e2019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e27a53640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e20104d50 0x7f4e2019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e27a53640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20109a50 0x7f4e201a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e27a53640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2010f4f0 con 0x7f4e20109a50 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e27a53640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2010f370 con 0x7f4e20104d50 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e27a53640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2010f670 con 0x7f4e20103b10 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20109a50 0x7f4e201a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20109a50 0x7f4e201a1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53226/0 (socket says 172.21.15.98:53226) 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 -- 172.21.15.98:0/1304106141 learned_addr learned my addr 172.21.15.98:0/1304106141 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e24fee640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e20104d50 0x7f4e2019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e257ef640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e20103b10 0x7f4e2019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 -- 172.21.15.98:0/1304106141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e20103b10 msgr2=0x7f4e2019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e20103b10 0x7f4e2019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 -- 172.21.15.98:0/1304106141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e20104d50 msgr2=0x7f4e2019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e20104d50 0x7f4e2019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 -- 172.21.15.98:0/1304106141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e201a1c10 con 0x7f4e20109a50 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e25ff0640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20109a50 0x7f4e201a1650 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800cad0 tx=0x7f4e1800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.603+0000 7f4e127fc640 1 -- 172.21.15.98:0/1304106141 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1800ae20 con 0x7f4e20109a50 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.604+0000 7f4e127fc640 1 -- 172.21.15.98:0/1304106141 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4e18005020 con 0x7f4e20109a50 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.604+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e201a1f00 con 0x7f4e20109a50 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.604+0000 7f4e127fc640 1 -- 172.21.15.98:0/1304106141 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1800f6b0 con 0x7f4e20109a50 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.604+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e201a2360 con 0x7f4e20109a50 2024-04-25T16:32:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.605+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e2006a190 con 0x7f4e20109a50 2024-04-25T16:32:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.606+0000 7f4e127fc640 1 -- 172.21.15.98:0/1304106141 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4e180051d0 con 0x7f4e20109a50 2024-04-25T16:32:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.607+0000 7f4e127fc640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e0c0afaf0 0x7f4e0c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.607+0000 7f4e127fc640 1 -- 172.21.15.98:0/1304106141 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f4e180df340 con 0x7f4e20109a50 2024-04-25T16:32:25.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.607+0000 7f4e257ef640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e0c0afaf0 0x7f4e0c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.607+0000 7f4e257ef640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e0c0afaf0 0x7f4e0c0b2000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4e140058c0 tx=0x7f4e14005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.610+0000 7f4e127fc640 1 -- 172.21.15.98:0/1304106141 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e18014030 con 0x7f4e20109a50 2024-04-25T16:32:25.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.807+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e2002dc10 con 0x7f4e20109a50 2024-04-25T16:32:25.808 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.809+0000 7f4e127fc640 1 -- 172.21.15.98:0/1304106141 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24806 (secure 0 0 0) 0x7f4e180a7b70 con 0x7f4e20109a50 2024-04-25T16:32:25.809 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:25.809 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:32:10.010488+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e0c0afaf0 msgr2=0x7f4e0c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e0c0afaf0 0x7f4e0c0b2000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4e140058c0 tx=0x7f4e14005850 comp rx=0 tx=0).stop 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20109a50 msgr2=0x7f4e201a1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20109a50 0x7f4e201a1650 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f4e1800cad0 tx=0x7f4e1800cf90 comp rx=0 tx=0).stop 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 shutdown_connections 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f4e0c0afaf0 0x7f4e0c0b2000 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4e20109a50 0x7f4e201a1650 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e20104d50 0x7f4e2019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 --2- 172.21.15.98:0/1304106141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4e20103b10 0x7f4e2019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 >> 172.21.15.98:0/1304106141 conn(0x7f4e200fed40 msgr2=0x7f4e20105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 shutdown_connections 2024-04-25T16:32:25.811 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.811+0000 7f4e27a53640 1 -- 172.21.15.98:0/1304106141 wait complete. 2024-04-25T16:32:25.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T16:30:09.342794+0000', 'modified': '2024-04-25T16:30:38.051368+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24539, 'mds_1': 24494, 'mds_2': 24484, 'mds_3': 24473, 'mds_4': 24457}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14634': {'gid': 14634, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6839/3308601909', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6838', 'nonce': 3308601909}, {'type': 'v1', 'addr': '172.21.15.98:6839', 'nonce': 3308601909}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14646': {'gid': 14646, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.98:6841/373690634', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.98:6840', 'nonce': 373690634}, {'type': 'v1', 'addr': '172.21.15.98:6841', 'nonce': 373690634}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24415': {'gid': 24415, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6833/1945729206', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6832', 'nonce': 1945729206}, {'type': 'v1', 'addr': '172.21.15.203:6833', 'nonce': 1945729206}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24443': {'gid': 24443, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.173:6833/3037734321', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 3037734321}, {'type': 'v1', 'addr': '172.21.15.173:6833', 'nonce': 3037734321}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24445': {'gid': 24445, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.203:6835/2230353021', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6834', 'nonce': 2230353021}, {'type': 'v1', 'addr': '172.21.15.203:6835', 'nonce': 2230353021}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24457': {'gid': 24457, 'name': 'c', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.203:6837/998442662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6836', 'nonce': 998442662}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 998442662}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24473': {'gid': 24473, 'name': 'e', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.173:6835/960684117', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6834', 'nonce': 960684117}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 960684117}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.203:6839/2982027693', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6838', 'nonce': 2982027693}, {'type': 'v1', 'addr': '172.21.15.203:6839', 'nonce': 2982027693}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.173:6837/311453164', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6836', 'nonce': 311453164}, {'type': 'v1', 'addr': '172.21.15.173:6837', 'nonce': 311453164}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.173:6839/2050448663', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6838', 'nonce': 2050448663}, {'type': 'v1', 'addr': '172.21.15.173:6839', 'nonce': 2050448663}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24539, 'qdb_cluster': [24484, 24539, 24457, 24473, 24494]} 2024-04-25T16:32:25.823 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T16:32:25.823 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 -- 172.21.15.98:0/1006757213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c41033d0 msgr2=0x7f69c41037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 --2- 172.21.15.98:0/1006757213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c41033d0 0x7f69c41037e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009900 tx=0x7f69ac02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 -- 172.21.15.98:0/1006757213 shutdown_connections 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 --2- 172.21.15.98:0/1006757213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c41080d0 0x7f69c410a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 --2- 172.21.15.98:0/1006757213 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c41033d0 0x7f69c41037e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 --2- 172.21.15.98:0/1006757213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c4102190 0x7f69c4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 -- 172.21.15.98:0/1006757213 >> 172.21.15.98:0/1006757213 conn(0x7f69c40fd380 msgr2=0x7f69c40ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 -- 172.21.15.98:0/1006757213 shutdown_connections 2024-04-25T16:32:25.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.932+0000 7f69caf7a640 1 -- 172.21.15.98:0/1006757213 wait complete. 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69caf7a640 1 Processor -- start 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69caf7a640 1 -- start start 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69caf7a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c4102190 0x7f69c40ff210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69caf7a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c41033d0 0x7f69c40fd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69caf7a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c41080d0 0x7f69c40fddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69caf7a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f69c410d2d0 con 0x7f69c4102190 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69caf7a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f69c410d150 con 0x7f69c41033d0 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69caf7a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f69c410d450 con 0x7f69c41080d0 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c4102190 0x7f69c40ff210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c4102190 0x7f69c40ff210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53240/0 (socket says 172.21.15.98:53240) 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 -- 172.21.15.98:0/4272161356 learned_addr learned my addr 172.21.15.98:0/4272161356 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c3fff640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c41033d0 0x7f69c40fd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 -- 172.21.15.98:0/4272161356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c41080d0 msgr2=0x7f69c40fddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c41080d0 0x7f69c40fddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 -- 172.21.15.98:0/4272161356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c41033d0 msgr2=0x7f69c40fd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c41033d0 0x7f69c40fd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 -- 172.21.15.98:0/4272161356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c40fe520 con 0x7f69c4102190 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c3fff640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c41033d0 0x7f69c40fd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.933+0000 7f69c8d16640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c4102190 0x7f69c40ff210 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00ead0 tx=0x7f69bc00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.934+0000 7f69c1ffb640 1 -- 172.21.15.98:0/4272161356 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc00cde0 con 0x7f69c4102190 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.934+0000 7f69c1ffb640 1 -- 172.21.15.98:0/4272161356 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69bc003e00 con 0x7f69c4102190 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.934+0000 7f69caf7a640 1 -- 172.21.15.98:0/4272161356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c41a4240 con 0x7f69c4102190 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.934+0000 7f69c1ffb640 1 -- 172.21.15.98:0/4272161356 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0116d0 con 0x7f69c4102190 2024-04-25T16:32:25.936 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.934+0000 7f69caf7a640 1 -- 172.21.15.98:0/4272161356 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c41a46b0 con 0x7f69c4102190 2024-04-25T16:32:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69c1ffb640 1 -- 172.21.15.98:0/4272161356 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f69bc003970 con 0x7f69c4102190 2024-04-25T16:32:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69d1e2a740 1 Processor -- start 2024-04-25T16:32:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69d1e2a740 1 -- start start 2024-04-25T16:32:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 0x558ad17bf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17bf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.943 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13fd590 0x558ad17c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558ad1724740 con 0x558ad13f53b0 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ad16c03a0 con 0x558ad13fd590 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.936+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558ad1720700 con 0x558ad141e270 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17bf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17bf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53256/0 (socket says 172.21.15.98:53256) 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 -- 172.21.15.98:0/123764441 learned_addr learned my addr 172.21.15.98:0/123764441 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c1ffb640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f69b00afaf0 0x7f69b00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c9517640 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 0x558ad17bf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c3fff640 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13fd590 0x558ad17c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 -- 172.21.15.98:0/123764441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 msgr2=0x558ad17bf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 0x558ad17bf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 -- 172.21.15.98:0/123764441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13fd590 msgr2=0x558ad17c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13fd590 0x558ad17c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 -- 172.21.15.98:0/123764441 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ad16dc750 con 0x558ad13f53b0 2024-04-25T16:32:25.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c9517640 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 0x558ad17bf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c3fff640 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13fd590 0x558ad17c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c9517640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f69b00afaf0 0x7f69b00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c1ffb640 1 -- 172.21.15.98:0/4272161356 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f69bc0df6b0 con 0x7f69c4102190 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69c8d16640 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17bf8f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0dd060 tx=0x7f69bc005f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12b27b0dbc9010e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69baffd640 1 -- 172.21.15.98:0/123764441 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0a7130 con 0x558ad13f53b0 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69baffd640 1 -- 172.21.15.98:0/123764441 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69bc0a72f0 con 0x558ad13f53b0 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.937+0000 7f69baffd640 1 -- 172.21.15.98:0/123764441 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc003c60 con 0x558ad13f53b0 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 -- 172.21.15.98:0/123764441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 msgr2=0x558ad17bf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17bf8f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0dd060 tx=0x7f69bc005f50 comp rx=0 tx=0).stop 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69c9517640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f69b00afaf0 0x7f69b00b2000 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x558ad17c0540 tx=0x7f69b40046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 -- 172.21.15.98:0/123764441 shutdown_connections 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13fd590 0x558ad17c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17bf8f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.945 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/123764441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 0x558ad17bf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 -- 172.21.15.98:0/123764441 >> 172.21.15.98:0/123764441 conn(0x558ad14123c0 msgr2=0x558ad17bad00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 -- 172.21.15.98:0/123764441 shutdown_connections 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.938+0000 7f69d1e2a740 1 -- 172.21.15.98:0/123764441 wait complete. 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 1 -- start start 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 1 Processor -- start 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 0x558ad17d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558ad17c7180 con 0x558ad13f53b0 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ad16abfe0 con 0x558ad141e270 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558ad17c7300 con 0x558ad13fd590 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69d1e2a740 10 client.0 _ll_register_callbacks cb 0x558ad16c2150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c8d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.946 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c8d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53260/0 (socket says 172.21.15.98:53260) 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c8d16640 1 -- 172.21.15.98:0/1732767054 learned_addr learned my addr 172.21.15.98:0/1732767054 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c9517640 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 0x558ad17d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c3fff640 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c8d16640 1 -- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 msgr2=0x558ad17c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c8d16640 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c8d16640 1 -- 172.21.15.98:0/1732767054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 msgr2=0x558ad17d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c8d16640 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 0x558ad17d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c8d16640 1 -- 172.21.15.98:0/1732767054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ad14362d0 con 0x558ad13f53b0 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.939+0000 7f69c3fff640 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69c8d16640 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c9130 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0abd50 tx=0x7f69bc009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a987913960fb68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69a3fff640 1 -- 172.21.15.98:0/1732767054 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0a6f50 con 0x558ad13f53b0 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69a3fff640 1 -- 172.21.15.98:0/1732767054 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f69bc0df3c0 con 0x558ad13f53b0 2024-04-25T16:32:25.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 1 -- 172.21.15.98:0/1732767054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69bc0a72f0 con 0x558ad13f53b0 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69a3fff640 1 -- 172.21.15.98:0/1732767054 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f69bc0a7130 con 0x558ad13f53b0 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 10 client.15444 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 1 -- 172.21.15.98:0/1732767054 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558ad15006b0 con 0x558ad13f53b0 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69a3fff640 1 -- 172.21.15.98:0/1732767054 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69bc0a72f0 con 0x558ad13f53b0 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69a3fff640 1 -- 172.21.15.98:0/1732767054 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f69bc021030 con 0x558ad13f53b0 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 10 client.15444 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 10 client.15444 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 20 client.15444 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x558ad13dce40 0x558ad17d6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 4 client.15444 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.940+0000 7f69d1e2a740 1 -- 172.21.15.98:0/1732767054 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ad1732350 con 0x558ad13dce40 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.941+0000 7f69c3fff640 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x558ad13dce40 0x558ad17d6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.941+0000 7f69a3fff640 10 client.15444 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:32:25.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.941+0000 7f69c3fff640 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x558ad13dce40 0x558ad17d6950 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e36fe94882bd43e3 server_cookie=dabc14bff8ffb05b in_seq=0 out_seq=0 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69a3fff640 1 -- 172.21.15.98:0/1732767054 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558ad1732350 con 0x558ad13dce40 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69a3fff640 10 client.15444 handle_command_reply: tid=0 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 client.15444 shutdown 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 -- 172.21.15.98:0/1732767054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 msgr2=0x558ad17c9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c9130 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0abd50 tx=0x7f69bc009280 comp rx=0 tx=0).stop 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 -- 172.21.15.98:0/1732767054 shutdown_connections 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x558ad13dce40 0x558ad17d6950 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 0x558ad17d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69a3fff640 0 client.15444 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/1732767054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c9130 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 -- 172.21.15.98:0/1732767054 >> 172.21.15.98:0/1732767054 conn(0x558ad14123c0 msgr2=0x558ad17b9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 -- 172.21.15.98:0/1732767054 shutdown_connections 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 1 -- 172.21.15.98:0/1732767054 wait complete. 2024-04-25T16:32:25.949 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.942+0000 7f69d1e2a740 20 client.15444 trim_cache size 0 max 16384 2024-04-25T16:32:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69d1e2a740 1 Processor -- start 2024-04-25T16:32:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69d1e2a740 1 -- start start 2024-04-25T16:32:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 0x558ad17ba6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558ad17d4c90 con 0x558ad13f53b0 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ad17d5360 con 0x558ad141e270 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558ad17d5a30 con 0x558ad13fd590 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53274/0 (socket says 172.21.15.98:53274) 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 -- 172.21.15.98:0/3993874150 learned_addr learned my addr 172.21.15.98:0/3993874150 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c3fff640 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 -- 172.21.15.98:0/3993874150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 msgr2=0x558ad17c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 -- 172.21.15.98:0/3993874150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 msgr2=0x558ad17ba6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 0x558ad17ba6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 -- 172.21.15.98:0/3993874150 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ad17c8ee0 con 0x558ad13f53b0 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c3fff640 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.961+0000 7f69c8d16640 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c90d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0dcfd0 tx=0x7f69bc004540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159984dacd5cd01d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69ba7fc640 1 -- 172.21.15.98:0/3993874150 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc0a6f50 con 0x558ad13f53b0 2024-04-25T16:32:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69ba7fc640 1 -- 172.21.15.98:0/3993874150 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f69bc0a77e0 con 0x558ad13f53b0 2024-04-25T16:32:25.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69ba7fc640 1 -- 172.21.15.98:0/3993874150 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69bc003c60 con 0x558ad13f53b0 2024-04-25T16:32:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3993874150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 msgr2=0x558ad17c90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c90d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0dcfd0 tx=0x7f69bc004540 comp rx=0 tx=0).stop 2024-04-25T16:32:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3993874150 shutdown_connections 2024-04-25T16:32:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad141e270 0x558ad17ba6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad13fd590 0x558ad17c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3993874150 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13f53b0 0x558ad17c90d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3993874150 >> 172.21.15.98:0/3993874150 conn(0x558ad14123c0 msgr2=0x558ad17b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3993874150 shutdown_connections 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.962+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3993874150 wait complete. 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 1 -- start start 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 1 Processor -- start 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13f53b0 0x558ad17cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13fd590 0x558ad17cb510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 0x558ad17ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558ad17d67d0 con 0x558ad13fd590 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558ad17d6100 con 0x558ad13f53b0 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558ad17d6ea0 con 0x558ad141e270 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69d1e2a740 10 client.0 _ll_register_callbacks cb 0x7f69bc0a3730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69c3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13fd590 0x558ad17cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69c3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13fd590 0x558ad17cb510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53278/0 (socket says 172.21.15.98:53278) 2024-04-25T16:32:25.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69c3fff640 1 -- 172.21.15.98:0/3399648365 learned_addr learned my addr 172.21.15.98:0/3399648365 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.963+0000 7f69c8d16640 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13f53b0 0x558ad17cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69c3fff640 1 -- 172.21.15.98:0/3399648365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 msgr2=0x558ad17ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69c3fff640 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 0x558ad17ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69c3fff640 1 -- 172.21.15.98:0/3399648365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13f53b0 msgr2=0x558ad17cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69c3fff640 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13f53b0 0x558ad17cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69c3fff640 1 -- 172.21.15.98:0/3399648365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ad17ce950 con 0x558ad13fd590 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69c3fff640 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13fd590 0x558ad17cb510 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f69ac006010 tx=0x7f69ac009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d969621745abd0cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69a3fff640 1 -- 172.21.15.98:0/3399648365 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac035070 con 0x558ad13fd590 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69a3fff640 1 -- 172.21.15.98:0/3399648365 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f69ac004250 con 0x558ad13fd590 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3399648365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558ad172c1c0 con 0x558ad13fd590 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69a3fff640 1 -- 172.21.15.98:0/3399648365 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f69ac004450 con 0x558ad13fd590 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 10 client.15450 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3399648365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558ad17cfbb0 con 0x558ad13fd590 2024-04-25T16:32:25.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69a3fff640 1 -- 172.21.15.98:0/3399648365 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x558ad172c1c0 con 0x558ad13fd590 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69a3fff640 1 -- 172.21.15.98:0/3399648365 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f69ac003b90 con 0x558ad13fd590 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 10 client.15450 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 10 client.15450 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 20 client.15450 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x558ad13b2280 0x558ad1842b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 4 client.15450 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.964+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3399648365 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "status"}) -- 0x558ad16a79a0 con 0x558ad13b2280 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.965+0000 7f69c8d16640 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x558ad13b2280 0x558ad1842b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.965+0000 7f69a3fff640 10 client.15450 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.965+0000 7f69c8d16640 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x558ad13b2280 0x558ad1842b00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b800d4de9d2e2dbf server_cookie=657be6d43cf0e59f in_seq=0 out_seq=0 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.965+0000 7f69a3fff640 1 -- 172.21.15.98:0/3399648365 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x558ad16a79a0 con 0x558ad13b2280 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.965+0000 7f69a3fff640 10 client.15450 handle_command_reply: tid=0 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.965+0000 7f69d1e2a740 1 client.15450 shutdown 2024-04-25T16:32:25.970 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3399648365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13fd590 msgr2=0x558ad17cb510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13fd590 0x558ad17cb510 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f69ac006010 tx=0x7f69ac009520 comp rx=0 tx=0).stop 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3399648365 shutdown_connections 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x558ad13b2280 0x558ad1842b00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558ad141e270 0x558ad17ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558ad13fd590 0x558ad17cb510 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69a3fff640 0 client.15450 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 --2- 172.21.15.98:0/3399648365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558ad13f53b0 0x558ad17cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3399648365 >> 172.21.15.98:0/3399648365 conn(0x558ad14123c0 msgr2=0x558ad17c90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3399648365 shutdown_connections 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 1 -- 172.21.15.98:0/3399648365 wait complete. 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69d1e2a740 20 client.15450 trim_cache size 0 max 16384 2024-04-25T16:32:25.971 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 4, 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24457, 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 108.933582019, 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 155.37429977799999 2024-04-25T16:32:25.972 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 -- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f69b00afaf0 msgr2=0x7f69b00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f69b00afaf0 0x7f69b00b2000 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x558ad17c0540 tx=0x7f69b40046c0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 -- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c4102190 msgr2=0x7f69c40ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c4102190 0x7f69c40ff210 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f69bc00ead0 tx=0x7f69bc00ef90 comp rx=0 tx=0).stop 2024-04-25T16:32:25.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 -- 172.21.15.98:0/4272161356 shutdown_connections 2024-04-25T16:32:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f69b00afaf0 0x7f69b00b2000 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69c41080d0 0x7f69c40fddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69c41033d0 0x7f69c40fd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 --2- 172.21.15.98:0/4272161356 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f69c4102190 0x7f69c40ff210 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 -- 172.21.15.98:0/4272161356 >> 172.21.15.98:0/4272161356 conn(0x7f69c40fd380 msgr2=0x7f69c41060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 -- 172.21.15.98:0/4272161356 shutdown_connections 2024-04-25T16:32:25.974 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:25.966+0000 7f69caf7a640 1 -- 172.21.15.98:0/4272161356 wait complete. 2024-04-25T16:32:25.977 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-25T16:32:26.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 -- 172.21.15.98:0/2890090963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f81029e0 msgr2=0x7f88f81036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 --2- 172.21.15.98:0/2890090963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f81029e0 0x7f88f81036e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f88f4009900 tx=0x7f88f402a410 comp rx=0 tx=0).stop 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 -- 172.21.15.98:0/2890090963 shutdown_connections 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 --2- 172.21.15.98:0/2890090963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88f8108920 0x7f88f810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 --2- 172.21.15.98:0/2890090963 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f8103c20 0x7f88f8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 --2- 172.21.15.98:0/2890090963 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f81029e0 0x7f88f81036e0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 -- 172.21.15.98:0/2890090963 >> 172.21.15.98:0/2890090963 conn(0x7f88f80fdbd0 msgr2=0x7f88f8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 -- 172.21.15.98:0/2890090963 shutdown_connections 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.090+0000 7f8900546640 1 -- 172.21.15.98:0/2890090963 wait complete. 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f8900546640 1 Processor -- start 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f8900546640 1 -- start start 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f8900546640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f81029e0 0x7f88f806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f8900546640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f8103c20 0x7f88f819f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f8900546640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88f8108920 0x7f88f81a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f8900546640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f88f810e370 con 0x7f88f8103c20 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f8900546640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f88f810e1f0 con 0x7f88f81029e0 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f8900546640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f88f810e4f0 con 0x7f88f8108920 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fdae1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f8103c20 0x7f88f819f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fdae1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f8103c20 0x7f88f819f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53302/0 (socket says 172.21.15.98:53302) 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fdae1640 1 -- 172.21.15.98:0/4206727858 learned_addr learned my addr 172.21.15.98:0/4206727858 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f81029e0 0x7f88f806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88feae3640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88f8108920 0x7f88f81a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fdae1640 1 -- 172.21.15.98:0/4206727858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88f8108920 msgr2=0x7f88f81a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fdae1640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88f8108920 0x7f88f81a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fdae1640 1 -- 172.21.15.98:0/4206727858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f81029e0 msgr2=0x7f88f806a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fdae1640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f81029e0 0x7f88f806a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.091+0000 7f88fdae1640 1 -- 172.21.15.98:0/4206727858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f81a1f60 con 0x7f88f8103c20 2024-04-25T16:32:26.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.092+0000 7f88fdae1640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f8103c20 0x7f88f819f1f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f88e800e9f0 tx=0x7f88e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.092+0000 7f88ef7fe640 1 -- 172.21.15.98:0/4206727858 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e800cd80 con 0x7f88f8103c20 2024-04-25T16:32:26.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.092+0000 7f8900546640 1 -- 172.21.15.98:0/4206727858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f81a2250 con 0x7f88f8103c20 2024-04-25T16:32:26.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.092+0000 7f88ef7fe640 1 -- 172.21.15.98:0/4206727858 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f88e80108c0 con 0x7f88f8103c20 2024-04-25T16:32:26.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.092+0000 7f88ef7fe640 1 -- 172.21.15.98:0/4206727858 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e80187c0 con 0x7f88f8103c20 2024-04-25T16:32:26.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.092+0000 7f8900546640 1 -- 172.21.15.98:0/4206727858 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f81a27a0 con 0x7f88f8103c20 2024-04-25T16:32:26.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.094+0000 7f89073fb740 1 Processor -- start 2024-04-25T16:32:26.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.094+0000 7f88ef7fe640 1 -- 172.21.15.98:0/4206727858 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f88e8010430 con 0x7f88f8103c20 2024-04-25T16:32:26.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f89073fb740 1 -- start start 2024-04-25T16:32:26.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f89073fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2aac780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f89073fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aacd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f89073fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f89073fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eac2a0d740 con 0x55eac26e6590 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f89073fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eac29a93a0 con 0x55eac26de3b0 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f89073fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eac2a09700 con 0x55eac2707270 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fdae1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fdae1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53306/0 (socket says 172.21.15.98:53306) 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fdae1640 1 -- 172.21.15.98:0/2825646664 learned_addr learned my addr 172.21.15.98:0/2825646664 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aacd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88feae3640 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2aac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88ef7fe640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f88e40afaf0 0x7f88e40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fdae1640 1 -- 172.21.15.98:0/2825646664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 msgr2=0x55eac2aac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fdae1640 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2aac780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fdae1640 1 -- 172.21.15.98:0/2825646664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 msgr2=0x55eac2aacd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fdae1640 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aacd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88fdae1640 1 -- 172.21.15.98:0/2825646664 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eac29c5750 con 0x55eac26e6590 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.095+0000 7f88feae3640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f88e40afaf0 0x7f88e40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.096+0000 7f88ef7fe640 1 -- 172.21.15.98:0/4206727858 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f88e80df350 con 0x7f88f8103c20 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.096+0000 7f88fdae1640 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab3c00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f88e80da320 tx=0x7f88e80d5bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f8dd798745ee55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.096+0000 7f88ecbfe640 1 -- 172.21.15.98:0/2825646664 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8026030 con 0x55eac26e6590 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.096+0000 7f88feae3640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f88e40afaf0 0x7f88e40b2000 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f88f00098b0 tx=0x7f88f0004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.096+0000 7f88ecbfe640 1 -- 172.21.15.98:0/2825646664 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f88e80d98d0 con 0x55eac26e6590 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.096+0000 7f88ecbfe640 1 -- 172.21.15.98:0/2825646664 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e80a6ca0 con 0x55eac26e6590 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- 172.21.15.98:0/2825646664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 msgr2=0x55eac2ab3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab3c00 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f88e80da320 tx=0x7f88e80d5bd0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- 172.21.15.98:0/2825646664 shutdown_connections 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab3c00 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aacd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 --2- 172.21.15.98:0/2825646664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2aac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- 172.21.15.98:0/2825646664 >> 172.21.15.98:0/2825646664 conn(0x55eac26fb3c0 msgr2=0x55eac2aa3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- 172.21.15.98:0/2825646664 shutdown_connections 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- 172.21.15.98:0/2825646664 wait complete. 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- start start 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 Processor -- start 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2ab65f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2abda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eac2ab45f0 con 0x55eac26e6590 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eac2994fe0 con 0x55eac26de3b0 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.097+0000 7f89073fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eac2ab4770 con 0x55eac2707270 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f89073fb740 10 client.0 _ll_register_callbacks cb 0x55eac29ab150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53314/0 (socket says 172.21.15.98:53314) 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 -- 172.21.15.98:0/3017187842 learned_addr learned my addr 172.21.15.98:0/3017187842 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2ab65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88feae3640 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2abda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 -- 172.21.15.98:0/3017187842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 msgr2=0x55eac2abda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2abda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 -- 172.21.15.98:0/3017187842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 msgr2=0x55eac2ab65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2ab65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 -- 172.21.15.98:0/3017187842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eac271f2d0 con 0x55eac26e6590 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88feae3640 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2abda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88fdae1640 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab6b30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f88e80a7470 tx=0x7f88e80a32a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a27c8784b17b3f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:26.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88e1ffb640 1 -- 172.21.15.98:0/3017187842 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e8010720 con 0x55eac26e6590 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88e1ffb640 1 -- 172.21.15.98:0/3017187842 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f88e80e2ac0 con 0x55eac26e6590 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f89073fb740 1 -- 172.21.15.98:0/3017187842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55eac2a1b350 con 0x55eac26e6590 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88e1ffb640 1 -- 172.21.15.98:0/3017187842 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88e80df1b0 con 0x55eac26e6590 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f89073fb740 10 client.15462 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f89073fb740 1 -- 172.21.15.98:0/3017187842 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eac27e96b0 con 0x55eac26e6590 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.098+0000 7f88e1ffb640 1 -- 172.21.15.98:0/3017187842 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55eac2a1b350 con 0x55eac26e6590 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.099+0000 7f88e1ffb640 1 -- 172.21.15.98:0/3017187842 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f88e80a37c0 con 0x55eac26e6590 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.099+0000 7f89073fb740 10 client.15462 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.099+0000 7f89073fb740 10 client.15462 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.099+0000 7f89073fb740 20 client.15462 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.099+0000 7f89073fb740 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55eac26c5e40 0x55eac2ac3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.099+0000 7f89073fb740 4 client.15462 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:26.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.099+0000 7f89073fb740 1 -- 172.21.15.98:0/3017187842 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eac2a151c0 con 0x55eac26c5e40 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.099+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55eac26c5e40 0x55eac2ac3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.100+0000 7f88e1ffb640 10 client.15462 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.100+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55eac26c5e40 0x55eac2ac3e00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e4dccb0da3be574b server_cookie=967d795e3562d6ba in_seq=0 out_seq=0 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.100+0000 7f88e1ffb640 1 -- 172.21.15.98:0/3017187842 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eac2a151c0 con 0x55eac26c5e40 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.100+0000 7f88e1ffb640 10 client.15462 handle_command_reply: tid=0 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 client.15462 shutdown 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 -- 172.21.15.98:0/3017187842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 msgr2=0x55eac2ab6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab6b30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f88e80a7470 tx=0x7f88e80a32a0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 -- 172.21.15.98:0/3017187842 shutdown_connections 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55eac26c5e40 0x55eac2ac3e00 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac2707270 0x55eac2abda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26e6590 0x55eac2ab6b30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f88e1ffb640 0 client.15462 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 --2- 172.21.15.98:0/3017187842 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2ab65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 -- 172.21.15.98:0/3017187842 >> 172.21.15.98:0/3017187842 conn(0x55eac26fb3c0 msgr2=0x55eac2aa2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 -- 172.21.15.98:0/3017187842 shutdown_connections 2024-04-25T16:32:26.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 1 -- 172.21.15.98:0/3017187842 wait complete. 2024-04-25T16:32:26.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.101+0000 7f89073fb740 20 client.15462 trim_cache size 0 max 16384 2024-04-25T16:32:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.119+0000 7f89073fb740 1 Processor -- start 2024-04-25T16:32:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.119+0000 7f89073fb740 1 -- start start 2024-04-25T16:32:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f89073fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aa3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f89073fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 0x55eac2aa3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f89073fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac2707270 0x55eac2aa2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f89073fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eac2ac2140 con 0x55eac2707270 2024-04-25T16:32:26.125 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f89073fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eac2ac2810 con 0x55eac26de3b0 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f89073fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eac2ac2ee0 con 0x55eac26e6590 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac2707270 0x55eac2aa2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac2707270 0x55eac2aa2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53330/0 (socket says 172.21.15.98:53330) 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 -- 172.21.15.98:0/452111977 learned_addr learned my addr 172.21.15.98:0/452111977 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88fdae1640 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 0x55eac2aa3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aa3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 -- 172.21.15.98:0/452111977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 msgr2=0x55eac2aa3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 0x55eac2aa3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 -- 172.21.15.98:0/452111977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 msgr2=0x55eac2aa3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aa3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 -- 172.21.15.98:0/452111977 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eac2ab1500 con 0x55eac2707270 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88fdae1640 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 0x55eac2aa3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aa3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:26.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88feae3640 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac2707270 0x55eac2aa2330 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f88f0003cd0 tx=0x7f88f0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212f5f6055805716 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88e3fff640 1 -- 172.21.15.98:0/452111977 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f00136c0 con 0x55eac2707270 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88e3fff640 1 -- 172.21.15.98:0/452111977 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f88f0017070 con 0x55eac2707270 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.120+0000 7f88e3fff640 1 -- 172.21.15.98:0/452111977 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f001da30 con 0x55eac2707270 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 -- 172.21.15.98:0/452111977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac2707270 msgr2=0x55eac2aa2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac2707270 0x55eac2aa2330 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f88f0003cd0 tx=0x7f88f0015000 comp rx=0 tx=0).stop 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 -- 172.21.15.98:0/452111977 shutdown_connections 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac2707270 0x55eac2aa2330 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 0x55eac2aa3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 --2- 172.21.15.98:0/452111977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac26de3b0 0x55eac2aa3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 -- 172.21.15.98:0/452111977 >> 172.21.15.98:0/452111977 conn(0x55eac26fb3c0 msgr2=0x55eac2ab4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 -- 172.21.15.98:0/452111977 shutdown_connections 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.121+0000 7f89073fb740 1 -- 172.21.15.98:0/452111977 wait complete. 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 1 -- start start 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 1 Processor -- start 2024-04-25T16:32:26.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26de3b0 0x55eac2ab8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 0x55eac2ab8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac2707270 0x55eac2abbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55eac2ac3c80 con 0x55eac26de3b0 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55eac2ac35b0 con 0x55eac2707270 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55eac2ac4350 con 0x55eac26e6590 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f89073fb740 10 client.0 _ll_register_callbacks cb 0x7f88f4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26de3b0 0x55eac2ab8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26de3b0 0x55eac2ab8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53338/0 (socket says 172.21.15.98:53338) 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 -- 172.21.15.98:0/486296103 learned_addr learned my addr 172.21.15.98:0/486296103 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88feae3640 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac2707270 0x55eac2abbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 -- 172.21.15.98:0/486296103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 msgr2=0x55eac2ab8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 0x55eac2ab8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 -- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac2707270 msgr2=0x55eac2abbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac2707270 0x55eac2abbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 -- 172.21.15.98:0/486296103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eac2abc470 con 0x55eac26de3b0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88feae3640 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac2707270 0x55eac2abbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88fe2e2640 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26de3b0 0x55eac2ab8a80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f88f4005ab0 tx=0x7f88f4004180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fb62c4a638cc08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.122+0000 7f88e1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f88e1ffb640 1 -- 172.21.15.98:0/486296103 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f4035040 con 0x55eac26de3b0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f88e1ffb640 1 -- 172.21.15.98:0/486296103 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f88f40319c0 con 0x55eac26de3b0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 1 -- 172.21.15.98:0/486296103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88f0017070 con 0x55eac26de3b0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f88e1ffb640 1 -- 172.21.15.98:0/486296103 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f88f4053050 con 0x55eac26de3b0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 10 client.15468 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 1 -- 172.21.15.98:0/486296103 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55eac2b2cdb0 con 0x55eac26de3b0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f88e1ffb640 1 -- 172.21.15.98:0/486296103 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f88f0017070 con 0x55eac26de3b0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f88e1ffb640 1 -- 172.21.15.98:0/486296103 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f88f404e030 con 0x55eac26de3b0 2024-04-25T16:32:26.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 10 client.15468 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 10 client.15468 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 20 client.15468 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55eac269b280 0x55eac2b30720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 4 client.15468 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f89073fb740 1 -- 172.21.15.98:0/486296103 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "status"}) -- 0x55eac29909a0 con 0x55eac269b280 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.123+0000 7f88fdae1640 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55eac269b280 0x55eac2b30720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f88e1ffb640 10 client.15468 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f88fdae1640 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55eac269b280 0x55eac2b30720 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fc2630486dc3e9e server_cookie=14adc9b6e8754ac3 in_seq=0 out_seq=0 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f88e1ffb640 1 -- 172.21.15.98:0/486296103 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55eac29909a0 con 0x55eac269b280 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f88e1ffb640 10 client.15468 handle_command_reply: tid=0 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 client.15468 shutdown 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 -- 172.21.15.98:0/486296103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26de3b0 msgr2=0x55eac2ab8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26de3b0 0x55eac2ab8a80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f88f4005ab0 tx=0x7f88f4004180 comp rx=0 tx=0).stop 2024-04-25T16:32:26.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 -- 172.21.15.98:0/486296103 shutdown_connections 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55eac269b280 0x55eac2b30720 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55eac2707270 0x55eac2abbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f88e1ffb640 0 client.15468 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55eac26e6590 0x55eac2ab8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 --2- 172.21.15.98:0/486296103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55eac26de3b0 0x55eac2ab8a80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 -- 172.21.15.98:0/486296103 >> 172.21.15.98:0/486296103 conn(0x55eac26fb3c0 msgr2=0x55eac2aa3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 -- 172.21.15.98:0/486296103 shutdown_connections 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 1 -- 172.21.15.98:0/486296103 wait complete. 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.124+0000 7f89073fb740 20 client.15468 trim_cache size 0 max 16384 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:26.131 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 3, 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24473, 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 110.098664132, 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 161.144056906 2024-04-25T16:32:26.132 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 -- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f88e40afaf0 msgr2=0x7f88e40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f88e40afaf0 0x7f88e40b2000 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f88f00098b0 tx=0x7f88f0004a40 comp rx=0 tx=0).stop 2024-04-25T16:32:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 -- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f8103c20 msgr2=0x7f88f819f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f8103c20 0x7f88f819f1f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f88e800e9f0 tx=0x7f88e800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 -- 172.21.15.98:0/4206727858 shutdown_connections 2024-04-25T16:32:26.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f88e40afaf0 0x7f88e40b2000 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f88f8108920 0x7f88f81a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f88f8103c20 0x7f88f819f1f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 --2- 172.21.15.98:0/4206727858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88f81029e0 0x7f88f806a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 -- 172.21.15.98:0/4206727858 >> 172.21.15.98:0/4206727858 conn(0x7f88f80fdbd0 msgr2=0x7f88f8104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 -- 172.21.15.98:0/4206727858 shutdown_connections 2024-04-25T16:32:26.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.125+0000 7f8900546640 1 -- 172.21.15.98:0/4206727858 wait complete. 2024-04-25T16:32:26.136 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T16:32:26.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 -- 172.21.15.98:0/3753235505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 msgr2=0x7ffb98104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/3753235505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 0x7ffb98104040 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009900 tx=0x7ffb8c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:26.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 -- 172.21.15.98:0/3753235505 shutdown_connections 2024-04-25T16:32:26.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/3753235505 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98108930 0x7ffb9810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/3753235505 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 0x7ffb98104040 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/3753235505 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb981029f0 0x7ffb981036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 -- 172.21.15.98:0/3753235505 >> 172.21.15.98:0/3753235505 conn(0x7ffb980fdc00 msgr2=0x7ffb98100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 -- 172.21.15.98:0/3753235505 shutdown_connections 2024-04-25T16:32:26.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.245+0000 7ffb9c88c640 1 -- 172.21.15.98:0/3753235505 wait complete. 2024-04-25T16:32:26.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9c88c640 1 Processor -- start 2024-04-25T16:32:26.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9c88c640 1 -- start start 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9c88c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb981029f0 0x7ffb9806bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9c88c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 0x7ffb9819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9c88c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98108930 0x7ffb981a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9c88c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffb9810e3d0 con 0x7ffb98103c30 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9c88c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffb9810e250 con 0x7ffb981029f0 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9c88c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffb9810e550 con 0x7ffb98108930 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb95d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 0x7ffb9819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb95d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 0x7ffb9819ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53350/0 (socket says 172.21.15.98:53350) 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb95d9b640 1 -- 172.21.15.98:0/4213576333 learned_addr learned my addr 172.21.15.98:0/4213576333 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98108930 0x7ffb981a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb9659c640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb981029f0 0x7ffb9806bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb95d9b640 1 -- 172.21.15.98:0/4213576333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98108930 msgr2=0x7ffb981a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98108930 0x7ffb981a14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb95d9b640 1 -- 172.21.15.98:0/4213576333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb981029f0 msgr2=0x7ffb9806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb981029f0 0x7ffb9806bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.246+0000 7ffb95d9b640 1 -- 172.21.15.98:0/4213576333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb981a1c70 con 0x7ffb98103c30 2024-04-25T16:32:26.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.247+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 0x7ffb9819ef50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009230 tx=0x7ffb8c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.247+0000 7ffb877fe640 1 -- 172.21.15.98:0/4213576333 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c03f030 con 0x7ffb98103c30 2024-04-25T16:32:26.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.247+0000 7ffb9c88c640 1 -- 172.21.15.98:0/4213576333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb981a1f00 con 0x7ffb98103c30 2024-04-25T16:32:26.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.247+0000 7ffb877fe640 1 -- 172.21.15.98:0/4213576333 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb8c003c90 con 0x7ffb98103c30 2024-04-25T16:32:26.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.247+0000 7ffb877fe640 1 -- 172.21.15.98:0/4213576333 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c035070 con 0x7ffb98103c30 2024-04-25T16:32:26.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.247+0000 7ffb9c88c640 1 -- 172.21.15.98:0/4213576333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb981a2310 con 0x7ffb98103c30 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.250+0000 7ffba3810740 1 Processor -- start 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.250+0000 7ffba3810740 1 -- start start 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.250+0000 7ffba3810740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4d4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.250+0000 7ffba3810740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea10a3b0 0x5654ea4d48f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.250+0000 7ffba3810740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4db7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.250+0000 7ffba3810740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5654ea439740 con 0x5654ea133270 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.250+0000 7ffba3810740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5654ea3d53a0 con 0x5654ea112590 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.250+0000 7ffba3810740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5654ea435700 con 0x5654ea10a3b0 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea10a3b0 0x5654ea4d48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea10a3b0 0x5654ea4d48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:35134/0 (socket says 172.21.15.98:35134) 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 -- 172.21.15.98:0/1372913462 learned_addr learned my addr 172.21.15.98:0/1372913462 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4d4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb877fe640 1 -- 172.21.15.98:0/4213576333 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ffb8c042030 con 0x7ffb98103c30 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4db7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 -- 172.21.15.98:0/1372913462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 msgr2=0x5654ea4db7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4db7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 -- 172.21.15.98:0/1372913462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 msgr2=0x5654ea4d4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4d4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 -- 172.21.15.98:0/1372913462 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654ea3f1750 con 0x5654ea10a3b0 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4db7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb9659c640 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea10a3b0 0x5654ea4d48f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffb900103a0 tx=0x7ffb90010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19c906b8d28cf894 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb84ff9640 1 -- 172.21.15.98:0/1372913462 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90016070 con 0x5654ea10a3b0 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb877fe640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffb800b3f80 0x7ffb800b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb84ff9640 1 -- 172.21.15.98:0/1372913462 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb90012ba0 con 0x5654ea10a3b0 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.251+0000 7ffb84ff9640 1 -- 172.21.15.98:0/1372913462 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9001bbe0 con 0x5654ea10a3b0 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.252+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffb800b3f80 0x7ffb800b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.252+0000 7ffb877fe640 1 -- 172.21.15.98:0/4213576333 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7ffb8c0ff590 con 0x7ffb98103c30 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.252+0000 7ffba3810740 1 -- 172.21.15.98:0/1372913462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea10a3b0 msgr2=0x5654ea4d48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.252+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffb800b3f80 0x7ffb800b6490 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x5654ea4d5540 tx=0x7ffb8800b2e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.252+0000 7ffba3810740 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea10a3b0 0x5654ea4d48f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffb900103a0 tx=0x7ffb90010860 comp rx=0 tx=0).stop 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 -- 172.21.15.98:0/1372913462 shutdown_connections 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4db7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea10a3b0 0x5654ea4d48f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 --2- 172.21.15.98:0/1372913462 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4d4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 -- 172.21.15.98:0/1372913462 >> 172.21.15.98:0/1372913462 conn(0x5654ea1273c0 msgr2=0x5654ea4cfd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 -- 172.21.15.98:0/1372913462 shutdown_connections 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 -- 172.21.15.98:0/1372913462 wait complete. 2024-04-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 -- start start 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 Processor -- start 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4de2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea112590 0x5654ea4de7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4e5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5654ea4dc240 con 0x5654ea133270 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5654ea3c0fe0 con 0x5654ea10a3b0 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5654ea4dc3c0 con 0x5654ea112590 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffba3810740 10 client.0 _ll_register_callbacks cb 0x5654ea3d7150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.253+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb95d9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea112590 0x5654ea4de7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4e5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53360/0 (socket says 172.21.15.98:53360) 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb96d9d640 1 -- 172.21.15.98:0/2720381983 learned_addr learned my addr 172.21.15.98:0/2720381983 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb9659c640 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4de2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb96d9d640 1 -- 172.21.15.98:0/2720381983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea112590 msgr2=0x5654ea4de7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea112590 0x5654ea4de7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb96d9d640 1 -- 172.21.15.98:0/2720381983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 msgr2=0x5654ea4de2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4de2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb96d9d640 1 -- 172.21.15.98:0/2720381983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654ea14b2d0 con 0x5654ea133270 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea112590 0x5654ea4de7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb9659c640 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4de2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4e5740 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ffb88004650 tx=0x7ffb88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d9562215e9b28b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2720381983 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8802c060 con 0x5654ea133270 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2720381983 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7ffb88017030 con 0x5654ea133270 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffba3810740 1 -- 172.21.15.98:0/2720381983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb90012ba0 con 0x5654ea133270 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2720381983 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffb88033050 con 0x5654ea133270 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffba3810740 10 client.15480 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:26.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.254+0000 7ffba3810740 1 -- 172.21.15.98:0/2720381983 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5654ea2156b0 con 0x5654ea133270 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2720381983 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb90012ba0 con 0x5654ea133270 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2720381983 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ffb88011440 con 0x5654ea133270 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffba3810740 10 client.15480 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffba3810740 10 client.15480 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffba3810740 20 client.15480 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffba3810740 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5654ea0f1e40 0x5654ea4ebf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffba3810740 4 client.15480 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffba3810740 1 -- 172.21.15.98:0/2720381983 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654ea447350 con 0x5654ea0f1e40 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.255+0000 7ffb9659c640 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5654ea0f1e40 0x5654ea4ebf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.256+0000 7ffb7dffb640 10 client.15480 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:32:26.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.256+0000 7ffb9659c640 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5654ea0f1e40 0x5654ea4ebf80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ccc592dd4b46ed6a server_cookie=4349da7a3e26e0e2 in_seq=0 out_seq=0 2024-04-25T16:32:26.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.256+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2720381983 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5654ea447350 con 0x5654ea0f1e40 2024-04-25T16:32:26.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.256+0000 7ffb7dffb640 10 client.15480 handle_command_reply: tid=0 2024-04-25T16:32:26.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.256+0000 7ffba3810740 1 client.15480 shutdown 2024-04-25T16:32:26.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 -- 172.21.15.98:0/2720381983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 msgr2=0x5654ea4e5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4e5740 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ffb88004650 tx=0x7ffb88004730 comp rx=0 tx=0).stop 2024-04-25T16:32:26.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 -- 172.21.15.98:0/2720381983 shutdown_connections 2024-04-25T16:32:26.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5654ea0f1e40 0x5654ea4ebf80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea133270 0x5654ea4e5740 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea112590 0x5654ea4de7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 --2- 172.21.15.98:0/2720381983 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4de2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffb7dffb640 0 client.15480 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:32:26.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 -- 172.21.15.98:0/2720381983 >> 172.21.15.98:0/2720381983 conn(0x5654ea1273c0 msgr2=0x5654ea4ced60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 -- 172.21.15.98:0/2720381983 shutdown_connections 2024-04-25T16:32:26.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 1 -- 172.21.15.98:0/2720381983 wait complete. 2024-04-25T16:32:26.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.257+0000 7ffba3810740 20 client.15480 trim_cache size 0 max 16384 2024-04-25T16:32:26.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffba3810740 1 Processor -- start 2024-04-25T16:32:26.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffba3810740 1 -- start start 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffba3810740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffba3810740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4e5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffba3810740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea112590 0x5654ea4cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffba3810740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5654ea4ea2c0 con 0x5654ea112590 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffba3810740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5654ea4ea990 con 0x5654ea10a3b0 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffba3810740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5654ea4eb060 con 0x5654ea0f1e40 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea112590 0x5654ea4cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb96d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea112590 0x5654ea4cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53372/0 (socket says 172.21.15.98:53372) 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb96d9d640 1 -- 172.21.15.98:0/482424635 learned_addr learned my addr 172.21.15.98:0/482424635 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4e5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb9659c640 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb96d9d640 1 -- 172.21.15.98:0/482424635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 msgr2=0x5654ea4cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb96d9d640 1 -- 172.21.15.98:0/482424635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 msgr2=0x5654ea4e5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4e5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb96d9d640 1 -- 172.21.15.98:0/482424635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654ea4d3f20 con 0x5654ea112590 2024-04-25T16:32:26.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4e5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:26.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.276+0000 7ffb9659c640 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.277+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea112590 0x5654ea4cfc20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffb88003a60 tx=0x7ffb88005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45594b0abd6b6cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.277+0000 7ffb7ffff640 1 -- 172.21.15.98:0/482424635 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8803c070 con 0x5654ea112590 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.277+0000 7ffb7ffff640 1 -- 172.21.15.98:0/482424635 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb8801c940 con 0x5654ea112590 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.277+0000 7ffb7ffff640 1 -- 172.21.15.98:0/482424635 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb88037690 con 0x5654ea112590 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 -- 172.21.15.98:0/482424635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea112590 msgr2=0x5654ea4cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea112590 0x5654ea4cfc20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffb88003a60 tx=0x7ffb88005970 comp rx=0 tx=0).stop 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 -- 172.21.15.98:0/482424635 shutdown_connections 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea112590 0x5654ea4cfc20 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea10a3b0 0x5654ea4e5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.285 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 --2- 172.21.15.98:0/482424635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 -- 172.21.15.98:0/482424635 >> 172.21.15.98:0/482424635 conn(0x5654ea1273c0 msgr2=0x5654ea4d86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 -- 172.21.15.98:0/482424635 shutdown_connections 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 -- 172.21.15.98:0/482424635 wait complete. 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 -- start start 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.278+0000 7ffba3810740 1 Processor -- start 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4e0930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea10a3b0 0x5654ea4e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4e3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5654ea4ebe00 con 0x5654ea10a3b0 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5654ea4eb730 con 0x5654ea112590 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5654ea4ec4d0 con 0x5654ea0f1e40 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 10 client.0 _ll_register_callbacks cb 0x7ffb880041d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea10a3b0 0x5654ea4e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea10a3b0 0x5654ea4e0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53384/0 (socket says 172.21.15.98:53384) 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 -- 172.21.15.98:0/2545888667 learned_addr learned my addr 172.21.15.98:0/2545888667 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb9659c640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4e0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4e3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 -- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 msgr2=0x5654ea4e0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4e0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 -- 172.21.15.98:0/2545888667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 msgr2=0x5654ea4e3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4e3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 -- 172.21.15.98:0/2545888667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654ea4e4300 con 0x5654ea10a3b0 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb9659c640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4e0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb96d9d640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4e3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb95d9b640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea10a3b0 0x5654ea4e0e70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c000c00 tx=0x7ffb8c0c7120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef18aad7bc91702e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:26.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2545888667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c049040 con 0x5654ea10a3b0 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2545888667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7ffb8c0ff1c0 con 0x5654ea10a3b0 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 1 -- 172.21.15.98:0/2545888667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffb8801c940 con 0x5654ea10a3b0 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2545888667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffb8c0c7710 con 0x5654ea10a3b0 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 10 client.15486 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.279+0000 7ffba3810740 1 -- 172.21.15.98:0/2545888667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5654ea4e5580 con 0x5654ea10a3b0 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2545888667 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffb8801c940 con 0x5654ea10a3b0 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2545888667 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ffb8c0c7970 con 0x5654ea10a3b0 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffba3810740 10 client.15486 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffba3810740 10 client.15486 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffba3810740 20 client.15486 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffba3810740 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5654ea133270 0x5654ea555870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffba3810740 4 client.15486 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffba3810740 1 -- 172.21.15.98:0/2545888667 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "status"}) -- 0x5654ea4411c0 con 0x5654ea133270 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.280+0000 7ffb9659c640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5654ea133270 0x5654ea555870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffb9659c640 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5654ea133270 0x5654ea555870 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6c715eb5dc3e3a17 server_cookie=9e1756cba42c2a6 in_seq=0 out_seq=0 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffb7dffb640 10 client.15486 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffb7dffb640 1 -- 172.21.15.98:0/2545888667 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5654ea4411c0 con 0x5654ea133270 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffb7dffb640 10 client.15486 handle_command_reply: tid=0 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 client.15486 shutdown 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 -- 172.21.15.98:0/2545888667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea10a3b0 msgr2=0x5654ea4e0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea10a3b0 0x5654ea4e0e70 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c000c00 tx=0x7ffb8c0c7120 comp rx=0 tx=0).stop 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 -- 172.21.15.98:0/2545888667 shutdown_connections 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5654ea133270 0x5654ea555870 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5654ea112590 0x5654ea4e3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5654ea10a3b0 0x5654ea4e0e70 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffb7dffb640 0 client.15486 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 --2- 172.21.15.98:0/2545888667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5654ea0f1e40 0x5654ea4e0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 -- 172.21.15.98:0/2545888667 >> 172.21.15.98:0/2545888667 conn(0x5654ea1273c0 msgr2=0x5654ea4cf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 -- 172.21.15.98:0/2545888667 shutdown_connections 2024-04-25T16:32:26.289 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 1 -- 172.21.15.98:0/2545888667 wait complete. 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.281+0000 7ffba3810740 20 client.15486 trim_cache size 0 max 16384 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 2, 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24484, 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:32:26.290 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 111.263970341, 2024-04-25T16:32:26.291 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:32:26.291 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-04-25T16:32:26.291 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:32:26.291 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 149.926929441 2024-04-25T16:32:26.291 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 -- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffb800b3f80 msgr2=0x7ffb800b6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffb800b3f80 0x7ffb800b6490 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x5654ea4d5540 tx=0x7ffb8800b2e0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 -- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 msgr2=0x7ffb9819ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 0x7ffb9819ef50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009230 tx=0x7ffb8c0042f0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 -- 172.21.15.98:0/4213576333 shutdown_connections 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffb800b3f80 0x7ffb800b6490 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffb98108930 0x7ffb981a14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffb98103c30 0x7ffb9819ef50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 --2- 172.21.15.98:0/4213576333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb981029f0 0x7ffb9806bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 -- 172.21.15.98:0/4213576333 >> 172.21.15.98:0/4213576333 conn(0x7ffb980fdc00 msgr2=0x7ffb98104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 -- 172.21.15.98:0/4213576333 shutdown_connections 2024-04-25T16:32:26.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.282+0000 7ffb9c88c640 1 -- 172.21.15.98:0/4213576333 wait complete. 2024-04-25T16:32:26.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T16:32:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 -- 172.21.15.98:0/1110912927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c106f40 msgr2=0x7f911c109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 --2- 172.21.15.98:0/1110912927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c106f40 0x7f911c109360 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f910c009900 tx=0x7f910c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 -- 172.21.15.98:0/1110912927 shutdown_connections 2024-04-25T16:32:26.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 --2- 172.21.15.98:0/1110912927 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f911c1098a0 0x7f911c10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 --2- 172.21.15.98:0/1110912927 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c106f40 0x7f911c109360 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 --2- 172.21.15.98:0/1110912927 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f911c1045e0 0x7f911c106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 -- 172.21.15.98:0/1110912927 >> 172.21.15.98:0/1110912927 conn(0x7f911c0fdbc0 msgr2=0x7f911c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 -- 172.21.15.98:0/1110912927 shutdown_connections 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 -- 172.21.15.98:0/1110912927 wait complete. 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 Processor -- start 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.404+0000 7f9123972640 1 -- start start 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9123972640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c1045e0 0x7f911c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9123972640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f911c106f40 0x7f911c19a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9123972640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f911c1098a0 0x7f911c1a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9123972640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f911c10f4d0 con 0x7f911c1045e0 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9123972640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f911c10f350 con 0x7f911c1098a0 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9123972640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f911c10f650 con 0x7f911c106f40 2024-04-25T16:32:26.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c1045e0 0x7f911c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c1045e0 0x7f911c19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53406/0 (socket says 172.21.15.98:53406) 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 -- 172.21.15.98:0/769735069 learned_addr learned my addr 172.21.15.98:0/769735069 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9121f0f640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f911c1098a0 0x7f911c1a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 -- 172.21.15.98:0/769735069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f911c106f40 msgr2=0x7f911c19a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f911c106f40 0x7f911c19a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 -- 172.21.15.98:0/769735069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f911c1098a0 msgr2=0x7f911c1a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f911c1098a0 0x7f911c1a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 -- 172.21.15.98:0/769735069 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911c1a1cb0 con 0x7f911c1045e0 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f912170e640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c1045e0 0x7f911c19a170 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9118005fd0 tx=0x7f911800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f91127fc640 1 -- 172.21.15.98:0/769735069 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118017070 con 0x7f911c1045e0 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f91127fc640 1 -- 172.21.15.98:0/769735069 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9118009c30 con 0x7f911c1045e0 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9123972640 1 -- 172.21.15.98:0/769735069 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f911c1a1fa0 con 0x7f911c1045e0 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f91127fc640 1 -- 172.21.15.98:0/769735069 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118003c70 con 0x7f911c1045e0 2024-04-25T16:32:26.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.405+0000 7f9123972640 1 -- 172.21.15.98:0/769735069 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f911c1a2460 con 0x7f911c1045e0 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f912a81e740 1 Processor -- start 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f912a81e740 1 -- start start 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f91127fc640 1 -- 172.21.15.98:0/769735069 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9118003e10 con 0x7f911c1045e0 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f912a81e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60388270 0x559b60729370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f912a81e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b607298f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f912a81e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b607307f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f912a81e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559b6068e740 con 0x559b6035f3b0 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f912a81e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559b6062a3a0 con 0x559b60367590 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.408+0000 7f912a81e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b6068a700 con 0x559b60388270 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b607298f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b607298f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53408/0 (socket says 172.21.15.98:53408) 2024-04-25T16:32:26.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 -- 172.21.15.98:0/1836265204 learned_addr learned my addr 172.21.15.98:0/1836265204 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f9120f0d640 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b607307f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f9121f0f640 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60388270 0x559b60729370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 -- 172.21.15.98:0/1836265204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60388270 msgr2=0x559b60729370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60388270 0x559b60729370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 -- 172.21.15.98:0/1836265204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 msgr2=0x559b607307f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b607307f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 -- 172.21.15.98:0/1836265204 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b60646750 con 0x559b6035f3b0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f91127fc640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91080afaf0 0x7f91080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f9121f0f640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91080afaf0 0x7f91080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f912170e640 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b607298f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f91180e2fc0 tx=0x7f91180e78d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a04f49aaed52c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f91127fc640 1 -- 172.21.15.98:0/769735069 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f91180ded40 con 0x7f911c1045e0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.409+0000 7f91077fe640 1 -- 172.21.15.98:0/1836265204 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91180a2d30 con 0x559b6035f3b0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f91077fe640 1 -- 172.21.15.98:0/1836265204 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f91180035f0 con 0x559b6035f3b0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f91077fe640 1 -- 172.21.15.98:0/1836265204 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91180a65e0 con 0x559b6035f3b0 2024-04-25T16:32:26.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 -- 172.21.15.98:0/1836265204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 msgr2=0x559b607298f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b607298f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f91180e2fc0 tx=0x7f91180e78d0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f9121f0f640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91080afaf0 0x7f91080b2000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x559b6072a540 tx=0x7f911400b340 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 -- 172.21.15.98:0/1836265204 shutdown_connections 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b607307f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b607298f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 --2- 172.21.15.98:0/1836265204 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60388270 0x559b60729370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 -- 172.21.15.98:0/1836265204 >> 172.21.15.98:0/1836265204 conn(0x559b6037c3c0 msgr2=0x559b60724d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 -- 172.21.15.98:0/1836265204 shutdown_connections 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.410+0000 7f912a81e740 1 -- 172.21.15.98:0/1836265204 wait complete. 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 1 -- start start 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 1 Processor -- start 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60733290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b607337d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b6073a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.417 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559b60731240 con 0x559b60388270 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559b60615fe0 con 0x559b60367590 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b607313c0 con 0x559b6035f3b0 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f912a81e740 10 client.0 _ll_register_callbacks cb 0x559b6062c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f9121f0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b6073a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f9121f0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b6073a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53416/0 (socket says 172.21.15.98:53416) 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f9121f0f640 1 -- 172.21.15.98:0/3324079630 learned_addr learned my addr 172.21.15.98:0/3324079630 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.411+0000 7f9120f0d640 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b607337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f912170e640 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60733290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9121f0f640 1 -- 172.21.15.98:0/3324079630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 msgr2=0x559b60733290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9121f0f640 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60733290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9121f0f640 1 -- 172.21.15.98:0/3324079630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 msgr2=0x559b607337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9121f0f640 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b607337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9121f0f640 1 -- 172.21.15.98:0/3324079630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b603a02d0 con 0x559b60388270 2024-04-25T16:32:26.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f912170e640 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60733290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9121f0f640 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b6073a730 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f91140042b0 tx=0x7f9114004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1acf679b61f8e29a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9104ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9104ff9640 1 -- 172.21.15.98:0/3324079630 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114022020 con 0x559b60388270 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f912a81e740 1 -- 172.21.15.98:0/3324079630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x559b6069c350 con 0x559b60388270 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9104ff9640 1 -- 172.21.15.98:0/3324079630 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f9114010bb0 con 0x559b60388270 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9104ff9640 1 -- 172.21.15.98:0/3324079630 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f911402d050 con 0x559b60388270 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f912a81e740 10 client.15498 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f912a81e740 1 -- 172.21.15.98:0/3324079630 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b6046a6b0 con 0x559b60388270 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.412+0000 7f9104ff9640 1 -- 172.21.15.98:0/3324079630 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x559b6069c350 con 0x559b60388270 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f9104ff9640 1 -- 172.21.15.98:0/3324079630 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f911401d390 con 0x559b60388270 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f912a81e740 10 client.15498 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f912a81e740 10 client.15498 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f912a81e740 20 client.15498 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:26.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f912a81e740 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x559b60346e40 0x559b60741160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f912a81e740 4 client.15498 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f912a81e740 1 -- 172.21.15.98:0/3324079630 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b606961c0 con 0x559b60346e40 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f912170e640 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x559b60346e40 0x559b60741160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f9104ff9640 10 client.15498 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.413+0000 7f912170e640 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x559b60346e40 0x559b60741160 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=67f9b244d4684832 server_cookie=c46db34aea784d02 in_seq=0 out_seq=0 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f9104ff9640 1 -- 172.21.15.98:0/3324079630 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559b606961c0 con 0x559b60346e40 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f9104ff9640 10 client.15498 handle_command_reply: tid=0 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 client.15498 shutdown 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 -- 172.21.15.98:0/3324079630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 msgr2=0x559b6073a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b6073a730 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f91140042b0 tx=0x7f9114004a40 comp rx=0 tx=0).stop 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 -- 172.21.15.98:0/3324079630 shutdown_connections 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x559b60346e40 0x559b60741160 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b6073a730 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b607337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 --2- 172.21.15.98:0/3324079630 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60733290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f9104ff9640 0 client.15498 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:32:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 -- 172.21.15.98:0/3324079630 >> 172.21.15.98:0/3324079630 conn(0x559b6037c3c0 msgr2=0x559b60723d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 -- 172.21.15.98:0/3324079630 shutdown_connections 2024-04-25T16:32:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.414+0000 7f912a81e740 1 -- 172.21.15.98:0/3324079630 wait complete. 2024-04-25T16:32:26.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.415+0000 7f912a81e740 20 client.15498 trim_cache size 0 max 16384 2024-04-25T16:32:26.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:26 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1400296525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:26.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:26 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4109057455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:26.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:26 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1304106141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:26.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:26 smithi203 ceph-mon[33115]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T16:32:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912a81e740 1 Processor -- start 2024-04-25T16:32:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912a81e740 1 -- start start 2024-04-25T16:32:26.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912a81e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b6072c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912a81e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60367590 0x559b6073a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912a81e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60388270 0x559b607246b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912a81e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559b6073f4a0 con 0x559b6035f3b0 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912a81e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559b6073fb70 con 0x559b60388270 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912a81e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b60740240 con 0x559b60367590 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.433+0000 7f912170e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b6072c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f912170e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b6072c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53422/0 (socket says 172.21.15.98:53422) 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f912170e640 1 -- 172.21.15.98:0/3781360015 learned_addr learned my addr 172.21.15.98:0/3781360015 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f9120f0d640 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60367590 0x559b6073a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f9121f0f640 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60388270 0x559b607246b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f912170e640 1 -- 172.21.15.98:0/3781360015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60367590 msgr2=0x559b6073a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f912170e640 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60367590 0x559b6073a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f912170e640 1 -- 172.21.15.98:0/3781360015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60388270 msgr2=0x559b607246b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f912170e640 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60388270 0x559b607246b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f912170e640 1 -- 172.21.15.98:0/3781360015 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b6072e0a0 con 0x559b6035f3b0 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f9120f0d640 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60367590 0x559b6073a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f9121f0f640 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60388270 0x559b607246b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f912170e640 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b6072c920 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f91180dc800 tx=0x7f91180e2900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1c320ca3c53759 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f9106ffd640 1 -- 172.21.15.98:0/3781360015 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91180a67c0 con 0x559b6035f3b0 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f9106ffd640 1 -- 172.21.15.98:0/3781360015 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9118009de0 con 0x559b6035f3b0 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.434+0000 7f9106ffd640 1 -- 172.21.15.98:0/3781360015 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91180a6c40 con 0x559b6035f3b0 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 -- 172.21.15.98:0/3781360015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 msgr2=0x559b6072c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b6072c920 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f91180dc800 tx=0x7f91180e2900 comp rx=0 tx=0).stop 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 -- 172.21.15.98:0/3781360015 shutdown_connections 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60388270 0x559b607246b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b60367590 0x559b6073a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 --2- 172.21.15.98:0/3781360015 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b6035f3b0 0x559b6072c920 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 -- 172.21.15.98:0/3781360015 >> 172.21.15.98:0/3781360015 conn(0x559b6037c3c0 msgr2=0x559b607318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 -- 172.21.15.98:0/3781360015 shutdown_connections 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 -- 172.21.15.98:0/3781360015 wait complete. 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 -- start start 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.435+0000 7f912a81e740 1 Processor -- start 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912a81e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60735a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912a81e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b60735fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912a81e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b60738ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912a81e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559b60740fe0 con 0x559b60388270 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912a81e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559b60740910 con 0x559b60367590 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912a81e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559b607416b0 con 0x559b6035f3b0 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912a81e740 10 client.0 _ll_register_callbacks cb 0x7f911400b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b60738ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b60738ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53424/0 (socket says 172.21.15.98:53424) 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 -- 172.21.15.98:0/960074214 learned_addr learned my addr 172.21.15.98:0/960074214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9120f0d640 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b60735fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912170e640 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60735a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 -- 172.21.15.98:0/960074214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 msgr2=0x559b60735a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60735a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 -- 172.21.15.98:0/960074214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 msgr2=0x559b60735fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b60735fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 -- 172.21.15.98:0/960074214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b607394a0 con 0x559b60388270 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912170e640 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60735a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9121f0f640 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b60738ee0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f911401d780 tx=0x7f911400fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2c9b87a27da805 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9104ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9104ff9640 1 -- 172.21.15.98:0/960074214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114022040 con 0x559b60388270 2024-04-25T16:32:26.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f9104ff9640 1 -- 172.21.15.98:0/960074214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f9114010a00 con 0x559b60388270 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.436+0000 7f912a81e740 1 -- 172.21.15.98:0/960074214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9118009de0 con 0x559b60388270 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f9104ff9640 1 -- 172.21.15.98:0/960074214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9114042050 con 0x559b60388270 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912a81e740 10 client.15504 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912a81e740 1 -- 172.21.15.98:0/960074214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b6073a780 con 0x559b60388270 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f9104ff9640 1 -- 172.21.15.98:0/960074214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9118009de0 con 0x559b60388270 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f9104ff9640 1 -- 172.21.15.98:0/960074214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f911401e950 con 0x559b60388270 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912a81e740 10 client.15504 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912a81e740 10 client.15504 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912a81e740 20 client.15504 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912a81e740 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x559b6031c280 0x559b607ad3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912a81e740 4 client.15504 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912a81e740 1 -- 172.21.15.98:0/960074214 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "status"}) -- 0x559b606119a0 con 0x559b6031c280 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.437+0000 7f912170e640 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x559b6031c280 0x559b607ad3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f9104ff9640 10 client.15504 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912170e640 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x559b6031c280 0x559b607ad3e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f15f34597d533b7 server_cookie=3c0f5bcd386c37d in_seq=0 out_seq=0 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f9104ff9640 1 -- 172.21.15.98:0/960074214 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559b606119a0 con 0x559b6031c280 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f9104ff9640 10 client.15504 handle_command_reply: tid=0 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 client.15504 shutdown 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 -- 172.21.15.98:0/960074214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 msgr2=0x559b60738ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b60738ee0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f911401d780 tx=0x7f911400fd30 comp rx=0 tx=0).stop 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 -- 172.21.15.98:0/960074214 shutdown_connections 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x559b6031c280 0x559b607ad3e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559b60388270 0x559b60738ee0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559b60367590 0x559b60735fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f9104ff9640 0 client.15504 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 --2- 172.21.15.98:0/960074214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559b6035f3b0 0x559b60735a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 -- 172.21.15.98:0/960074214 >> 172.21.15.98:0/960074214 conn(0x559b6037c3c0 msgr2=0x559b6072c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 -- 172.21.15.98:0/960074214 shutdown_connections 2024-04-25T16:32:26.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 1 -- 172.21.15.98:0/960074214 wait complete. 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.438+0000 7f912a81e740 20 client.15504 trim_cache size 0 max 16384 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 1, 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24494, 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 112.43205005999999, 2024-04-25T16:32:26.446 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:32:26.447 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-04-25T16:32:26.447 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:32:26.447 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 152.96095466099999 2024-04-25T16:32:26.447 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 -- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91080afaf0 msgr2=0x7f91080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91080afaf0 0x7f91080b2000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x559b6072a540 tx=0x7f911400b340 comp rx=0 tx=0).stop 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 -- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c1045e0 msgr2=0x7f911c19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c1045e0 0x7f911c19a170 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9118005fd0 tx=0x7f911800dde0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 -- 172.21.15.98:0/769735069 shutdown_connections 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f91080afaf0 0x7f91080b2000 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f911c1098a0 0x7f911c1a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f911c106f40 0x7f911c19a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 --2- 172.21.15.98:0/769735069 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f911c1045e0 0x7f911c19a170 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 -- 172.21.15.98:0/769735069 >> 172.21.15.98:0/769735069 conn(0x7f911c0fdbc0 msgr2=0x7f911c106630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 -- 172.21.15.98:0/769735069 shutdown_connections 2024-04-25T16:32:26.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.439+0000 7f9123972640 1 -- 172.21.15.98:0/769735069 wait complete. 2024-04-25T16:32:26.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:26 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1400296525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:26.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:26 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4109057455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:26.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:26 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1304106141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:26.449 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:26 smithi098 ceph-mon[29162]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T16:32:26.450 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T16:32:26.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:26 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1400296525' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:26.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:26 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4109057455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:26.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:26 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1304106141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:26.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:26 smithi173 ceph-mon[32556]: pgmap v354: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-04-25T16:32:26.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 -- 172.21.15.98:0/1118159003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248102a00 msgr2=0x7f8248103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 --2- 172.21.15.98:0/1118159003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248102a00 0x7f8248103700 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8238009900 tx=0x7f823802a410 comp rx=0 tx=0).stop 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 -- 172.21.15.98:0/1118159003 shutdown_connections 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 --2- 172.21.15.98:0/1118159003 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8248108940 0x7f824810ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 --2- 172.21.15.98:0/1118159003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8248103c40 0x7f8248104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 --2- 172.21.15.98:0/1118159003 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248102a00 0x7f8248103700 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 -- 172.21.15.98:0/1118159003 >> 172.21.15.98:0/1118159003 conn(0x7f82480fdbf0 msgr2=0x7f82481000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 -- 172.21.15.98:0/1118159003 shutdown_connections 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.560+0000 7f824d527640 1 -- 172.21.15.98:0/1118159003 wait complete. 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f824d527640 1 Processor -- start 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f824d527640 1 -- start start 2024-04-25T16:32:26.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f824d527640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8248102a00 0x7f8248199140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f824d527640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8248103c40 0x7f8248199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f824d527640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248108940 0x7f82481a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f824d527640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f824810e3b0 con 0x7f8248108940 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f824d527640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f824810e230 con 0x7f8248103c40 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f824d527640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f824810e530 con 0x7f8248102a00 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f82477fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248108940 0x7f82481a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f82477fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248108940 0x7f82481a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53452/0 (socket says 172.21.15.98:53452) 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f82477fe640 1 -- 172.21.15.98:0/4233735681 learned_addr learned my addr 172.21.15.98:0/4233735681 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f8246ffd640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8248102a00 0x7f8248199140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f82477fe640 1 -- 172.21.15.98:0/4233735681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8248102a00 msgr2=0x7f8248199140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f82477fe640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8248102a00 0x7f8248199140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.561+0000 7f82477fe640 1 -- 172.21.15.98:0/4233735681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8248103c40 msgr2=0x7f8248199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f823ffff640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8248103c40 0x7f8248199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f82477fe640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8248103c40 0x7f8248199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f82477fe640 1 -- 172.21.15.98:0/4233735681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82481a0b80 con 0x7f8248108940 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f8246ffd640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8248102a00 0x7f8248199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f82477fe640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248108940 0x7f82481a05c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f824000eb10 tx=0x7f824000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f8244ff9640 1 -- 172.21.15.98:0/4233735681 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240004b20 con 0x7f8248108940 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f8244ff9640 1 -- 172.21.15.98:0/4233735681 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8240004cc0 con 0x7f8248108940 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f824d527640 1 -- 172.21.15.98:0/4233735681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82481a0e70 con 0x7f8248108940 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f8244ff9640 1 -- 172.21.15.98:0/4233735681 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8240010690 con 0x7f8248108940 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.562+0000 7f824d527640 1 -- 172.21.15.98:0/4233735681 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82481a12d0 con 0x7f8248108940 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.564+0000 7f8244ff9640 1 -- 172.21.15.98:0/4233735681 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8240005020 con 0x7f8248108940 2024-04-25T16:32:26.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.564+0000 7f82543d6740 1 Processor -- start 2024-04-25T16:32:26.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.564+0000 7f82543d6740 1 -- start start 2024-04-25T16:32:26.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f82543d6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 0x55ee07e7b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f82543d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07aaf3b0 0x55ee07e7bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f82543d6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ab7590 0x55ee07e82a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f82543d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee07dde740 con 0x55ee07aaf3b0 2024-04-25T16:32:26.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f82543d6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ee07d7a3a0 con 0x55ee07ab7590 2024-04-25T16:32:26.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f82543d6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee07dda700 con 0x55ee07ad8270 2024-04-25T16:32:26.574 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8246ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07aaf3b0 0x55ee07e7bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8246ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07aaf3b0 0x55ee07e7bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53454/0 (socket says 172.21.15.98:53454) 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8246ffd640 1 -- 172.21.15.98:0/2115311119 learned_addr learned my addr 172.21.15.98:0/2115311119 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f823ffff640 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ab7590 0x55ee07e82a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8244ff9640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f82340afaf0 0x7f82340b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f823ffff640 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 0x55ee07e7b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8246ffd640 1 -- 172.21.15.98:0/2115311119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 msgr2=0x55ee07e7b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8246ffd640 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 0x55ee07e7b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8246ffd640 1 -- 172.21.15.98:0/2115311119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ab7590 msgr2=0x55ee07e82a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8246ffd640 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ab7590 0x55ee07e82a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8246ffd640 1 -- 172.21.15.98:0/2115311119 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee07d96750 con 0x55ee07aaf3b0 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f82477fe640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f82340afaf0 0x7f82340b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.575 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.565+0000 7f8244ff9640 1 -- 172.21.15.98:0/4233735681 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f82400df650 con 0x7f8248108940 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.566+0000 7f8246ffd640 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07aaf3b0 0x55ee07e7bb00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f82380096c0 tx=0x7f82380317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7f250fccb84582 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.566+0000 7f823d7fa640 1 -- 172.21.15.98:0/2115311119 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238035030 con 0x55ee07aaf3b0 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.566+0000 7f823d7fa640 1 -- 172.21.15.98:0/2115311119 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82380044b0 con 0x55ee07aaf3b0 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.566+0000 7f823d7fa640 1 -- 172.21.15.98:0/2115311119 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238005580 con 0x55ee07aaf3b0 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- 172.21.15.98:0/2115311119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07aaf3b0 msgr2=0x55ee07e7bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82477fe640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f82340afaf0 0x7f82340b2000 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f82400a7480 tx=0x7f82400ca040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07aaf3b0 0x55ee07e7bb00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f82380096c0 tx=0x7f82380317c0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- 172.21.15.98:0/2115311119 shutdown_connections 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ab7590 0x55ee07e82a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07aaf3b0 0x55ee07e7bb00 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.576 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 --2- 172.21.15.98:0/2115311119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 0x55ee07e7b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- 172.21.15.98:0/2115311119 >> 172.21.15.98:0/2115311119 conn(0x55ee07acc3c0 msgr2=0x55ee07e74d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- 172.21.15.98:0/2115311119 shutdown_connections 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- 172.21.15.98:0/2115311119 wait complete. 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- start start 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 Processor -- start 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07aaf3b0 0x55ee07e853e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e85920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ad8270 0x55ee07e8c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee07e833f0 con 0x55ee07ab7590 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ee07d65fe0 con 0x55ee07ad8270 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee07e83570 con 0x55ee07aaf3b0 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.567+0000 7f82543d6740 10 client.0 _ll_register_callbacks cb 0x55ee07d7c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e85920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e85920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53458/0 (socket says 172.21.15.98:53458) 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 -- 172.21.15.98:0/602694389 learned_addr learned my addr 172.21.15.98:0/602694389 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.577 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f823ffff640 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07aaf3b0 0x55ee07e853e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 -- 172.21.15.98:0/602694389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07aaf3b0 msgr2=0x55ee07e853e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07aaf3b0 0x55ee07e853e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 -- 172.21.15.98:0/602694389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ad8270 msgr2=0x55ee07e8c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f823ffff640 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ad8270 0x55ee07e8c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ad8270 0x55ee07e8c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 -- 172.21.15.98:0/602694389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee07af02d0 con 0x55ee07ab7590 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f8246ffd640 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e85920 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f82380057c0 tx=0x7f8238005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8558a4f0eadc3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f822e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f822e7fc640 1 -- 172.21.15.98:0/602694389 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8238035030 con 0x55ee07ab7590 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f822e7fc640 1 -- 172.21.15.98:0/602694389 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f82380048d0 con 0x55ee07ab7590 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f82543d6740 1 -- 172.21.15.98:0/602694389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82380044b0 con 0x55ee07ab7590 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f822e7fc640 1 -- 172.21.15.98:0/602694389 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8238056050 con 0x55ee07ab7590 2024-04-25T16:32:26.578 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f82543d6740 10 client.15516 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.568+0000 7f82543d6740 1 -- 172.21.15.98:0/602694389 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ee07bba6b0 con 0x55ee07ab7590 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f822e7fc640 1 -- 172.21.15.98:0/602694389 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f82380044b0 con 0x55ee07ab7590 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f822e7fc640 1 -- 172.21.15.98:0/602694389 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f82380438e0 con 0x55ee07ab7590 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f82543d6740 10 client.15516 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f82543d6740 10 client.15516 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f82543d6740 20 client.15516 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f82543d6740 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55ee07a96e40 0x55ee07e92d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f82543d6740 4 client.15516 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f82543d6740 1 -- 172.21.15.98:0/602694389 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee07dec350 con 0x55ee07a96e40 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f823ffff640 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55ee07a96e40 0x55ee07e92d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f822e7fc640 10 client.15516 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.569+0000 7f823ffff640 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55ee07a96e40 0x55ee07e92d00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f058a334d54bc0c server_cookie=4e7ff23429edf626 in_seq=0 out_seq=0 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f822e7fc640 1 -- 172.21.15.98:0/602694389 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ee07dec350 con 0x55ee07a96e40 2024-04-25T16:32:26.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f822e7fc640 10 client.15516 handle_command_reply: tid=0 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 client.15516 shutdown 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 -- 172.21.15.98:0/602694389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 msgr2=0x55ee07e85920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e85920 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f82380057c0 tx=0x7f8238005f30 comp rx=0 tx=0).stop 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 -- 172.21.15.98:0/602694389 shutdown_connections 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55ee07a96e40 0x55ee07e92d00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07ad8270 0x55ee07e8c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e85920 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 --2- 172.21.15.98:0/602694389 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07aaf3b0 0x55ee07e853e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f822e7fc640 0 client.15516 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 -- 172.21.15.98:0/602694389 >> 172.21.15.98:0/602694389 conn(0x55ee07acc3c0 msgr2=0x55ee07e73d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 -- 172.21.15.98:0/602694389 shutdown_connections 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 1 -- 172.21.15.98:0/602694389 wait complete. 2024-04-25T16:32:26.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.570+0000 7f82543d6740 20 client.15516 trim_cache size 0 max 16384 2024-04-25T16:32:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f82543d6740 1 Processor -- start 2024-04-25T16:32:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f82543d6740 1 -- start start 2024-04-25T16:32:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f82543d6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 0x55ee07e840b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f82543d6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ab7590 0x55ee07e6fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f82543d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ad8270 0x55ee07e746b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f82543d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee07e91040 con 0x55ee07ad8270 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f82543d6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ee07e91710 con 0x55ee07aaf3b0 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f82543d6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee07e91de0 con 0x55ee07ab7590 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f823ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ad8270 0x55ee07e746b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f823ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ad8270 0x55ee07e746b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53460/0 (socket says 172.21.15.98:53460) 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f823ffff640 1 -- 172.21.15.98:0/2496695636 learned_addr learned my addr 172.21.15.98:0/2496695636 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f8246ffd640 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ab7590 0x55ee07e6fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.589+0000 7f823ffff640 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 0x55ee07e840b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823ffff640 1 -- 172.21.15.98:0/2496695636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ab7590 msgr2=0x55ee07e6fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823ffff640 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ab7590 0x55ee07e6fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823ffff640 1 -- 172.21.15.98:0/2496695636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 msgr2=0x55ee07e840b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823ffff640 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 0x55ee07e840b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823ffff640 1 -- 172.21.15.98:0/2496695636 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee07e80260 con 0x55ee07ad8270 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823ffff640 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ad8270 0x55ee07e746b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f823000def0 tx=0x7f8230005b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98228b3fa7406819 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823cff9640 1 -- 172.21.15.98:0/2496695636 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230013b30 con 0x55ee07ad8270 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823cff9640 1 -- 172.21.15.98:0/2496695636 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8230023070 con 0x55ee07ad8270 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.590+0000 7f823cff9640 1 -- 172.21.15.98:0/2496695636 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230004b20 con 0x55ee07ad8270 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- 172.21.15.98:0/2496695636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ad8270 msgr2=0x55ee07e746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ad8270 0x55ee07e746b0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f823000def0 tx=0x7f8230005b60 comp rx=0 tx=0).stop 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- 172.21.15.98:0/2496695636 shutdown_connections 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ad8270 0x55ee07e746b0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ab7590 0x55ee07e6fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 --2- 172.21.15.98:0/2496695636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 0x55ee07e840b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- 172.21.15.98:0/2496695636 >> 172.21.15.98:0/2496695636 conn(0x55ee07acc3c0 msgr2=0x55ee07e839a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- 172.21.15.98:0/2496695636 shutdown_connections 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- 172.21.15.98:0/2496695636 wait complete. 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- start start 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 Processor -- start 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 0x55ee07e877a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e87ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 0x55ee07e8abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55ee07e92b80 con 0x55ee07ab7590 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55ee07e924b0 con 0x55ee07aaf3b0 2024-04-25T16:32:26.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55ee07e93250 con 0x55ee07ad8270 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f82543d6740 10 client.0 _ll_register_callbacks cb 0x7f823000d3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.591+0000 7f8246ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e87ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f8246ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e87ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53468/0 (socket says 172.21.15.98:53468) 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f8246ffd640 1 -- 172.21.15.98:0/1440295897 learned_addr learned my addr 172.21.15.98:0/1440295897 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f823ffff640 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 0x55ee07e8abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f8246ffd640 1 -- 172.21.15.98:0/1440295897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 msgr2=0x55ee07e8abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f8246ffd640 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 0x55ee07e8abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f8246ffd640 1 -- 172.21.15.98:0/1440295897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 msgr2=0x55ee07e877a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f823ffff640 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 0x55ee07e877a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f8246ffd640 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 0x55ee07e877a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f8246ffd640 1 -- 172.21.15.98:0/1440295897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee07e8b190 con 0x55ee07ab7590 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f8246ffd640 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e87ce0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f823802a450 tx=0x7f8238043670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2643138d651af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f822e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f822e7fc640 1 -- 172.21.15.98:0/1440295897 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823805f060 con 0x55ee07ab7590 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f822e7fc640 1 -- 172.21.15.98:0/1440295897 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f8238049040 con 0x55ee07ab7590 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f82543d6740 1 -- 172.21.15.98:0/1440295897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8230023070 con 0x55ee07ab7590 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f822e7fc640 1 -- 172.21.15.98:0/1440295897 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8238066050 con 0x55ee07ab7590 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f82543d6740 10 client.15522 fetch_fsmap learned FSMap version 28 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f82543d6740 1 -- 172.21.15.98:0/1440295897 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ee07efbb80 con 0x55ee07ab7590 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.592+0000 7f822e7fc640 1 -- 172.21.15.98:0/1440295897 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8230023070 con 0x55ee07ab7590 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f822e7fc640 1 -- 172.21.15.98:0/1440295897 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8238004f60 con 0x55ee07ab7590 2024-04-25T16:32:26.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f82543d6740 10 client.15522 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f82543d6740 10 client.15522 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f82543d6740 20 client.15522 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f82543d6740 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55ee07a6c280 0x55ee07eff390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f82543d6740 4 client.15522 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f82543d6740 1 -- 172.21.15.98:0/1440295897 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "status"}) -- 0x55ee07de61c0 con 0x55ee07a6c280 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f823ffff640 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55ee07a6c280 0x55ee07eff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f822e7fc640 10 client.15522 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.593+0000 7f823ffff640 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55ee07a6c280 0x55ee07eff390 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c78fbd583d967d9 server_cookie=211b325afca1c490 in_seq=0 out_seq=0 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f822e7fc640 1 -- 172.21.15.98:0/1440295897 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ee07de61c0 con 0x55ee07a6c280 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f822e7fc640 10 client.15522 handle_command_reply: tid=0 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 client.15522 shutdown 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 -- 172.21.15.98:0/1440295897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 msgr2=0x55ee07e87ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e87ce0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f823802a450 tx=0x7f8238043670 comp rx=0 tx=0).stop 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 -- 172.21.15.98:0/1440295897 shutdown_connections 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55ee07a6c280 0x55ee07eff390 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55ee07ad8270 0x55ee07e8abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55ee07ab7590 0x55ee07e87ce0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f822e7fc640 0 client.15522 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 --2- 172.21.15.98:0/1440295897 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55ee07aaf3b0 0x55ee07e877a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 -- 172.21.15.98:0/1440295897 >> 172.21.15.98:0/1440295897 conn(0x55ee07acc3c0 msgr2=0x55ee07e746b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 -- 172.21.15.98:0/1440295897 shutdown_connections 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 1 -- 172.21.15.98:0/1440295897 wait complete. 2024-04-25T16:32:26.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f82543d6740 20 client.15522 trim_cache size 0 max 16384 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "cluster_fsid": "c520eaa8-031f-11ef-bc93-c7b262605968", 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "whoami": 0, 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 24539, 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "want_state": "up:active", 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "state": "up:active", 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs", 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "rank_uptime": 136.25718362699999, 2024-04-25T16:32:26.602 INFO:teuthology.orchestra.run.smithi098.stdout: "mdsmap_epoch": 28, 2024-04-25T16:32:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch": 85, 2024-04-25T16:32:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_epoch_barrier": 81, 2024-04-25T16:32:26.603 INFO:teuthology.orchestra.run.smithi098.stdout: "uptime": 141.978423018 2024-04-25T16:32:26.603 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:26.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f824d527640 1 -- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f82340afaf0 msgr2=0x7f82340b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f824d527640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f82340afaf0 0x7f82340b2000 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f82400a7480 tx=0x7f82400ca040 comp rx=0 tx=0).stop 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f824d527640 1 -- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248108940 msgr2=0x7f82481a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.594+0000 7f824d527640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248108940 0x7f82481a05c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f824000eb10 tx=0x7f824000c2d0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.595+0000 7f824d527640 1 -- 172.21.15.98:0/4233735681 shutdown_connections 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.595+0000 7f824d527640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f82340afaf0 0x7f82340b2000 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.595+0000 7f824d527640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8248108940 0x7f82481a05c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.595+0000 7f824d527640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8248103c40 0x7f8248199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.595+0000 7f824d527640 1 --2- 172.21.15.98:0/4233735681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8248102a00 0x7f8248199140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.595+0000 7f824d527640 1 -- 172.21.15.98:0/4233735681 >> 172.21.15.98:0/4233735681 conn(0x7f82480fdbf0 msgr2=0x7f8248104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.595+0000 7f824d527640 1 -- 172.21.15.98:0/4233735681 shutdown_connections 2024-04-25T16:32:26.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:26.595+0000 7f824d527640 1 -- 172.21.15.98:0/4233735681 wait complete. 2024-04-25T16:32:26.605 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T16:32:26.616 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T16:32:26.616 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T16:32:26.616 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T16:32:26.616 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-25T16:32:26.666 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-25T16:32:26.666 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-04-25T16:32:26.666 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2024-04-25T16:32:26.666 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T16:32:26.667 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:26.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:26.667 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-25T16:32:26.682 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T16:32:26.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:26.683 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T16:32:26.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:26.746 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-04-25T16:32:26.844 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-25T16:32:26.844 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T16:32:27.082 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:32:28.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:28 smithi098 ceph-mon[29162]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:32:28.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:28 smithi203 ceph-mon[33115]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:32:28.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:28 smithi173 ceph-mon[32556]: pgmap v355: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:32:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1838724671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 msgr2=0x7fb7c41091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1838724671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 0x7fb7c41091f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80099c0 tx=0x7fb7b8009cd0 comp rx=0 tx=0).stop 2024-04-25T16:32:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1838724671 shutdown_connections 2024-04-25T16:32:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1838724671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7c4109730 0x7fb7c4111090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1838724671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c41044b0 0x7fb7c41048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:28.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1838724671 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 0x7fb7c41091f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1838724671 >> 172.21.15.98:0/1838724671 conn(0x7fb7c40ffd40 msgr2=0x7fb7c4102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1838724671 shutdown_connections 2024-04-25T16:32:28.768 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.767+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1838724671 wait complete. 2024-04-25T16:32:28.769 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.768+0000 7fb7c8cb4700 1 Processor -- start 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.768+0000 7fb7c8cb4700 1 -- start start 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c8cb4700 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7c41044b0 0x7fb7c41a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c8cb4700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 0x7fb7c41a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c8cb4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4109730 0x7fb7c419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c8cb4700 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c4114260 con 0x7fb7c4108de0 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c8cb4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c41140e0 con 0x7fb7c4109730 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c8cb4700 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c41143e0 con 0x7fb7c41044b0 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 0x7fb7c41a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 0x7fb7c41a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53496/0 (socket says 172.21.15.98:53496) 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 -- 172.21.15.98:0/1890723979 learned_addr learned my addr 172.21.15.98:0/1890723979 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c2d9d700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4109730 0x7fb7c419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c259c700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7c41044b0 0x7fb7c41a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:28.770 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 -- 172.21.15.98:0/1890723979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7c41044b0 msgr2=0x7fb7c41a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7c41044b0 0x7fb7c41a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 -- 172.21.15.98:0/1890723979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4109730 msgr2=0x7fb7c419aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4109730 0x7fb7c419aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 -- 172.21.15.98:0/1890723979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c419b130 con 0x7fb7c4108de0 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c2d9d700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4109730 0x7fb7c419aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7c1d9b700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 0x7fb7c41a0da0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800b580 tx=0x7fb7b800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7b37fe700 1 -- 172.21.15.98:0/1890723979 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8026070 con 0x7fb7c4108de0 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7b37fe700 1 -- 172.21.15.98:0/1890723979 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb7b8005030 con 0x7fb7c4108de0 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.769+0000 7fb7b37fe700 1 -- 172.21.15.98:0/1890723979 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80049f0 con 0x7fb7c4108de0 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.770+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c419b3c0 con 0x7fb7c4108de0 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.770+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c419b850 con 0x7fb7c4108de0 2024-04-25T16:32:28.771 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.770+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c4051190 con 0x7fb7c4108de0 2024-04-25T16:32:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.772+0000 7fb7b37fe700 1 -- 172.21.15.98:0/1890723979 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb7b8029070 con 0x7fb7c4108de0 2024-04-25T16:32:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.772+0000 7fb7b37fe700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb7ac0afb80 0x7fb7ac0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.773+0000 7fb7c259c700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb7ac0afb80 0x7fb7ac0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.773+0000 7fb7b37fe700 1 -- 172.21.15.98:0/1890723979 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fb7b80dfd90 con 0x7fb7c4108de0 2024-04-25T16:32:28.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.773+0000 7fb7c259c700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb7ac0afb80 0x7fb7ac0b2090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc006fd0 tx=0x7fb7bc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:28.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:28.776+0000 7fb7b37fe700 1 -- 172.21.15.98:0/1890723979 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb7b8030080 con 0x7fb7c4108de0 2024-04-25T16:32:29.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.022+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb7c402e510 con 0x7fb7c4108de0 2024-04-25T16:32:29.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.022+0000 7fb7b37fe700 1 -- 172.21.15.98:0/1890723979 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) ==== 53+0+88 (secure 0 0 0) 0x7fb7b8013070 con 0x7fb7c4108de0 2024-04-25T16:32:29.022 INFO:teuthology.orchestra.run.smithi098.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.024+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb7ac0afb80 msgr2=0x7fb7ac0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.024+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb7ac0afb80 0x7fb7ac0b2090 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc006fd0 tx=0x7fb7bc008040 comp rx=0 tx=0).stop 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.024+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 msgr2=0x7fb7c41a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.024+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 0x7fb7c41a0da0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb7b800b580 tx=0x7fb7b800f510 comp rx=0 tx=0).stop 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.024+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 shutdown_connections 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.024+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb7ac0afb80 0x7fb7ac0b2090 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.025+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb7c41044b0 0x7fb7c41a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.025+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7c4109730 0x7fb7c419aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.025+0000 7fb7c8cb4700 1 --2- 172.21.15.98:0/1890723979 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb7c4108de0 0x7fb7c41a0da0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.025+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 >> 172.21.15.98:0/1890723979 conn(0x7fb7c40ffd40 msgr2=0x7fb7c410a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:29.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.025+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 shutdown_connections 2024-04-25T16:32:29.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29.025+0000 7fb7c8cb4700 1 -- 172.21.15.98:0/1890723979 wait complete. 2024-04-25T16:32:29.394 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:29 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1890723979' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:32:29.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:29 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1890723979' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:32:29.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:29 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1890723979' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:32:29.722 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T16:32:29.722 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T16:32:29.723 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi098.front.sepia.ceph.com 2024-04-25T16:32:29.723 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T16:32:29.723 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:29.723 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T16:32:29.723 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T16:32:29.723 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T16:32:29.723 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T16:32:29.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:29.723 DEBUG:teuthology.orchestra.run.smithi098:> ip addr 2024-04-25T16:32:29.739 INFO:teuthology.orchestra.run.smithi098.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T16:32:29.739 INFO:teuthology.orchestra.run.smithi098.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 ::1/128 scope host 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b8 brd ff:ff:ff:ff:ff:ff 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f0 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:d8:d9:b9 brd ff:ff:ff:ff:ff:ff 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: altname enp7s0f1 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6a brd ff:ff:ff:ff:ff:ff 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T16:32:29.740 INFO:teuthology.orchestra.run.smithi098.stdout: link/ether 0c:c4:7a:8f:d2:6b brd ff:ff:ff:ff:ff:ff 2024-04-25T16:32:29.741 INFO:teuthology.orchestra.run.smithi098.stdout: inet 172.21.15.98/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T16:32:29.741 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft 42107sec preferred_lft 42107sec 2024-04-25T16:32:29.741 INFO:teuthology.orchestra.run.smithi098.stdout: inet6 fe80::f29f:9dff:ddbc:4615/64 scope link noprefixroute 2024-04-25T16:32:29.741 INFO:teuthology.orchestra.run.smithi098.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:32:29.741 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T16:32:29.741 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:32:29.741 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:32:29.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add name ceph-brx type bridge 2024-04-25T16:32:29.741 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr flush dev ceph-brx 2024-04-25T16:32:29.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set ceph-brx up 2024-04-25T16:32:29.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T16:32:29.742 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:32:29.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:32:29.871 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:32:29.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:29.877 DEBUG:teuthology.orchestra.run.smithi098:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T16:32:29.899 INFO:teuthology.orchestra.run.smithi098.stdout:1 2024-04-25T16:32:29.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:29.901 DEBUG:teuthology.orchestra.run.smithi098:> route 2024-04-25T16:32:29.957 INFO:teuthology.orchestra.run.smithi098.stdout:Kernel IP routing table 2024-04-25T16:32:29.957 INFO:teuthology.orchestra.run.smithi098.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T16:32:29.957 INFO:teuthology.orchestra.run.smithi098.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T16:32:29.957 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T16:32:29.957 INFO:teuthology.orchestra.run.smithi098.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T16:32:29.957 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:32:29.958 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:32:29.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T16:32:29.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T16:32:29.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T16:32:29.958 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:32:30.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:32:30.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:32:30.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:30.083 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-25T16:32:30.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:30.098 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list-id 2024-04-25T16:32:30.153 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:32:30.153 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:32:30.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:32:30.153 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T16:32:30.153 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:32:30.241 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:32:30.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:30 smithi098 ceph-mon[29162]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:32:30.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:32:30.269 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T16:32:30.269 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:32:30.269 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:32:30.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T16:32:30.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T16:32:30.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T16:32:30.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T16:32:30.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T16:32:30.269 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:32:30.317 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:32:30.371 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:32:30.378 DEBUG:teuthology.orchestra.run.smithi098:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:32:30.378 DEBUG:teuthology.orchestra.run.smithi098:> set -e 2024-04-25T16:32:30.378 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set brx.0 up 2024-04-25T16:32:30.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T16:32:30.379 DEBUG:teuthology.orchestra.run.smithi098:> ') 2024-04-25T16:32:30.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:32:30.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:30 smithi203 ceph-mon[33115]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:32:30.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:32:30.456 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2024-04-25T16:32:30.456 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T16:32:30.456 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:30.470 INFO:teuthology.orchestra.run.smithi098.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T16:32:30.470 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T16:32:30.470 DEBUG:teuthology.orchestra.run.smithi098:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:30.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:30 smithi173 ceph-mon[32556]: pgmap v356: 129 pgs: 129 active+clean; 646 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:32:30.524 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-25T16:32:30.589 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T16:32:30.644 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-04-25T16:32:30.645 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-25T16:32:30.646 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/260d120b743fdefc47097bdfa51ca2592ce504e600e0b8aa9a7681e4cc8f09a3/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c3e265ddc44ba607adedfa4bb7c4a1d5a16bf204960d9b3e7f03b229a2fd58e8/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b11a75d862ab4ca91fa24f775f6dfcfe5f4198c0807f0f4909bf047ffb1350b9/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/39738d8aa159ba108f0d78e1529bb35664af715ac389a39fa6d060bcf363cfa4/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b8b9dc7ee394ce0927571fb358b669ca734319cbfaf2342876758534c9e46e2d/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3c1452187a1df91e01df6afd698ac4a4f426c127b144f2bce196417fec79813a/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c0fbdfa79403832c61b1e9461210414679dd82f9527d126e63606fdb9abe4097/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/7107c4da4556b4a7b2a6e9113780c8e9f29261d548251acae86d609e859597e3/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/12e8d5140c0933c0da3677f0f5235e29dcbea42d6a1e700d2d28d98996739286/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/8f45f93f747246be0f168199353f8c0eb210046bac0c70867ae9f250862cd38b/merged 2024-04-25T16:32:30.647 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-25T16:32:30.648 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:32:30.648 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:32:30.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:30.648 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-25T16:32:30.700 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T16:32:30.700 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/qa/sv_0/a4566d7d-75ee-46c1-9e38-d177d26b4a22 --client_fs=cephfs) 2024-04-25T16:32:30.742 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-25T16:32:30.765 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T16:32:30.822 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-25T16:32:30.822 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-25T16:32:30.822 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-25T16:32:30.823 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-25T16:32:30.824 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/260d120b743fdefc47097bdfa51ca2592ce504e600e0b8aa9a7681e4cc8f09a3/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c3e265ddc44ba607adedfa4bb7c4a1d5a16bf204960d9b3e7f03b229a2fd58e8/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b11a75d862ab4ca91fa24f775f6dfcfe5f4198c0807f0f4909bf047ffb1350b9/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/39738d8aa159ba108f0d78e1529bb35664af715ac389a39fa6d060bcf363cfa4/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b8b9dc7ee394ce0927571fb358b669ca734319cbfaf2342876758534c9e46e2d/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3c1452187a1df91e01df6afd698ac4a4f426c127b144f2bce196417fec79813a/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c0fbdfa79403832c61b1e9461210414679dd82f9527d126e63606fdb9abe4097/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/7107c4da4556b4a7b2a6e9113780c8e9f29261d548251acae86d609e859597e3/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/12e8d5140c0933c0da3677f0f5235e29dcbea42d6a1e700d2d28d98996739286/merged 2024-04-25T16:32:30.825 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/8f45f93f747246be0f168199353f8c0eb210046bac0c70867ae9f250862cd38b/merged 2024-04-25T16:32:30.826 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-25T16:32:30.826 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:32:30.826 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:32:30.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:30.826 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-25T16:32:30.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:2024-04-25T16:32:30.831+0000 7f2f1c509480 -1 init, newargv = 0x7f2f0c056f20 newargc=15 2024-04-25T16:32:30.831 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[83058]: starting ceph client 2024-04-25T16:32:30.854 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[83058]: starting fuse 2024-04-25T16:32:31.947 DEBUG:teuthology.orchestra.run.smithi098:> sudo modprobe fuse 2024-04-25T16:32:31.973 DEBUG:teuthology.orchestra.run.smithi098:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/proc 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/sys 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/dev 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/security 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/shm 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/pts 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/run 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/cgroup 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/pstore 2024-04-25T16:32:32.031 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/bpf 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/ 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/selinux 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/mqueue 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/debug 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/dev/hugepages 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/tracing 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/kernel/config 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/sys/fs/fuse/connections 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T16:32:32.032 INFO:teuthology.orchestra.run.smithi098.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/1000 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/ceph 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/260d120b743fdefc47097bdfa51ca2592ce504e600e0b8aa9a7681e4cc8f09a3/merged 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c3e265ddc44ba607adedfa4bb7c4a1d5a16bf204960d9b3e7f03b229a2fd58e8/merged 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/run/user/0 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b11a75d862ab4ca91fa24f775f6dfcfe5f4198c0807f0f4909bf047ffb1350b9/merged 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/39738d8aa159ba108f0d78e1529bb35664af715ac389a39fa6d060bcf363cfa4/merged 2024-04-25T16:32:32.033 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/b8b9dc7ee394ce0927571fb358b669ca734319cbfaf2342876758534c9e46e2d/merged 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/3c1452187a1df91e01df6afd698ac4a4f426c127b144f2bce196417fec79813a/merged 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/c0fbdfa79403832c61b1e9461210414679dd82f9527d126e63606fdb9abe4097/merged 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/7107c4da4556b4a7b2a6e9113780c8e9f29261d548251acae86d609e859597e3/merged 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/12e8d5140c0933c0da3677f0f5235e29dcbea42d6a1e700d2d28d98996739286/merged 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/var/lib/containers/storage/overlay/8f45f93f747246be0f168199353f8c0eb210046bac0c70867ae9f250862cd38b/merged 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run.smithi098.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:32.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:32.035 DEBUG:teuthology.orchestra.run.smithi098:> ls /sys/fs/fuse/connections 2024-04-25T16:32:32.085 INFO:teuthology.orchestra.run.smithi098.stdout:115 2024-04-25T16:32:32.086 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [115] 2024-04-25T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi098:> sudo stdin-killer -- python3 -c ' 2024-04-25T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi098:> import glob 2024-04-25T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi098:> import re 2024-04-25T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi098:> import os 2024-04-25T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi098:> import subprocess 2024-04-25T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-25T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi098:> def _find_admin_socket(client_name): 2024-04-25T16:32:32.086 DEBUG:teuthology.orchestra.run.smithi098:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> files = glob.glob(asok_path) 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> # Given a non-glob path, it better be there 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> if "*" not in asok_path: 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> assert(len(files) == 1) 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> return files[0] 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> for f in files: 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> contents = proc_f.read() 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> if mountpoint in contents: 2024-04-25T16:32:32.087 DEBUG:teuthology.orchestra.run.smithi098:> return f 2024-04-25T16:32:32.088 DEBUG:teuthology.orchestra.run.smithi098:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T16:32:32.088 DEBUG:teuthology.orchestra.run.smithi098:> 2024-04-25T16:32:32.088 DEBUG:teuthology.orchestra.run.smithi098:> print(_find_admin_socket("client.0")) 2024-04-25T16:32:32.088 DEBUG:teuthology.orchestra.run.smithi098:> ' 2024-04-25T16:32:32.131 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:32 smithi098 ceph-mon[29162]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:32:32.205 INFO:teuthology.orchestra.run.smithi098.stdout:/var/run/ceph/ceph-client.0.83058.asok 2024-04-25T16:32:32.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:32:32.216 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.83058.asok 2024-04-25T16:32:32.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:32.216 DEBUG:teuthology.orchestra.run.smithi098:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.83058.asok status 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "metadata": { 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "entity_id": "0", 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": "smithi098", 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "pid": "83058", 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "root": "/volumes/qa/sv_0/a4566d7d-75ee-46c1-9e38-d177d26b4a22" 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_count": 0, 2024-04-25T16:32:32.361 INFO:teuthology.orchestra.run.smithi098.stdout: "dentry_pinned_count": 0, 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "id": 15534, 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "inst": { 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": { 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "client", 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "num": 15534 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4080322749 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:32:32.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": { 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "192.168.0.1:0", 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4080322749 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "inst_str": "client.15534 192.168.0.1:0/4080322749", 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "addr_str": "192.168.0.1:0/4080322749", 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "inode_count": 5, 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_epoch": 28, 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch": 85, 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_epoch_barrier": 0, 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "blocklisted": false, 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout: "fs_name": "cephfs" 2024-04-25T16:32:32.363 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:32.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:32.370 DEBUG:teuthology.orchestra.run.smithi098:> stat --file-system '--printf=%T 2024-04-25T16:32:32.370 DEBUG:teuthology.orchestra.run.smithi098:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:32.386 INFO:teuthology.orchestra.run.smithi098.stdout:fuseblk 2024-04-25T16:32:32.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:32.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:32.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:32.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:32 smithi203 ceph-mon[33115]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:32:32.452 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T16:32:32.462 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T16:32:32.462 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T16:32:32.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-25T16:32:32.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:32 smithi173 ceph-mon[32556]: pgmap v357: 129 pgs: 129 active+clean; 647 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 -- 172.21.15.98:0/3038394448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 msgr2=0x7f3c6c10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3038394448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 0x7f3c6c10ac00 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009ca0 tx=0x7f3c6402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 -- 172.21.15.98:0/3038394448 shutdown_connections 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3038394448 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 0x7f3c6c10ac00 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3038394448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c6c103990 0x7f3c6c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3038394448 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c6c102fb0 0x7f3c6c1033c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 -- 172.21.15.98:0/3038394448 >> 172.21.15.98:0/3038394448 conn(0x7f3c6c0fd390 msgr2=0x7f3c6c0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 -- 172.21.15.98:0/3038394448 shutdown_connections 2024-04-25T16:32:32.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.605+0000 7f3c70e51640 1 -- 172.21.15.98:0/3038394448 wait complete. 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c70e51640 1 Processor -- start 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c70e51640 1 -- start start 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c70e51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c6c102fb0 0x7f3c6c1022b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c70e51640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c6c103990 0x7f3c6c19ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c70e51640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 0x7f3c6c1a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c70e51640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10e460 con 0x7f3c6c1044b0 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c70e51640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10e2e0 con 0x7f3c6c103990 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c70e51640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c10e5e0 con 0x7f3c6c102fb0 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c6ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 0x7f3c6c1a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c6ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 0x7f3c6c1a13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53512/0 (socket says 172.21.15.98:53512) 2024-04-25T16:32:32.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c6ad9d640 1 -- 172.21.15.98:0/3323022386 learned_addr learned my addr 172.21.15.98:0/3323022386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c69d9b640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c6c103990 0x7f3c6c19ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c6ad9d640 1 -- 172.21.15.98:0/3323022386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c6c102fb0 msgr2=0x7f3c6c1022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.606+0000 7f3c6a59c640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c6c102fb0 0x7f3c6c1022b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c6ad9d640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c6c102fb0 0x7f3c6c1022b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c6ad9d640 1 -- 172.21.15.98:0/3323022386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c6c103990 msgr2=0x7f3c6c19ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c6ad9d640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c6c103990 0x7f3c6c19ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c6ad9d640 1 -- 172.21.15.98:0/3323022386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c1a1bf0 con 0x7f3c6c1044b0 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c6ad9d640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 0x7f3c6c1a13d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3c64031040 tx=0x7f3c64003970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c5b7fe640 1 -- 172.21.15.98:0/3323022386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c64003d60 con 0x7f3c6c1044b0 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6c1a1e80 con 0x7f3c6c1044b0 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c5b7fe640 1 -- 172.21.15.98:0/3323022386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c640044d0 con 0x7f3c6c1044b0 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c5b7fe640 1 -- 172.21.15.98:0/3323022386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c64030c50 con 0x7f3c6c1044b0 2024-04-25T16:32:32.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.607+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c6c1a22e0 con 0x7f3c6c1044b0 2024-04-25T16:32:32.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.608+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c6c050740 con 0x7f3c6c1044b0 2024-04-25T16:32:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.610+0000 7f3c5b7fe640 1 -- 172.21.15.98:0/3323022386 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3c64004040 con 0x7f3c6c1044b0 2024-04-25T16:32:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.611+0000 7f3c5b7fe640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3c540afae0 0x7f3c540b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.611+0000 7f3c5b7fe640 1 -- 172.21.15.98:0/3323022386 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f3c641005e0 con 0x7f3c6c1044b0 2024-04-25T16:32:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.611+0000 7f3c6a59c640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3c540afae0 0x7f3c540b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:32.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.611+0000 7f3c6a59c640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3c540afae0 0x7f3c540b1ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c06a100 tx=0x7f3c5c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:32.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.615+0000 7f3c5b7fe640 1 -- 172.21.15.98:0/3323022386 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c640c8e10 con 0x7f3c6c1044b0 2024-04-25T16:32:32.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:32.843+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f3c6c109960 con 0x7f3c6c1044b0 2024-04-25T16:32:33.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.066+0000 7f3c5b7fe640 1 -- 172.21.15.98:0/3323022386 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7f3c640c8710 con 0x7f3c6c1044b0 2024-04-25T16:32:33.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3c540afae0 msgr2=0x7f3c540b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3c540afae0 0x7f3c540b1ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c06a100 tx=0x7f3c5c005f90 comp rx=0 tx=0).stop 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 msgr2=0x7f3c6c1a13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 0x7f3c6c1a13d0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3c64031040 tx=0x7f3c64003970 comp rx=0 tx=0).stop 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 shutdown_connections 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3c540afae0 0x7f3c540b1ff0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3c6c1044b0 0x7f3c6c1a13d0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c6c103990 0x7f3c6c19ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 --2- 172.21.15.98:0/3323022386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c6c102fb0 0x7f3c6c1022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 >> 172.21.15.98:0/3323022386 conn(0x7f3c6c0fd390 msgr2=0x7f3c6c106b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 shutdown_connections 2024-04-25T16:32:33.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.069+0000 7f3c70e51640 1 -- 172.21.15.98:0/3323022386 wait complete. 2024-04-25T16:32:33.080 DEBUG:teuthology.orchestra.run.smithi098:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 -- 172.21.15.98:0/4040646741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514108970 msgr2=0x7f051410ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 --2- 172.21.15.98:0/4040646741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514108970 0x7f051410ad90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f050c009ca0 tx=0x7f050c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 -- 172.21.15.98:0/4040646741 shutdown_connections 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 --2- 172.21.15.98:0/4040646741 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514108970 0x7f051410ad90 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 --2- 172.21.15.98:0/4040646741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0514103c70 0x7f0514104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 --2- 172.21.15.98:0/4040646741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0514102a30 0x7f0514103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 -- 172.21.15.98:0/4040646741 >> 172.21.15.98:0/4040646741 conn(0x7f05140fdc20 msgr2=0x7f05141000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 -- 172.21.15.98:0/4040646741 shutdown_connections 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 -- 172.21.15.98:0/4040646741 wait complete. 2024-04-25T16:32:33.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.194+0000 7f051b196640 1 Processor -- start 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f051b196640 1 -- start start 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f051b196640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0514102a30 0x7f0514075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f051b196640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514103c70 0x7f051419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f051b196640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0514108970 0x7f05141a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f051b196640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f051410e460 con 0x7f0514103c70 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f051b196640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f051410e2e0 con 0x7f0514102a30 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f051b196640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f051410e5e0 con 0x7f0514108970 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514103c70 0x7f051419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514103c70 0x7f051419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53532/0 (socket says 172.21.15.98:53532) 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0518f32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0514102a30 0x7f0514075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 -- 172.21.15.98:0/2787342362 learned_addr learned my addr 172.21.15.98:0/2787342362 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0519733640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0514108970 0x7f05141a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 -- 172.21.15.98:0/2787342362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0514108970 msgr2=0x7f05141a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0514108970 0x7f05141a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 -- 172.21.15.98:0/2787342362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0514102a30 msgr2=0x7f0514075860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0514102a30 0x7f0514075860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 -- 172.21.15.98:0/2787342362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05141a1bf0 con 0x7f0514103c70 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0519733640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0514108970 0x7f05141a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0513fff640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514103c70 0x7f051419ee60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f050000c970 tx=0x7f050000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f0511ffb640 1 -- 172.21.15.98:0/2787342362 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500007bb0 con 0x7f0514103c70 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.195+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05141a1ee0 con 0x7f0514103c70 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.196+0000 7f0511ffb640 1 -- 172.21.15.98:0/2787342362 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0500007d50 con 0x7f0514103c70 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.196+0000 7f0511ffb640 1 -- 172.21.15.98:0/2787342362 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0500004e40 con 0x7f0514103c70 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.196+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05141a2340 con 0x7f0514103c70 2024-04-25T16:32:33.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.197+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0514107780 con 0x7f0514103c70 2024-04-25T16:32:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.198+0000 7f0511ffb640 1 -- 172.21.15.98:0/2787342362 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0500005020 con 0x7f0514103c70 2024-04-25T16:32:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.198+0000 7f0511ffb640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05040afae0 0x7f05040b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.199+0000 7f0518f32640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05040afae0 0x7f05040b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.199+0000 7f0511ffb640 1 -- 172.21.15.98:0/2787342362 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f05000ded30 con 0x7f0514103c70 2024-04-25T16:32:33.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.199+0000 7f0518f32640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05040afae0 0x7f05040b1ff0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0514103630 tx=0x7f0508005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:33.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.203+0000 7f0511ffb640 1 -- 172.21.15.98:0/2787342362 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0500014030 con 0x7f0514103c70 2024-04-25T16:32:33.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.406+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f0514063080 con 0x7f05040afae0 2024-04-25T16:32:33.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:33 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3323022386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T16:32:33.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.423+0000 7f0511ffb640 1 -- 172.21.15.98:0/2787342362 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f0514063080 con 0x7f05040afae0 2024-04-25T16:32:33.422 INFO:teuthology.orchestra.run.smithi098.stdout:{} 2024-04-25T16:32:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05040afae0 msgr2=0x7f05040b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05040afae0 0x7f05040b1ff0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0514103630 tx=0x7f0508005f90 comp rx=0 tx=0).stop 2024-04-25T16:32:33.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514103c70 msgr2=0x7f051419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514103c70 0x7f051419ee60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f050000c970 tx=0x7f050000ce30 comp rx=0 tx=0).stop 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 shutdown_connections 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f05040afae0 0x7f05040b1ff0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0514108970 0x7f05141a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0514103c70 0x7f051419ee60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 --2- 172.21.15.98:0/2787342362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0514102a30 0x7f0514075860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 >> 172.21.15.98:0/2787342362 conn(0x7f05140fdc20 msgr2=0x7f0514104950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 shutdown_connections 2024-04-25T16:32:33.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.425+0000 7f051b196640 1 -- 172.21.15.98:0/2787342362 wait complete. 2024-04-25T16:32:33.435 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T16:32:33.447 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T16:32:33.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:32:33.482 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:33 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3323022386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T16:32:33.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:33 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3323022386' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-25T16:32:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.589+0000 7f7b956af640 1 -- 172.21.15.98:0/4141874114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 msgr2=0x7f7b90109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.589+0000 7f7b956af640 1 --2- 172.21.15.98:0/4141874114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 0x7f7b90109350 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f7b84009900 tx=0x7f7b8402a410 comp rx=0 tx=0).stop 2024-04-25T16:32:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 -- 172.21.15.98:0/4141874114 shutdown_connections 2024-04-25T16:32:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 --2- 172.21.15.98:0/4141874114 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b90109890 0x7f7b9010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 --2- 172.21.15.98:0/4141874114 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 0x7f7b90109350 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 --2- 172.21.15.98:0/4141874114 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b901045d0 0x7f7b901069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 -- 172.21.15.98:0/4141874114 >> 172.21.15.98:0/4141874114 conn(0x7f7b900fdbf0 msgr2=0x7f7b90100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:33.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 -- 172.21.15.98:0/4141874114 shutdown_connections 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 -- 172.21.15.98:0/4141874114 wait complete. 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 Processor -- start 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 -- start start 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.590+0000 7f7b956af640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b901045d0 0x7f7b9019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b956af640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 0x7f7b9019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b956af640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b90109890 0x7f7b901a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b956af640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9010f580 con 0x7f7b90106f30 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b956af640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9010f400 con 0x7f7b901045d0 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b956af640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9010f700 con 0x7f7b90109890 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 0x7f7b9019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 0x7f7b9019a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53564/0 (socket says 172.21.15.98:53564) 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 -- 172.21.15.98:0/2814390340 learned_addr learned my addr 172.21.15.98:0/2814390340 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8effd640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b901045d0 0x7f7b9019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 -- 172.21.15.98:0/2814390340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b90109890 msgr2=0x7f7b901a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b90109890 0x7f7b901a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 -- 172.21.15.98:0/2814390340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b901045d0 msgr2=0x7f7b9019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b901045d0 0x7f7b9019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 -- 172.21.15.98:0/2814390340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b901a1d50 con 0x7f7b90106f30 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b8e7fc640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 0x7f7b9019a750 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7b84031460 tx=0x7f7b84003c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b77fff640 1 -- 172.21.15.98:0/2814390340 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8403f070 con 0x7f7b90106f30 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b77fff640 1 -- 172.21.15.98:0/2814390340 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7b84031da0 con 0x7f7b90106f30 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b901a1fe0 con 0x7f7b90106f30 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b77fff640 1 -- 172.21.15.98:0/2814390340 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b84035030 con 0x7f7b90106f30 2024-04-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.591+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b901a24c0 con 0x7f7b90106f30 2024-04-25T16:32:33.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.593+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b900ff710 con 0x7f7b90106f30 2024-04-25T16:32:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.594+0000 7f7b77fff640 1 -- 172.21.15.98:0/2814390340 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7b84042070 con 0x7f7b90106f30 2024-04-25T16:32:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.594+0000 7f7b77fff640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7b780afaf0 0x7f7b780b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.594+0000 7f7b8effd640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7b780afaf0 0x7f7b780b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.594+0000 7f7b77fff640 1 -- 172.21.15.98:0/2814390340 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7b840ff850 con 0x7f7b90106f30 2024-04-25T16:32:33.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.595+0000 7f7b8effd640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7b780afaf0 0x7f7b780b2000 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7b900757f0 tx=0x7f7b80009680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:33.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.599+0000 7f7b77fff640 1 -- 172.21.15.98:0/2814390340 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b840c8080 con 0x7f7b90106f30 2024-04-25T16:32:33.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:32:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:32:33.411+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:32:33.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:32:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:32:33.411+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:32:33.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:32:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:32:33.411+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:32:33.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:32:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:32:33.411+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:32:33.752 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:32:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x[29395]: 2024-04-25T16:32:33.411+0000 7fafd64e7700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T16:32:33.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.777+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7b9002dd70 con 0x7f7b90106f30 2024-04-25T16:32:33.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.779+0000 7f7b77fff640 1 -- 172.21.15.98:0/2814390340 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24806 (secure 0 0 0) 0x7f7b840c7980 con 0x7f7b90106f30 2024-04-25T16:32:33.778 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:33.779 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:32:10.010488+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:32:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7b780afaf0 msgr2=0x7f7b780b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7b780afaf0 0x7f7b780b2000 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7b900757f0 tx=0x7f7b80009680 comp rx=0 tx=0).stop 2024-04-25T16:32:33.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 msgr2=0x7f7b9019a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 0x7f7b9019a750 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f7b84031460 tx=0x7f7b84003c80 comp rx=0 tx=0).stop 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 shutdown_connections 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7b780afaf0 0x7f7b780b2000 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b90109890 0x7f7b901a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7b90106f30 0x7f7b9019a750 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 --2- 172.21.15.98:0/2814390340 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b901045d0 0x7f7b9019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 >> 172.21.15.98:0/2814390340 conn(0x7f7b900fdbf0 msgr2=0x7f7b90105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 shutdown_connections 2024-04-25T16:32:33.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.781+0000 7f7b956af640 1 -- 172.21.15.98:0/2814390340 wait complete. 2024-04-25T16:32:33.792 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:32:33.792 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 -- 172.21.15.98:0/2938267837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a741033d0 msgr2=0x7f9a741037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/2938267837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a741033d0 0x7f9a741037e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400aa10 tx=0x7f9a6402b720 comp rx=0 tx=0).stop 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 -- 172.21.15.98:0/2938267837 shutdown_connections 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/2938267837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a741080d0 0x7f9a7410a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/2938267837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a741033d0 0x7f9a741037e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/2938267837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a74102190 0x7f9a74102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 -- 172.21.15.98:0/2938267837 >> 172.21.15.98:0/2938267837 conn(0x7f9a740fd380 msgr2=0x7f9a740ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 -- 172.21.15.98:0/2938267837 shutdown_connections 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.904+0000 7f9a7c40c640 1 -- 172.21.15.98:0/2938267837 wait complete. 2024-04-25T16:32:33.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7c40c640 1 Processor -- start 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7c40c640 1 -- start start 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7c40c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a74102190 0x7f9a7419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7c40c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a741033d0 0x7f9a7419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7c40c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a741080d0 0x7f9a741a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7c40c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7410d2d0 con 0x7f9a741033d0 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7c40c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7410d150 con 0x7f9a74102190 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7c40c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7410d450 con 0x7f9a741080d0 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a799a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a741033d0 0x7f9a7419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a799a7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a741033d0 0x7f9a7419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53584/0 (socket says 172.21.15.98:53584) 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a799a7640 1 -- 172.21.15.98:0/869456884 learned_addr learned my addr 172.21.15.98:0/869456884 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.905+0000 7f9a7a1a8640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a74102190 0x7f9a7419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a7a9a9640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a741080d0 0x7f9a741a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a799a7640 1 -- 172.21.15.98:0/869456884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a741080d0 msgr2=0x7f9a741a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a799a7640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a741080d0 0x7f9a741a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a799a7640 1 -- 172.21.15.98:0/869456884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a74102190 msgr2=0x7f9a7419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a799a7640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a74102190 0x7f9a7419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a799a7640 1 -- 172.21.15.98:0/869456884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a741a1cd0 con 0x7f9a741033d0 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a799a7640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a741033d0 0x7f9a7419a740 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400a340 tx=0x7f9a640042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a6b7fe640 1 -- 172.21.15.98:0/869456884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6403f070 con 0x7f9a741033d0 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a741a1f60 con 0x7f9a741033d0 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a6b7fe640 1 -- 172.21.15.98:0/869456884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9a64030980 con 0x7f9a741033d0 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a6b7fe640 1 -- 172.21.15.98:0/869456884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64036040 con 0x7f9a741033d0 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.906+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a741a23c0 con 0x7f9a741033d0 2024-04-25T16:32:33.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.907+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a741a2620 con 0x7f9a741033d0 2024-04-25T16:32:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.908+0000 7f9a6b7fe640 1 -- 172.21.15.98:0/869456884 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9a64042070 con 0x7f9a741033d0 2024-04-25T16:32:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.909+0000 7f9a6b7fe640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a600afaf0 0x7f9a600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.909+0000 7f9a7a1a8640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a600afaf0 0x7f9a600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.909+0000 7f9a6b7fe640 1 -- 172.21.15.98:0/869456884 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f9a640ffb30 con 0x7f9a741033d0 2024-04-25T16:32:33.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.910+0000 7f9a7a1a8640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a600afaf0 0x7f9a600b2000 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a70005fd0 tx=0x7f9a7000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:33.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:33.913+0000 7f9a6b7fe640 1 -- 172.21.15.98:0/869456884 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a640c8360 con 0x7f9a741033d0 2024-04-25T16:32:34.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.088+0000 7f9a6b7fe640 1 -- 172.21.15.98:0/869456884 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f9a64032400 con 0x7f9a741033d0 2024-04-25T16:32:34.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.101+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9a741a27f0 con 0x7f9a741033d0 2024-04-25T16:32:34.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.101+0000 7f9a6b7fe640 1 -- 172.21.15.98:0/869456884 <== mon.0 v2:172.21.15.98:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f9a640f6590 con 0x7f9a741033d0 2024-04-25T16:32:34.101 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:32:34.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a600afaf0 msgr2=0x7f9a600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a600afaf0 0x7f9a600b2000 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9a70005fd0 tx=0x7f9a7000b410 comp rx=0 tx=0).stop 2024-04-25T16:32:34.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a741033d0 msgr2=0x7f9a7419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a741033d0 0x7f9a7419a740 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400a340 tx=0x7f9a640042f0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 shutdown_connections 2024-04-25T16:32:34.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9a600afaf0 0x7f9a600b2000 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9a741080d0 0x7f9a741a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9a741033d0 0x7f9a7419a740 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 --2- 172.21.15.98:0/869456884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9a74102190 0x7f9a7419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 >> 172.21.15.98:0/869456884 conn(0x7f9a740fd380 msgr2=0x7f9a741060c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:34.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 shutdown_connections 2024-04-25T16:32:34.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.103+0000 7f9a7c40c640 1 -- 172.21.15.98:0/869456884 wait complete. 2024-04-25T16:32:34.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:32:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 -- 172.21.15.98:0/468644035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738109790 msgr2=0x7ff73810bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 --2- 172.21.15.98:0/468644035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738109790 0x7ff73810bca0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7ff730009ca0 tx=0x7ff73002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 -- 172.21.15.98:0/468644035 shutdown_connections 2024-04-25T16:32:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 --2- 172.21.15.98:0/468644035 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738109790 0x7ff73810bca0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 --2- 172.21.15.98:0/468644035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff738106c20 0x7ff738109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 --2- 172.21.15.98:0/468644035 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff738101e20 0x7ff738102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 -- 172.21.15.98:0/468644035 >> 172.21.15.98:0/468644035 conn(0x7ff7380fd350 msgr2=0x7ff7380ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:34.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 -- 172.21.15.98:0/468644035 shutdown_connections 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.225+0000 7ff73f877640 1 -- 172.21.15.98:0/468644035 wait complete. 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73f877640 1 Processor -- start 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73f877640 1 -- start start 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73f877640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff738101e20 0x7ff73819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73f877640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738106c20 0x7ff73819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73f877640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff738109790 0x7ff7381a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73f877640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810f520 con 0x7ff738106c20 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73f877640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810f3a0 con 0x7ff738101e20 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73f877640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff73810f6a0 con 0x7ff738109790 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73ce12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738106c20 0x7ff73819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73ce12640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738106c20 0x7ff73819a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53602/0 (socket says 172.21.15.98:53602) 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73ce12640 1 -- 172.21.15.98:0/655734370 learned_addr learned my addr 172.21.15.98:0/655734370 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73d613640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff738101e20 0x7ff73819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73de14640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff738109790 0x7ff7381a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73ce12640 1 -- 172.21.15.98:0/655734370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff738109790 msgr2=0x7ff7381a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73ce12640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff738109790 0x7ff7381a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73ce12640 1 -- 172.21.15.98:0/655734370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff738101e20 msgr2=0x7ff73819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73ce12640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff738101e20 0x7ff73819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73ce12640 1 -- 172.21.15.98:0/655734370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7381a1c30 con 0x7ff738106c20 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.226+0000 7ff73de14640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff738109790 0x7ff7381a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.227+0000 7ff73ce12640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738106c20 0x7ff73819a630 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff72800e9f0 tx=0x7ff72800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.227+0000 7ff72e7fc640 1 -- 172.21.15.98:0/655734370 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72800cd80 con 0x7ff738106c20 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.227+0000 7ff72e7fc640 1 -- 172.21.15.98:0/655734370 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff7280108c0 con 0x7ff738106c20 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.227+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7381a1f20 con 0x7ff738106c20 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.227+0000 7ff72e7fc640 1 -- 172.21.15.98:0/655734370 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7280187c0 con 0x7ff738106c20 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.227+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7381a2390 con 0x7ff738106c20 2024-04-25T16:32:34.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.228+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7380fee70 con 0x7ff738106c20 2024-04-25T16:32:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.229+0000 7ff72e7fc640 1 -- 172.21.15.98:0/655734370 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7ff728010430 con 0x7ff738106c20 2024-04-25T16:32:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.230+0000 7ff72e7fc640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7240afb80 0x7ff7240b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.230+0000 7ff73d613640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7240afb80 0x7ff7240b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.230 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.230+0000 7ff72e7fc640 1 -- 172.21.15.98:0/655734370 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7ff7280dfa40 con 0x7ff738106c20 2024-04-25T16:32:34.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.230+0000 7ff73d613640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7240afb80 0x7ff7240b2090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff734005fd0 tx=0x7ff73400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:34.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.233+0000 7ff72e7fc640 1 -- 172.21.15.98:0/655734370 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7280a8110 con 0x7ff738106c20 2024-04-25T16:32:34.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.413+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff73802dce0 con 0x7ff738106c20 2024-04-25T16:32:34.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.414+0000 7ff72e7fc640 1 -- 172.21.15.98:0/655734370 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7ff7280a7a10 con 0x7ff738106c20 2024-04-25T16:32:34.413 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:34.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.415+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7240afb80 msgr2=0x7ff7240b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7240afb80 0x7ff7240b2090 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff734005fd0 tx=0x7ff73400b390 comp rx=0 tx=0).stop 2024-04-25T16:32:34.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738106c20 msgr2=0x7ff73819a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738106c20 0x7ff73819a630 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7ff72800e9f0 tx=0x7ff72800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 shutdown_connections 2024-04-25T16:32:34.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff7240afb80 0x7ff7240b2090 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff738109790 0x7ff7381a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff738106c20 0x7ff73819a630 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 --2- 172.21.15.98:0/655734370 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff738101e20 0x7ff73819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 >> 172.21.15.98:0/655734370 conn(0x7ff7380fd350 msgr2=0x7ff738107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:34.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 shutdown_connections 2024-04-25T16:32:34.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.416+0000 7ff73f877640 1 -- 172.21.15.98:0/655734370 wait complete. 2024-04-25T16:32:34.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:34 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3323022386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T16:32:34.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:34 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:32:34.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:34 smithi098 ceph-mon[29162]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:34.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:34 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2814390340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:34.426 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:34 smithi098 ceph-mon[29162]: pgmap v358: 129 pgs: 129 active+clean; 649 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 11 op/s 2024-04-25T16:32:34.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:32:34.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:34 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3323022386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T16:32:34.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:34 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:32:34.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:34 smithi203 ceph-mon[33115]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:34.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:34 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2814390340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:34.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:34 smithi203 ceph-mon[33115]: pgmap v358: 129 pgs: 129 active+clean; 649 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 11 op/s 2024-04-25T16:32:34.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:34 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3323022386' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-25T16:32:34.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:34 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:32:34.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:34 smithi173 ceph-mon[32556]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:32:34.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:34 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2814390340' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:34.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:34 smithi173 ceph-mon[32556]: pgmap v358: 129 pgs: 129 active+clean; 649 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 11 op/s 2024-04-25T16:32:34.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 -- 172.21.15.98:0/776833917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 msgr2=0x7f1f701036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 --2- 172.21.15.98:0/776833917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 0x7f1f701036f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009900 tx=0x7f1f6002a410 comp rx=0 tx=0).stop 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 -- 172.21.15.98:0/776833917 shutdown_connections 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 --2- 172.21.15.98:0/776833917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70108930 0x7f1f7010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 --2- 172.21.15.98:0/776833917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f70103c30 0x7f1f70104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 --2- 172.21.15.98:0/776833917 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 0x7f1f701036f0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 -- 172.21.15.98:0/776833917 >> 172.21.15.98:0/776833917 conn(0x7f1f700fdbe0 msgr2=0x7f1f70100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 -- 172.21.15.98:0/776833917 shutdown_connections 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.538+0000 7f1f75496640 1 -- 172.21.15.98:0/776833917 wait complete. 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f75496640 1 Processor -- start 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f75496640 1 -- start start 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f75496640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 0x7f1f7006bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f75496640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70103c30 0x7f1f7019ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f75496640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f70108930 0x7f1f701a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f75496640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7010e380 con 0x7f1f701029f0 2024-04-25T16:32:34.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f75496640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7010e200 con 0x7f1f70103c30 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f75496640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7010e500 con 0x7f1f70108930 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f6effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 0x7f1f7006bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f6effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 0x7f1f7006bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53614/0 (socket says 172.21.15.98:53614) 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f6effd640 1 -- 172.21.15.98:0/2843151341 learned_addr learned my addr 172.21.15.98:0/2843151341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f67fff640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70103c30 0x7f1f7019ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f6effd640 1 -- 172.21.15.98:0/2843151341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f70108930 msgr2=0x7f1f701a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f6effd640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f70108930 0x7f1f701a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f6effd640 1 -- 172.21.15.98:0/2843151341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70103c30 msgr2=0x7f1f7019ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f6effd640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70103c30 0x7f1f7019ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.539+0000 7f1f6effd640 1 -- 172.21.15.98:0/2843151341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f701a1c70 con 0x7f1f701029f0 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.540+0000 7f1f6effd640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 0x7f1f7006bd80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009230 tx=0x7f1f600040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.540+0000 7f1f6cff9640 1 -- 172.21.15.98:0/2843151341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f6003f030 con 0x7f1f701029f0 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.540+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f701a1f00 con 0x7f1f701029f0 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.540+0000 7f1f6cff9640 1 -- 172.21.15.98:0/2843151341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1f600044c0 con 0x7f1f701029f0 2024-04-25T16:32:34.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.540+0000 7f1f6cff9640 1 -- 172.21.15.98:0/2843151341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f60035070 con 0x7f1f701029f0 2024-04-25T16:32:34.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.540+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f701a2310 con 0x7f1f701029f0 2024-04-25T16:32:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.541+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f70107740 con 0x7f1f701029f0 2024-04-25T16:32:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.542+0000 7f1f6cff9640 1 -- 172.21.15.98:0/2843151341 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f1f60042030 con 0x7f1f701029f0 2024-04-25T16:32:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.543+0000 7f1f6cff9640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1f5c0afb80 0x7f1f5c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.543+0000 7f1f67fff640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1f5c0afb80 0x7f1f5c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.543+0000 7f1f6cff9640 1 -- 172.21.15.98:0/2843151341 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f1f600ff6d0 con 0x7f1f701029f0 2024-04-25T16:32:34.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.543+0000 7f1f67fff640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1f5c0afb80 0x7f1f5c0b2090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1f58005e10 tx=0x7f1f58005da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:34.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.547+0000 7f1f6cff9640 1 -- 172.21.15.98:0/2843151341 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f600c7da0 con 0x7f1f701029f0 2024-04-25T16:32:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.725+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1f7002de00 con 0x7f1f701029f0 2024-04-25T16:32:34.725 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.726+0000 7f1f6cff9640 1 -- 172.21.15.98:0/2843151341 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f1f600c76a0 con 0x7f1f701029f0 2024-04-25T16:32:34.725 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:32:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.727+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1f5c0afb80 msgr2=0x7f1f5c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.727+0000 7f1f75496640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1f5c0afb80 0x7f1f5c0b2090 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1f58005e10 tx=0x7f1f58005da0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.727+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 msgr2=0x7f1f7006bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.727 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.727+0000 7f1f75496640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 0x7f1f7006bd80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1f60009230 tx=0x7f1f600040c0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.728+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 shutdown_connections 2024-04-25T16:32:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.728+0000 7f1f75496640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1f5c0afb80 0x7f1f5c0b2090 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.728+0000 7f1f75496640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f70108930 0x7f1f701a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.728+0000 7f1f75496640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f70103c30 0x7f1f7019ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.728+0000 7f1f75496640 1 --2- 172.21.15.98:0/2843151341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1f701029f0 0x7f1f7006bd80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.728+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 >> 172.21.15.98:0/2843151341 conn(0x7f1f700fdbe0 msgr2=0x7f1f70104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.728+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 shutdown_connections 2024-04-25T16:32:34.728 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.728+0000 7f1f75496640 1 -- 172.21.15.98:0/2843151341 wait complete. 2024-04-25T16:32:34.738 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 -- 172.21.15.98:0/2581555309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c103c20 msgr2=0x7f353c104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 --2- 172.21.15.98:0/2581555309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c103c20 0x7f353c104030 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f352c009900 tx=0x7f352c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 -- 172.21.15.98:0/2581555309 shutdown_connections 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 --2- 172.21.15.98:0/2581555309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f353c108920 0x7f353c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 --2- 172.21.15.98:0/2581555309 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c103c20 0x7f353c104030 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 --2- 172.21.15.98:0/2581555309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f353c1029e0 0x7f353c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 -- 172.21.15.98:0/2581555309 >> 172.21.15.98:0/2581555309 conn(0x7f353c0fdbd0 msgr2=0x7f353c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 -- 172.21.15.98:0/2581555309 shutdown_connections 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 -- 172.21.15.98:0/2581555309 wait complete. 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 Processor -- start 2024-04-25T16:32:34.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.850+0000 7f354271f640 1 -- start start 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f354271f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f353c1029e0 0x7f353c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f354271f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f353c103c20 0x7f353c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f354271f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c108920 0x7f353c1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f354271f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f353c10f4c0 con 0x7f353c108920 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f354271f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f353c10f340 con 0x7f353c1029e0 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f354271f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f353c10f640 con 0x7f353c103c20 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c108920 0x7f353c1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c108920 0x7f353c1a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53628/0 (socket says 172.21.15.98:53628) 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 -- 172.21.15.98:0/1095287354 learned_addr learned my addr 172.21.15.98:0/1095287354 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f353bfff640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f353c1029e0 0x7f353c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 -- 172.21.15.98:0/1095287354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f353c103c20 msgr2=0x7f353c19a720 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f353c103c20 0x7f353c19a720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 -- 172.21.15.98:0/1095287354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f353c1029e0 msgr2=0x7f353c19a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f353c1029e0 0x7f353c19a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 -- 172.21.15.98:0/1095287354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353c1a1c70 con 0x7f353c108920 2024-04-25T16:32:34.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.851+0000 7f3540cbc640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c108920 0x7f353c1a16b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f353400cce0 tx=0x7f353400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.852+0000 7f35397fa640 1 -- 172.21.15.98:0/1095287354 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353400f450 con 0x7f353c108920 2024-04-25T16:32:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.852+0000 7f35397fa640 1 -- 172.21.15.98:0/1095287354 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f353400fa80 con 0x7f353c108920 2024-04-25T16:32:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.852+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353c1a1f60 con 0x7f353c108920 2024-04-25T16:32:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.852+0000 7f35397fa640 1 -- 172.21.15.98:0/1095287354 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534018ad0 con 0x7f353c108920 2024-04-25T16:32:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.852+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f353c1a23d0 con 0x7f353c108920 2024-04-25T16:32:34.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.853+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f353c107730 con 0x7f353c108920 2024-04-25T16:32:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.854+0000 7f35397fa640 1 -- 172.21.15.98:0/1095287354 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f353400f5f0 con 0x7f353c108920 2024-04-25T16:32:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.855+0000 7f35397fa640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35280afb80 0x7f35280b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.855+0000 7f353bfff640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35280afb80 0x7f35280b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.855+0000 7f35397fa640 1 -- 172.21.15.98:0/1095287354 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f35340dfa10 con 0x7f353c108920 2024-04-25T16:32:34.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.855+0000 7f353bfff640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35280afb80 0x7f35280b2090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3524005fd0 tx=0x7f352400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:34.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:34.858+0000 7f35397fa640 1 -- 172.21.15.98:0/1095287354 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3534014030 con 0x7f353c108920 2024-04-25T16:32:35.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.035+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f353c02dce0 con 0x7f353c108920 2024-04-25T16:32:35.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.036+0000 7f35397fa640 1 -- 172.21.15.98:0/1095287354 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f35340a8170 con 0x7f353c108920 2024-04-25T16:32:35.035 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35280afb80 msgr2=0x7f35280b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.037 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35280afb80 0x7f35280b2090 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3524005fd0 tx=0x7f352400b390 comp rx=0 tx=0).stop 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c108920 msgr2=0x7f353c1a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c108920 0x7f353c1a16b0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f353400cce0 tx=0x7f353400a620 comp rx=0 tx=0).stop 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 shutdown_connections 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35280afb80 0x7f35280b2090 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f353c108920 0x7f353c1a16b0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f353c103c20 0x7f353c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 --2- 172.21.15.98:0/1095287354 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f353c1029e0 0x7f353c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 >> 172.21.15.98:0/1095287354 conn(0x7f353c0fdbd0 msgr2=0x7f353c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 shutdown_connections 2024-04-25T16:32:35.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.038+0000 7f354271f640 1 -- 172.21.15.98:0/1095287354 wait complete. 2024-04-25T16:32:35.048 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:32:35.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.157+0000 7f2e897eb640 1 -- 172.21.15.98:0/1293942997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84109a20 msgr2=0x7f2e8410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.157+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1293942997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84109a20 0x7f2e8410be40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009900 tx=0x7f2e7c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.157+0000 7f2e897eb640 1 -- 172.21.15.98:0/1293942997 shutdown_connections 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.157+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1293942997 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84109a20 0x7f2e8410be40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.157+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1293942997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e84104d20 0x7f2e84105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.157+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1293942997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e84103ae0 0x7f2e841047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.157+0000 7f2e897eb640 1 -- 172.21.15.98:0/1293942997 >> 172.21.15.98:0/1293942997 conn(0x7f2e840fed10 msgr2=0x7f2e84101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.157+0000 7f2e897eb640 1 -- 172.21.15.98:0/1293942997 shutdown_connections 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 -- 172.21.15.98:0/1293942997 wait complete. 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 Processor -- start 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 -- start start 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e84103ae0 0x7f2e8419a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84104d20 0x7f2e8419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e84109a20 0x7f2e841a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8410f4c0 con 0x7f2e84104d20 2024-04-25T16:32:35.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8410f340 con 0x7f2e84103ae0 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.158+0000 7f2e897eb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8410f640 con 0x7f2e84109a20 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84104d20 0x7f2e8419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84104d20 0x7f2e8419a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:53652/0 (socket says 172.21.15.98:53652) 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 -- 172.21.15.98:0/1187215710 learned_addr learned my addr 172.21.15.98:0/1187215710 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e837fe640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e84109a20 0x7f2e841a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 -- 172.21.15.98:0/1187215710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e84109a20 msgr2=0x7f2e841a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e82ffd640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e84103ae0 0x7f2e8419a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e84109a20 0x7f2e841a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 -- 172.21.15.98:0/1187215710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e84103ae0 msgr2=0x7f2e8419a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e84103ae0 0x7f2e8419a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 -- 172.21.15.98:0/1187215710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e841a1b50 con 0x7f2e84104d20 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e82ffd640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e84103ae0 0x7f2e8419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e827fc640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84104d20 0x7f2e8419a600 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c920 tx=0x7f2e7800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:35.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e6bfff640 1 -- 172.21.15.98:0/1187215710 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78007c70 con 0x7f2e84104d20 2024-04-25T16:32:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e6bfff640 1 -- 172.21.15.98:0/1187215710 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2e78007e10 con 0x7f2e84104d20 2024-04-25T16:32:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e841a1e40 con 0x7f2e84104d20 2024-04-25T16:32:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e6bfff640 1 -- 172.21.15.98:0/1187215710 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78018710 con 0x7f2e84104d20 2024-04-25T16:32:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.159+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e841a22b0 con 0x7f2e84104d20 2024-04-25T16:32:35.162 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.160+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e84108830 con 0x7f2e84104d20 2024-04-25T16:32:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.162+0000 7f2e6bfff640 1 -- 172.21.15.98:0/1187215710 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f2e7800f450 con 0x7f2e84104d20 2024-04-25T16:32:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.162+0000 7f2e6bfff640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2e6c0afb80 0x7f2e6c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.162+0000 7f2e82ffd640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2e6c0afb80 0x7f2e6c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.163+0000 7f2e6bfff640 1 -- 172.21.15.98:0/1187215710 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f2e780df6a0 con 0x7f2e84104d20 2024-04-25T16:32:35.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.163+0000 7f2e82ffd640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2e6c0afb80 0x7f2e6c0b2090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2e841046e0 tx=0x7f2e74009400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:35.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.166+0000 7f2e6bfff640 1 -- 172.21.15.98:0/1187215710 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2e78014030 con 0x7f2e84104d20 2024-04-25T16:32:35.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.346+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e8402dda0 con 0x7f2e84104d20 2024-04-25T16:32:35.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.348+0000 7f2e6bfff640 1 -- 172.21.15.98:0/1187215710 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24806 (secure 0 0 0) 0x7f2e780a7e00 con 0x7f2e84104d20 2024-04-25T16:32:35.347 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:35.348 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:32:10.010488+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:32:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2e6c0afb80 msgr2=0x7f2e6c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2e6c0afb80 0x7f2e6c0b2090 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2e841046e0 tx=0x7f2e74009400 comp rx=0 tx=0).stop 2024-04-25T16:32:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84104d20 msgr2=0x7f2e8419a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84104d20 0x7f2e8419a600 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f2e7800c920 tx=0x7f2e7800cde0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 shutdown_connections 2024-04-25T16:32:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2e6c0afb80 0x7f2e6c0b2090 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2e84109a20 0x7f2e841a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2e84104d20 0x7f2e8419a600 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 --2- 172.21.15.98:0/1187215710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e84103ae0 0x7f2e8419a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 >> 172.21.15.98:0/1187215710 conn(0x7f2e840fed10 msgr2=0x7f2e84105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 shutdown_connections 2024-04-25T16:32:35.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.350+0000 7f2e897eb640 1 -- 172.21.15.98:0/1187215710 wait complete. 2024-04-25T16:32:35.359 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:35 smithi098 ceph-mon[29162]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T16:32:35.359 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:35 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/869456884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.359 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:35 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/655734370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.360 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:35 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2843151341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.360 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:35 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1095287354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.361 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:32:35.361 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:32:35.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:35 smithi203 ceph-mon[33115]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T16:32:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:35 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/869456884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:35 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/655734370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:35 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2843151341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:35 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1095287354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 -- 172.21.15.98:0/3022371100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 msgr2=0x7fc95c10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 --2- 172.21.15.98:0/3022371100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 0x7fc95c10ad40 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc954009ca0 tx=0x7fc95402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 -- 172.21.15.98:0/3022371100 shutdown_connections 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 --2- 172.21.15.98:0/3022371100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 0x7fc95c10ad40 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 --2- 172.21.15.98:0/3022371100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc95c103c20 0x7fc95c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 --2- 172.21.15.98:0/3022371100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc95c1029e0 0x7fc95c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 -- 172.21.15.98:0/3022371100 >> 172.21.15.98:0/3022371100 conn(0x7fc95c0fdbd0 msgr2=0x7fc95c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 -- 172.21.15.98:0/3022371100 shutdown_connections 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.472+0000 7fc9625ee640 1 -- 172.21.15.98:0/3022371100 wait complete. 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc9625ee640 1 Processor -- start 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc9625ee640 1 -- start start 2024-04-25T16:32:35.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc9625ee640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc95c1029e0 0x7fc95c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc9625ee640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc95c103c20 0x7fc95c19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc9625ee640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 0x7fc95c1a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc9625ee640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c10e3f0 con 0x7fc95c108920 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc9625ee640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c10e270 con 0x7fc95c1029e0 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc9625ee640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c10e570 con 0x7fc95c103c20 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc960b8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 0x7fc95c1a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc960b8b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 0x7fc95c1a18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37008/0 (socket says 172.21.15.98:37008) 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.473+0000 7fc960b8b640 1 -- 172.21.15.98:0/493883615 learned_addr learned my addr 172.21.15.98:0/493883615 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc95b7fe640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc95c103c20 0x7fc95c19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc95bfff640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc95c1029e0 0x7fc95c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc960b8b640 1 -- 172.21.15.98:0/493883615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc95c103c20 msgr2=0x7fc95c19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc960b8b640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc95c103c20 0x7fc95c19a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc960b8b640 1 -- 172.21.15.98:0/493883615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc95c1029e0 msgr2=0x7fc95c19a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc960b8b640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc95c1029e0 0x7fc95c19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc960b8b640 1 -- 172.21.15.98:0/493883615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95c1a1f40 con 0x7fc95c108920 2024-04-25T16:32:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc960b8b640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 0x7fc95c1a18d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc9540098d0 tx=0x7fc954003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc9597fa640 1 -- 172.21.15.98:0/493883615 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc954003d90 con 0x7fc95c108920 2024-04-25T16:32:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc9597fa640 1 -- 172.21.15.98:0/493883615 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9540044d0 con 0x7fc95c108920 2024-04-25T16:32:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95c1a21d0 con 0x7fc95c108920 2024-04-25T16:32:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc9597fa640 1 -- 172.21.15.98:0/493883615 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc954030d10 con 0x7fc95c108920 2024-04-25T16:32:35.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.474+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc95c1a2720 con 0x7fc95c108920 2024-04-25T16:32:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.475+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc95c06c2a0 con 0x7fc95c108920 2024-04-25T16:32:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.476+0000 7fc9597fa640 1 -- 172.21.15.98:0/493883615 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fc954004040 con 0x7fc95c108920 2024-04-25T16:32:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.477+0000 7fc9597fa640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9440afb80 0x7fc9440b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.477+0000 7fc9597fa640 1 -- 172.21.15.98:0/493883615 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fc9541007c0 con 0x7fc95c108920 2024-04-25T16:32:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.477+0000 7fc95bfff640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9440afb80 0x7fc9440b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.477+0000 7fc95bfff640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9440afb80 0x7fc9440b2090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc94c005fd0 tx=0x7fc94c005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:35.480 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.481+0000 7fc9597fa640 1 -- 172.21.15.98:0/493883615 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9540c8fa0 con 0x7fc95c108920 2024-04-25T16:32:35.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:35 smithi173 ceph-mon[32556]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-04-25T16:32:35.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:35 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/869456884' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:35 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/655734370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:35 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2843151341' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:35 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1095287354' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.656+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc95c1a2980 con 0x7fc95c108920 2024-04-25T16:32:35.655 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.656+0000 7fc9597fa640 1 -- 172.21.15.98:0/493883615 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7fc9540c88a0 con 0x7fc95c108920 2024-04-25T16:32:35.655 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:32:35.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9440afb80 msgr2=0x7fc9440b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9440afb80 0x7fc9440b2090 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc94c005fd0 tx=0x7fc94c005850 comp rx=0 tx=0).stop 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 msgr2=0x7fc95c1a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 0x7fc95c1a18d0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc9540098d0 tx=0x7fc954003ba0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 shutdown_connections 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9440afb80 0x7fc9440b2090 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc95c108920 0x7fc95c1a18d0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc95c103c20 0x7fc95c19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 --2- 172.21.15.98:0/493883615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc95c1029e0 0x7fc95c19a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 >> 172.21.15.98:0/493883615 conn(0x7fc95c0fdbd0 msgr2=0x7fc95c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 shutdown_connections 2024-04-25T16:32:35.658 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.658+0000 7fc9625ee640 1 -- 172.21.15.98:0/493883615 wait complete. 2024-04-25T16:32:35.669 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:32:35.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 -- 172.21.15.98:0/4076858379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4104d40 msgr2=0x7fdad4105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 --2- 172.21.15.98:0/4076858379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4104d40 0x7fdad4105150 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fdac4009900 tx=0x7fdac402a410 comp rx=0 tx=0).stop 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 -- 172.21.15.98:0/4076858379 shutdown_connections 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 --2- 172.21.15.98:0/4076858379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdad4109a40 0x7fdad410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 --2- 172.21.15.98:0/4076858379 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4104d40 0x7fdad4105150 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 --2- 172.21.15.98:0/4076858379 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdad4103b00 0x7fdad4104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 -- 172.21.15.98:0/4076858379 >> 172.21.15.98:0/4076858379 conn(0x7fdad40fecf0 msgr2=0x7fdad41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 -- 172.21.15.98:0/4076858379 shutdown_connections 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.779+0000 7fdada03c640 1 -- 172.21.15.98:0/4076858379 wait complete. 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdada03c640 1 Processor -- start 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdada03c640 1 -- start start 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdada03c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4103b00 0x7fdad419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdada03c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdad4104d40 0x7fdad419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdada03c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdad4109a40 0x7fdad41a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdada03c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdad410f4d0 con 0x7fdad4103b00 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdada03c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdad410f350 con 0x7fdad4104d40 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdada03c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdad410f650 con 0x7fdad4109a40 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdad37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4103b00 0x7fdad419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdad37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4103b00 0x7fdad419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37032/0 (socket says 172.21.15.98:37032) 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdad37fe640 1 -- 172.21.15.98:0/1591285216 learned_addr learned my addr 172.21.15.98:0/1591285216 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdad3fff640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdad4109a40 0x7fdad41a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.780+0000 7fdad2ffd640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdad4104d40 0x7fdad419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad37fe640 1 -- 172.21.15.98:0/1591285216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdad4109a40 msgr2=0x7fdad41a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad37fe640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdad4109a40 0x7fdad41a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad37fe640 1 -- 172.21.15.98:0/1591285216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdad4104d40 msgr2=0x7fdad419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad37fe640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdad4104d40 0x7fdad419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad37fe640 1 -- 172.21.15.98:0/1591285216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad41a1c70 con 0x7fdad4103b00 2024-04-25T16:32:35.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad37fe640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4103b00 0x7fdad419a130 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00e9f0 tx=0x7fdabc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad0ff9640 1 -- 172.21.15.98:0/1591285216 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc00ccc0 con 0x7fdad4103b00 2024-04-25T16:32:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad41a1f60 con 0x7fdad4103b00 2024-04-25T16:32:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad0ff9640 1 -- 172.21.15.98:0/1591285216 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdabc0035f0 con 0x7fdad4103b00 2024-04-25T16:32:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdad0ff9640 1 -- 172.21.15.98:0/1591285216 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc00ce60 con 0x7fdad4103b00 2024-04-25T16:32:35.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.781+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad41a23c0 con 0x7fdad4103b00 2024-04-25T16:32:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.782+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad4108850 con 0x7fdad4103b00 2024-04-25T16:32:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.783+0000 7fdad0ff9640 1 -- 172.21.15.98:0/1591285216 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fdabc004000 con 0x7fdad4103b00 2024-04-25T16:32:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.784+0000 7fdad0ff9640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdac00afb80 0x7fdac00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.784+0000 7fdad2ffd640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdac00afb80 0x7fdac00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.784+0000 7fdad0ff9640 1 -- 172.21.15.98:0/1591285216 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fdabc0df5a0 con 0x7fdad4103b00 2024-04-25T16:32:35.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.785+0000 7fdad2ffd640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdac00afb80 0x7fdac00b2090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdad419b1d0 tx=0x7fdac4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:35.787 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.788+0000 7fdad0ff9640 1 -- 172.21.15.98:0/1591285216 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdabc0a7c70 con 0x7fdad4103b00 2024-04-25T16:32:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.965+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdad402dda0 con 0x7fdad4103b00 2024-04-25T16:32:35.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.966+0000 7fdad0ff9640 1 -- 172.21.15.98:0/1591285216 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7fdabc0a7570 con 0x7fdad4103b00 2024-04-25T16:32:35.965 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdac00afb80 msgr2=0x7fdac00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdac00afb80 0x7fdac00b2090 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fdad419b1d0 tx=0x7fdac4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.967 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4103b00 msgr2=0x7fdad419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4103b00 0x7fdad419a130 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00e9f0 tx=0x7fdabc00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 shutdown_connections 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdac00afb80 0x7fdac00b2090 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdad4109a40 0x7fdad41a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdad4104d40 0x7fdad419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 --2- 172.21.15.98:0/1591285216 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdad4103b00 0x7fdad419a130 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 >> 172.21.15.98:0/1591285216 conn(0x7fdad40fecf0 msgr2=0x7fdad4105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 shutdown_connections 2024-04-25T16:32:35.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:35.968+0000 7fdada03c640 1 -- 172.21.15.98:0/1591285216 wait complete. 2024-04-25T16:32:35.978 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:32:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 -- 172.21.15.98:0/3077616702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee001029e0 msgr2=0x7fee001036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 --2- 172.21.15.98:0/3077616702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee001029e0 0x7fee001036e0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009900 tx=0x7fedf002a410 comp rx=0 tx=0).stop 2024-04-25T16:32:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 -- 172.21.15.98:0/3077616702 shutdown_connections 2024-04-25T16:32:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 --2- 172.21.15.98:0/3077616702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee00108920 0x7fee0010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 --2- 172.21.15.98:0/3077616702 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee00103c20 0x7fee00104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 --2- 172.21.15.98:0/3077616702 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee001029e0 0x7fee001036e0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 -- 172.21.15.98:0/3077616702 >> 172.21.15.98:0/3077616702 conn(0x7fee000fdbd0 msgr2=0x7fee00100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 -- 172.21.15.98:0/3077616702 shutdown_connections 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.091+0000 7fee06630640 1 -- 172.21.15.98:0/3077616702 wait complete. 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee06630640 1 Processor -- start 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee06630640 1 -- start start 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee06630640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee001029e0 0x7fee00199060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee06630640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee00103c20 0x7fee001995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee06630640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee00108920 0x7fee001a0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee06630640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fee0010e390 con 0x7fee00108920 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee06630640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fee0010e210 con 0x7fee00103c20 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee06630640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fee0010e510 con 0x7fee001029e0 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee04bcd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee00108920 0x7fee001a0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee04bcd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee00108920 0x7fee001a0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37040/0 (socket says 172.21.15.98:37040) 2024-04-25T16:32:36.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee04bcd640 1 -- 172.21.15.98:0/1326802175 learned_addr learned my addr 172.21.15.98:0/1326802175 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fedfffff640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee001029e0 0x7fee00199060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fedff7fe640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee00103c20 0x7fee001995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee04bcd640 1 -- 172.21.15.98:0/1326802175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee001029e0 msgr2=0x7fee00199060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee04bcd640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee001029e0 0x7fee00199060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee04bcd640 1 -- 172.21.15.98:0/1326802175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee00103c20 msgr2=0x7fee001995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee04bcd640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee00103c20 0x7fee001995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fee04bcd640 1 -- 172.21.15.98:0/1326802175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee001a0ba0 con 0x7fee00108920 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.092+0000 7fedff7fe640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee00103c20 0x7fee001995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.093+0000 7fee04bcd640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee00108920 0x7fee001a0530 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fedf800eaa0 tx=0x7fedf800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.093+0000 7fedfd7fa640 1 -- 172.21.15.98:0/1326802175 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf800cdb0 con 0x7fee00108920 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.093+0000 7fedfd7fa640 1 -- 172.21.15.98:0/1326802175 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fedf8005020 con 0x7fee00108920 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.093+0000 7fedfd7fa640 1 -- 172.21.15.98:0/1326802175 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf80116d0 con 0x7fee00108920 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.093+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee001a0e90 con 0x7fee00108920 2024-04-25T16:32:36.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.093+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee001a12a0 con 0x7fee00108920 2024-04-25T16:32:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.094+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee00107730 con 0x7fee00108920 2024-04-25T16:32:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.095+0000 7fedfd7fa640 1 -- 172.21.15.98:0/1326802175 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fedf80051d0 con 0x7fee00108920 2024-04-25T16:32:36.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.096+0000 7fedfd7fa640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fede80afb80 0x7fede80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.096+0000 7fedfffff640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fede80afb80 0x7fede80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.096+0000 7fedfd7fa640 1 -- 172.21.15.98:0/1326802175 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fedf80df6c0 con 0x7fee00108920 2024-04-25T16:32:36.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.096+0000 7fedfffff640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fede80afb80 0x7fede80b2090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fedf0005760 tx=0x7fedf0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:36.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.099+0000 7fedfd7fa640 1 -- 172.21.15.98:0/1326802175 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fedf8016030 con 0x7fee00108920 2024-04-25T16:32:36.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1187215710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:36.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/493883615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:36.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:36 smithi098 ceph-mon[29162]: pgmap v359: 129 pgs: 129 active+clean; 649 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T16:32:36.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:36 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1591285216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:36.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.279+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fee0002dda0 con 0x7fee00108920 2024-04-25T16:32:36.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.280+0000 7fedfd7fa640 1 -- 172.21.15.98:0/1326802175 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7fedf80a1e20 con 0x7fee00108920 2024-04-25T16:32:36.279 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:32:36.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fede80afb80 msgr2=0x7fede80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fede80afb80 0x7fede80b2090 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fedf0005760 tx=0x7fedf0000bc0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee00108920 msgr2=0x7fee001a0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee00108920 0x7fee001a0530 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fedf800eaa0 tx=0x7fedf800ef60 comp rx=0 tx=0).stop 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 shutdown_connections 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fede80afb80 0x7fede80b2090 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fee00108920 0x7fee001a0530 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fee00103c20 0x7fee001995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 --2- 172.21.15.98:0/1326802175 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee001029e0 0x7fee00199060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 >> 172.21.15.98:0/1326802175 conn(0x7fee000fdbd0 msgr2=0x7fee00104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 shutdown_connections 2024-04-25T16:32:36.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.282+0000 7fee06630640 1 -- 172.21.15.98:0/1326802175 wait complete. 2024-04-25T16:32:36.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:32:36.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 -- 172.21.15.98:0/3990257521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc103960 msgr2=0x7f94bc103df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 --2- 172.21.15.98:0/3990257521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc103960 0x7f94bc103df0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f94ac009900 tx=0x7f94ac02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 -- 172.21.15.98:0/3990257521 shutdown_connections 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 --2- 172.21.15.98:0/3990257521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94bc104480 0x7f94bc10abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 --2- 172.21.15.98:0/3990257521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc103960 0x7f94bc103df0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 --2- 172.21.15.98:0/3990257521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc102f80 0x7f94bc103390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 -- 172.21.15.98:0/3990257521 >> 172.21.15.98:0/3990257521 conn(0x7f94bc0fd380 msgr2=0x7f94bc0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 -- 172.21.15.98:0/3990257521 shutdown_connections 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 -- 172.21.15.98:0/3990257521 wait complete. 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 Processor -- start 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.404+0000 7f94c476c640 1 -- start start 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c476c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc102f80 0x7f94bc199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c476c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc103960 0x7f94bc199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c476c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94bc104480 0x7f94bc1a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c476c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc10e450 con 0x7f94bc102f80 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c476c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc10e2d0 con 0x7f94bc104480 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c476c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f94bc10e5d0 con 0x7f94bc103960 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc102f80 0x7f94bc199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc102f80 0x7f94bc199050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37060/0 (socket says 172.21.15.98:37060) 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 -- 172.21.15.98:0/3236520517 learned_addr learned my addr 172.21.15.98:0/3236520517 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c1d07640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc103960 0x7f94bc199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 -- 172.21.15.98:0/3236520517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc103960 msgr2=0x7f94bc199590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc103960 0x7f94bc199590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 -- 172.21.15.98:0/3236520517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94bc104480 msgr2=0x7f94bc1a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2d09640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94bc104480 0x7f94bc1a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94bc104480 0x7f94bc1a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 -- 172.21.15.98:0/3236520517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94bc1a0ae0 con 0x7f94bc102f80 2024-04-25T16:32:36.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2d09640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94bc104480 0x7f94bc1a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c2508640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc102f80 0x7f94bc199050 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f94b800ccb0 tx=0x7f94b800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94b37fe640 1 -- 172.21.15.98:0/3236520517 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b800ce60 con 0x7f94bc102f80 2024-04-25T16:32:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94b37fe640 1 -- 172.21.15.98:0/3236520517 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f94b800fae0 con 0x7f94bc102f80 2024-04-25T16:32:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94bc1a0dd0 con 0x7f94bc102f80 2024-04-25T16:32:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94b37fe640 1 -- 172.21.15.98:0/3236520517 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b8018aa0 con 0x7f94bc102f80 2024-04-25T16:32:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.405+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94bc1a12b0 con 0x7f94bc102f80 2024-04-25T16:32:36.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.406+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94bc06a190 con 0x7f94bc102f80 2024-04-25T16:32:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.408+0000 7f94b37fe640 1 -- 172.21.15.98:0/3236520517 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f94b8018c40 con 0x7f94bc102f80 2024-04-25T16:32:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.408+0000 7f94b37fe640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afb80 0x7f94a80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.408+0000 7f94c1d07640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afb80 0x7f94a80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.408+0000 7f94b37fe640 1 -- 172.21.15.98:0/3236520517 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f94b80df9e0 con 0x7f94bc102f80 2024-04-25T16:32:36.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.409+0000 7f94c1d07640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afb80 0x7f94a80b2090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f94ac006010 tx=0x7f94ac000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:36.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.412+0000 7f94b37fe640 1 -- 172.21.15.98:0/3236520517 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94bc06a190 con 0x7f94bc102f80 2024-04-25T16:32:36.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1187215710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:36.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/493883615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:36.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:36 smithi203 ceph-mon[33115]: pgmap v359: 129 pgs: 129 active+clean; 649 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T16:32:36.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:36 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1591285216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:36.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1187215710' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:36.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/493883615' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:36.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:36 smithi173 ceph-mon[32556]: pgmap v359: 129 pgs: 129 active+clean; 649 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T16:32:36.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:36 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1591285216' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.586+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f94bc02dbf0 con 0x7f94bc102f80 2024-04-25T16:32:36.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.587+0000 7f94b37fe640 1 -- 172.21.15.98:0/3236520517 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f94b8014030 con 0x7f94bc102f80 2024-04-25T16:32:36.587 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afb80 msgr2=0x7f94a80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afb80 0x7f94a80b2090 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f94ac006010 tx=0x7f94ac000bc0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc102f80 msgr2=0x7f94bc199050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc102f80 0x7f94bc199050 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f94b800ccb0 tx=0x7f94b800a620 comp rx=0 tx=0).stop 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 shutdown_connections 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f94a80afb80 0x7f94a80b2090 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94bc104480 0x7f94bc1a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94bc103960 0x7f94bc199590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 --2- 172.21.15.98:0/3236520517 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f94bc102f80 0x7f94bc199050 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 >> 172.21.15.98:0/3236520517 conn(0x7f94bc0fd380 msgr2=0x7f94bc104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 shutdown_connections 2024-04-25T16:32:36.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.589+0000 7f94c476c640 1 -- 172.21.15.98:0/3236520517 wait complete. 2024-04-25T16:32:36.599 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T16:32:36.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:32:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 -- 172.21.15.98:0/1172049935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 msgr2=0x7f9258105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 --2- 172.21.15.98:0/1172049935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 0x7f9258105150 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f924c009900 tx=0x7f924c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 -- 172.21.15.98:0/1172049935 shutdown_connections 2024-04-25T16:32:36.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 --2- 172.21.15.98:0/1172049935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9258109a40 0x7f925810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 --2- 172.21.15.98:0/1172049935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 0x7f9258105150 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 --2- 172.21.15.98:0/1172049935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9258103b00 0x7f9258104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 -- 172.21.15.98:0/1172049935 >> 172.21.15.98:0/1172049935 conn(0x7f92580fed10 msgr2=0x7f92581011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 -- 172.21.15.98:0/1172049935 shutdown_connections 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.708+0000 7f925cc27640 1 -- 172.21.15.98:0/1172049935 wait complete. 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.709+0000 7f925cc27640 1 Processor -- start 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.709+0000 7f925cc27640 1 -- start start 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.709+0000 7f925cc27640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9258103b00 0x7f925819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.709+0000 7f925cc27640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 0x7f925819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.709+0000 7f925cc27640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9258109a40 0x7f92581a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.709+0000 7f925cc27640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f925810f4d0 con 0x7f9258104d40 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.709+0000 7f925cc27640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f925810f350 con 0x7f9258103b00 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.709+0000 7f925cc27640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f925810f650 con 0x7f9258109a40 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 0x7f925819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 0x7f925819a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37080/0 (socket says 172.21.15.98:37080) 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 -- 172.21.15.98:0/4169282731 learned_addr learned my addr 172.21.15.98:0/4169282731 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9256d9d640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9258109a40 0x7f92581a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f925659c640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9258103b00 0x7f925819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 -- 172.21.15.98:0/4169282731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9258109a40 msgr2=0x7f92581a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9258109a40 0x7f92581a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 -- 172.21.15.98:0/4169282731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9258103b00 msgr2=0x7f925819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9258103b00 0x7f925819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 -- 172.21.15.98:0/4169282731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92581a1c80 con 0x7f9258104d40 2024-04-25T16:32:36.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f925659c640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9258103b00 0x7f925819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f9255d9b640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 0x7f925819a6d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f924c009230 tx=0x7f924c0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f92477fe640 1 -- 172.21.15.98:0/4169282731 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c03f030 con 0x7f9258104d40 2024-04-25T16:32:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f92477fe640 1 -- 172.21.15.98:0/4169282731 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f924c0044c0 con 0x7f9258104d40 2024-04-25T16:32:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f92477fe640 1 -- 172.21.15.98:0/4169282731 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f924c035070 con 0x7f9258104d40 2024-04-25T16:32:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92581a1f10 con 0x7f9258104d40 2024-04-25T16:32:36.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.710+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92581a22c0 con 0x7f9258104d40 2024-04-25T16:32:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.712+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9258108850 con 0x7f9258104d40 2024-04-25T16:32:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.713+0000 7f92477fe640 1 -- 172.21.15.98:0/4169282731 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f924c042030 con 0x7f9258104d40 2024-04-25T16:32:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.713+0000 7f92477fe640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f92400afb80 0x7f92400b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.713+0000 7f925659c640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f92400afb80 0x7f92400b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.713+0000 7f92477fe640 1 -- 172.21.15.98:0/4169282731 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f924c0ff740 con 0x7f9258104d40 2024-04-25T16:32:36.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.714+0000 7f925659c640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f92400afb80 0x7f92400b2090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9258104700 tx=0x7f9248005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:36.717 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.718+0000 7f92477fe640 1 -- 172.21.15.98:0/4169282731 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f924c0c7e10 con 0x7f9258104d40 2024-04-25T16:32:36.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.934+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f925802db50 con 0x7f9258104d40 2024-04-25T16:32:36.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.935+0000 7f92477fe640 1 -- 172.21.15.98:0/4169282731 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11935 (secure 0 0 0) 0x7f924c0c7710 con 0x7f9258104d40 2024-04-25T16:32:36.934 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:36.934 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:32:32.844737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24494,24473,24457,24484,24539]},"id":1}]} 2024-04-25T16:32:36.935 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f92400afb80 msgr2=0x7f92400b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f92400afb80 0x7f92400b2090 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9258104700 tx=0x7f9248005f90 comp rx=0 tx=0).stop 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 msgr2=0x7f925819a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 0x7f925819a6d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f924c009230 tx=0x7f924c0040c0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 shutdown_connections 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f92400afb80 0x7f92400b2090 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9258109a40 0x7f92581a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9258104d40 0x7f925819a6d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 --2- 172.21.15.98:0/4169282731 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9258103b00 0x7f925819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:36.937 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 >> 172.21.15.98:0/4169282731 conn(0x7f92580fed10 msgr2=0x7f9258105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 shutdown_connections 2024-04-25T16:32:36.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:36.937+0000 7f925cc27640 1 -- 172.21.15.98:0/4169282731 wait complete. 2024-04-25T16:32:36.948 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T16:32:36.948 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T16:32:36.948 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:32:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.057+0000 7fed2de0c640 1 -- 172.21.15.98:0/3111513364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 msgr2=0x7fed281036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.057+0000 7fed2de0c640 1 --2- 172.21.15.98:0/3111513364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 0x7fed281036c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fed18009900 tx=0x7fed1802a410 comp rx=0 tx=0).stop 2024-04-25T16:32:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.057+0000 7fed2de0c640 1 -- 172.21.15.98:0/3111513364 shutdown_connections 2024-04-25T16:32:37.059 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 --2- 172.21.15.98:0/3111513364 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed28108900 0x7fed2810ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 --2- 172.21.15.98:0/3111513364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed28103c00 0x7fed28104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 --2- 172.21.15.98:0/3111513364 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 0x7fed281036c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 -- 172.21.15.98:0/3111513364 >> 172.21.15.98:0/3111513364 conn(0x7fed280fdbd0 msgr2=0x7fed28100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 -- 172.21.15.98:0/3111513364 shutdown_connections 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 -- 172.21.15.98:0/3111513364 wait complete. 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 Processor -- start 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 -- start start 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 0x7fed2819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.058+0000 7fed2de0c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed28103c00 0x7fed2819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed2de0c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed28108900 0x7fed281a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed2de0c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fed2810f4a0 con 0x7fed281029c0 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed2de0c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fed2810f320 con 0x7fed28103c00 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed2de0c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fed2810f620 con 0x7fed28108900 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 0x7fed2819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.060 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 0x7fed2819a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37096/0 (socket says 172.21.15.98:37096) 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 -- 172.21.15.98:0/2100789542 learned_addr learned my addr 172.21.15.98:0/2100789542 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 -- 172.21.15.98:0/2100789542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed28108900 msgr2=0x7fed281a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed28108900 0x7fed281a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 -- 172.21.15.98:0/2100789542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed28103c00 msgr2=0x7fed2819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed26ffd640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed28103c00 0x7fed2819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed28103c00 0x7fed2819a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 -- 172.21.15.98:0/2100789542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed281a1cb0 con 0x7fed281029c0 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed277fe640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 0x7fed2819a170 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fed18005ab0 tx=0x7fed18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed24ff9640 1 -- 172.21.15.98:0/2100789542 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1803f070 con 0x7fed281029c0 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed24ff9640 1 -- 172.21.15.98:0/2100789542 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fed18003c80 con 0x7fed281029c0 2024-04-25T16:32:37.061 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed281a1f40 con 0x7fed281029c0 2024-04-25T16:32:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed24ff9640 1 -- 172.21.15.98:0/2100789542 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18035070 con 0x7fed281029c0 2024-04-25T16:32:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.059+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed281a2340 con 0x7fed281029c0 2024-04-25T16:32:37.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.060+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed28107710 con 0x7fed281029c0 2024-04-25T16:32:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.062+0000 7fed24ff9640 1 -- 172.21.15.98:0/2100789542 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fed1803b610 con 0x7fed281029c0 2024-04-25T16:32:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.062+0000 7fed24ff9640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fed100afb80 0x7fed100b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.062+0000 7fed26ffd640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fed100afb80 0x7fed100b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.062+0000 7fed24ff9640 1 -- 172.21.15.98:0/2100789542 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fed181001d0 con 0x7fed281029c0 2024-04-25T16:32:37.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.063+0000 7fed26ffd640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fed100afb80 0x7fed100b2090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fed1c0058c0 tx=0x7fed1c005c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:37.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.066+0000 7fed24ff9640 1 -- 172.21.15.98:0/2100789542 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed180c88a0 con 0x7fed281029c0 2024-04-25T16:32:37.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.243+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fed28062fe0 con 0x7fed100afb80 2024-04-25T16:32:37.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:37 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1326802175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:37.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:37 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3236520517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:37.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:37 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4169282731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:37.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.253+0000 7fed24ff9640 1 -- 172.21.15.98:0/2100789542 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340525 (secure 0 0 0) 0x7fed28062fe0 con 0x7fed100afb80 2024-04-25T16:32:37.257 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:37.259 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T16:32:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fed100afb80 msgr2=0x7fed100b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fed100afb80 0x7fed100b2090 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fed1c0058c0 tx=0x7fed1c005c80 comp rx=0 tx=0).stop 2024-04-25T16:32:37.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 msgr2=0x7fed2819a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 0x7fed2819a170 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fed18005ab0 tx=0x7fed18004270 comp rx=0 tx=0).stop 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 shutdown_connections 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fed100afb80 0x7fed100b2090 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fed28108900 0x7fed281a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed28103c00 0x7fed2819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 --2- 172.21.15.98:0/2100789542 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fed281029c0 0x7fed2819a170 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.256+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 >> 172.21.15.98:0/2100789542 conn(0x7fed280fdbd0 msgr2=0x7fed281048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.257+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 shutdown_connections 2024-04-25T16:32:37.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.257+0000 7fed2de0c640 1 -- 172.21.15.98:0/2100789542 wait complete. 2024-04-25T16:32:37.267 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-04-25T16:32:35.964692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664158,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1262,"num_read_kb":736,"num_write":398,"num_write_kb":2316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2951957,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":829,"ondisk_log_size":829,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":395960,"kb_used_data":13520,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124545864,"statfs":{"total":1151940427776,"available":1151534964736,"internally_reserved":0,"allocated":13844480,"data_stored":7663477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2465,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":109,"num_read_kb":59,"num_write":22,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003530"},"pg_stats":[{"pgid":"3.3f","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021251+0000","last_change":"2024-04-25T16:30:09.990700+0000","last_active":"2024-04-25T16:32:10.021251+0000","last_peered":"2024-04-25T16:32:10.021251+0000","last_clean":"2024-04-25T16:32:10.021251+0000","last_became_active":"2024-04-25T16:30:09.990225+0000","last_became_peered":"2024-04-25T16:30:09.990225+0000","last_unstale":"2024-04-25T16:32:10.021251+0000","last_undegraded":"2024-04-25T16:32:10.021251+0000","last_fullsized":"2024-04-25T16:32:10.021251+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:33.897277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.782026+0000","last_change":"2024-04-25T16:30:09.989618+0000","last_active":"2024-04-25T16:32:10.782026+0000","last_peered":"2024-04-25T16:32:10.782026+0000","last_clean":"2024-04-25T16:32:10.782026+0000","last_became_active":"2024-04-25T16:30:09.989270+0000","last_became_peered":"2024-04-25T16:30:09.989270+0000","last_unstale":"2024-04-25T16:32:10.782026+0000","last_undegraded":"2024-04-25T16:32:10.782026+0000","last_fullsized":"2024-04-25T16:32:10.782026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:03.952233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781848+0000","last_change":"2024-04-25T16:30:09.992442+0000","last_active":"2024-04-25T16:32:10.781848+0000","last_peered":"2024-04-25T16:32:10.781848+0000","last_clean":"2024-04-25T16:32:10.781848+0000","last_became_active":"2024-04-25T16:30:09.992115+0000","last_became_peered":"2024-04-25T16:30:09.992115+0000","last_unstale":"2024-04-25T16:32:10.781848+0000","last_undegraded":"2024-04-25T16:32:10.781848+0000","last_fullsized":"2024-04-25T16:32:10.781848+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:00.668427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017939+0000","last_change":"2024-04-25T16:31:09.006824+0000","last_active":"2024-04-25T16:32:10.017939+0000","last_peered":"2024-04-25T16:32:10.017939+0000","last_clean":"2024-04-25T16:32:10.017939+0000","last_became_active":"2024-04-25T16:31:09.005560+0000","last_became_peered":"2024-04-25T16:31:09.005560+0000","last_unstale":"2024-04-25T16:32:10.017939+0000","last_undegraded":"2024-04-25T16:32:10.017939+0000","last_fullsized":"2024-04-25T16:32:10.017939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:41.979642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020683+0000","last_change":"2024-04-25T16:30:09.991372+0000","last_active":"2024-04-25T16:32:10.020683+0000","last_peered":"2024-04-25T16:32:10.020683+0000","last_clean":"2024-04-25T16:32:10.020683+0000","last_became_active":"2024-04-25T16:30:09.991053+0000","last_became_peered":"2024-04-25T16:30:09.991053+0000","last_unstale":"2024-04-25T16:32:10.020683+0000","last_undegraded":"2024-04-25T16:32:10.020683+0000","last_fullsized":"2024-04-25T16:32:10.020683+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:28.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014900+0000","last_change":"2024-04-25T16:30:09.997563+0000","last_active":"2024-04-25T16:32:10.014900+0000","last_peered":"2024-04-25T16:32:10.014900+0000","last_clean":"2024-04-25T16:32:10.014900+0000","last_became_active":"2024-04-25T16:30:09.997158+0000","last_became_peered":"2024-04-25T16:30:09.997158+0000","last_unstale":"2024-04-25T16:32:10.014900+0000","last_undegraded":"2024-04-25T16:32:10.014900+0000","last_fullsized":"2024-04-25T16:32:10.014900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:21.441957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021712+0000","last_change":"2024-04-25T16:31:09.009888+0000","last_active":"2024-04-25T16:32:10.021712+0000","last_peered":"2024-04-25T16:32:10.021712+0000","last_clean":"2024-04-25T16:32:10.021712+0000","last_became_active":"2024-04-25T16:31:09.009201+0000","last_became_peered":"2024-04-25T16:31:09.009201+0000","last_unstale":"2024-04-25T16:32:10.021712+0000","last_undegraded":"2024-04-25T16:32:10.021712+0000","last_fullsized":"2024-04-25T16:32:10.021712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:52.171410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016068+0000","last_change":"2024-04-25T16:30:09.994820+0000","last_active":"2024-04-25T16:32:10.016068+0000","last_peered":"2024-04-25T16:32:10.016068+0000","last_clean":"2024-04-25T16:32:10.016068+0000","last_became_active":"2024-04-25T16:30:09.994494+0000","last_became_peered":"2024-04-25T16:30:09.994494+0000","last_unstale":"2024-04-25T16:32:10.016068+0000","last_undegraded":"2024-04-25T16:32:10.016068+0000","last_fullsized":"2024-04-25T16:32:10.016068+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:39.373266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530879+0000","last_change":"2024-04-25T16:30:09.994870+0000","last_active":"2024-04-25T16:32:10.530879+0000","last_peered":"2024-04-25T16:32:10.530879+0000","last_clean":"2024-04-25T16:32:10.530879+0000","last_became_active":"2024-04-25T16:30:09.994521+0000","last_became_peered":"2024-04-25T16:30:09.994521+0000","last_unstale":"2024-04-25T16:32:10.530879+0000","last_undegraded":"2024-04-25T16:32:10.530879+0000","last_fullsized":"2024-04-25T16:32:10.530879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:18.093904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.015411+0000","last_change":"2024-04-25T16:31:09.010148+0000","last_active":"2024-04-25T16:32:10.015411+0000","last_peered":"2024-04-25T16:32:10.015411+0000","last_clean":"2024-04-25T16:32:10.015411+0000","last_became_active":"2024-04-25T16:31:09.009655+0000","last_became_peered":"2024-04-25T16:31:09.009655+0000","last_unstale":"2024-04-25T16:32:10.015411+0000","last_undegraded":"2024-04-25T16:32:10.015411+0000","last_fullsized":"2024-04-25T16:32:10.015411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:18.637202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022213+0000","last_change":"2024-04-25T16:30:09.999519+0000","last_active":"2024-04-25T16:32:10.022213+0000","last_peered":"2024-04-25T16:32:10.022213+0000","last_clean":"2024-04-25T16:32:10.022213+0000","last_became_active":"2024-04-25T16:30:09.999158+0000","last_became_peered":"2024-04-25T16:30:09.999158+0000","last_unstale":"2024-04-25T16:32:10.022213+0000","last_undegraded":"2024-04-25T16:32:10.022213+0000","last_fullsized":"2024-04-25T16:32:10.022213+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:38:06.686541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212538+0000","last_change":"2024-04-25T16:30:10.005455+0000","last_active":"2024-04-25T16:32:10.212538+0000","last_peered":"2024-04-25T16:32:10.212538+0000","last_clean":"2024-04-25T16:32:10.212538+0000","last_became_active":"2024-04-25T16:30:10.004891+0000","last_became_peered":"2024-04-25T16:30:10.004891+0000","last_unstale":"2024-04-25T16:32:10.212538+0000","last_undegraded":"2024-04-25T16:32:10.212538+0000","last_fullsized":"2024-04-25T16:32:10.212538+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.598321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014090+0000","last_change":"2024-04-25T16:30:09.992870+0000","last_active":"2024-04-25T16:32:10.014090+0000","last_peered":"2024-04-25T16:32:10.014090+0000","last_clean":"2024-04-25T16:32:10.014090+0000","last_became_active":"2024-04-25T16:30:09.992399+0000","last_became_peered":"2024-04-25T16:30:09.992399+0000","last_unstale":"2024-04-25T16:32:10.014090+0000","last_undegraded":"2024-04-25T16:32:10.014090+0000","last_fullsized":"2024-04-25T16:32:10.014090+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.127287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781631+0000","last_change":"2024-04-25T16:30:09.988707+0000","last_active":"2024-04-25T16:32:10.781631+0000","last_peered":"2024-04-25T16:32:10.781631+0000","last_clean":"2024-04-25T16:32:10.781631+0000","last_became_active":"2024-04-25T16:30:09.988386+0000","last_became_peered":"2024-04-25T16:30:09.988386+0000","last_unstale":"2024-04-25T16:32:10.781631+0000","last_undegraded":"2024-04-25T16:32:10.781631+0000","last_fullsized":"2024-04-25T16:32:10.781631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:03:15.973591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781712+0000","last_change":"2024-04-25T16:30:10.002780+0000","last_active":"2024-04-25T16:32:10.781712+0000","last_peered":"2024-04-25T16:32:10.781712+0000","last_clean":"2024-04-25T16:32:10.781712+0000","last_became_active":"2024-04-25T16:30:10.002055+0000","last_became_peered":"2024-04-25T16:30:10.002055+0000","last_unstale":"2024-04-25T16:32:10.781712+0000","last_undegraded":"2024-04-25T16:32:10.781712+0000","last_fullsized":"2024-04-25T16:32:10.781712+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:23.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530394+0000","last_change":"2024-04-25T16:30:09.998270+0000","last_active":"2024-04-25T16:32:10.530394+0000","last_peered":"2024-04-25T16:32:10.530394+0000","last_clean":"2024-04-25T16:32:10.530394+0000","last_became_active":"2024-04-25T16:30:09.997903+0000","last_became_peered":"2024-04-25T16:30:09.997903+0000","last_unstale":"2024-04-25T16:32:10.530394+0000","last_undegraded":"2024-04-25T16:32:10.530394+0000","last_fullsized":"2024-04-25T16:32:10.530394+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:14.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021350+0000","last_change":"2024-04-25T16:30:09.988894+0000","last_active":"2024-04-25T16:32:10.021350+0000","last_peered":"2024-04-25T16:32:10.021350+0000","last_clean":"2024-04-25T16:32:10.021350+0000","last_became_active":"2024-04-25T16:30:09.988508+0000","last_became_peered":"2024-04-25T16:30:09.988508+0000","last_unstale":"2024-04-25T16:32:10.021350+0000","last_undegraded":"2024-04-25T16:32:10.021350+0000","last_fullsized":"2024-04-25T16:32:10.021350+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:48.226193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018080+0000","last_change":"2024-04-25T16:30:09.989910+0000","last_active":"2024-04-25T16:32:10.018080+0000","last_peered":"2024-04-25T16:32:10.018080+0000","last_clean":"2024-04-25T16:32:10.018080+0000","last_became_active":"2024-04-25T16:30:09.989560+0000","last_became_peered":"2024-04-25T16:30:09.989560+0000","last_unstale":"2024-04-25T16:32:10.018080+0000","last_undegraded":"2024-04-25T16:32:10.018080+0000","last_fullsized":"2024-04-25T16:32:10.018080+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:12.834034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017567+0000","last_change":"2024-04-25T16:30:09.987930+0000","last_active":"2024-04-25T16:32:10.017567+0000","last_peered":"2024-04-25T16:32:10.017567+0000","last_clean":"2024-04-25T16:32:10.017567+0000","last_became_active":"2024-04-25T16:30:09.987595+0000","last_became_peered":"2024-04-25T16:30:09.987595+0000","last_unstale":"2024-04-25T16:32:10.017567+0000","last_undegraded":"2024-04-25T16:32:10.017567+0000","last_fullsized":"2024-04-25T16:32:10.017567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:04.778172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016791+0000","last_change":"2024-04-25T16:30:14.006360+0000","last_active":"2024-04-25T16:32:10.016791+0000","last_peered":"2024-04-25T16:32:10.016791+0000","last_clean":"2024-04-25T16:32:10.016791+0000","last_became_active":"2024-04-25T16:30:14.005894+0000","last_became_peered":"2024-04-25T16:30:14.005894+0000","last_unstale":"2024-04-25T16:32:10.016791+0000","last_undegraded":"2024-04-25T16:32:10.016791+0000","last_fullsized":"2024-04-25T16:32:10.016791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:33.150381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'14","reported_seq":164,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:33.234677+0000","last_change":"2024-04-25T16:30:08.992779+0000","last_active":"2024-04-25T16:32:33.234677+0000","last_peered":"2024-04-25T16:32:33.234677+0000","last_clean":"2024-04-25T16:32:33.234677+0000","last_became_active":"2024-04-25T16:30:08.992458+0000","last_became_peered":"2024-04-25T16:30:08.992458+0000","last_unstale":"2024-04-25T16:32:33.234677+0000","last_undegraded":"2024-04-25T16:32:33.234677+0000","last_fullsized":"2024-04-25T16:32:33.234677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:21.430855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016381+0000","last_change":"2024-04-25T16:30:09.995353+0000","last_active":"2024-04-25T16:32:10.016381+0000","last_peered":"2024-04-25T16:32:10.016381+0000","last_clean":"2024-04-25T16:32:10.016381+0000","last_became_active":"2024-04-25T16:30:09.995011+0000","last_became_peered":"2024-04-25T16:30:09.995011+0000","last_unstale":"2024-04-25T16:32:10.016381+0000","last_undegraded":"2024-04-25T16:32:10.016381+0000","last_fullsized":"2024-04-25T16:32:10.016381+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:33.367268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211820+0000","last_change":"2024-04-25T16:31:09.187252+0000","last_active":"2024-04-25T16:32:10.211820+0000","last_peered":"2024-04-25T16:32:10.211820+0000","last_clean":"2024-04-25T16:32:10.211820+0000","last_became_active":"2024-04-25T16:31:09.186689+0000","last_became_peered":"2024-04-25T16:31:09.186689+0000","last_unstale":"2024-04-25T16:32:10.211820+0000","last_undegraded":"2024-04-25T16:32:10.211820+0000","last_fullsized":"2024-04-25T16:32:10.211820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:18.287476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016628+0000","last_change":"2024-04-25T16:30:08.997663+0000","last_active":"2024-04-25T16:32:10.016628+0000","last_peered":"2024-04-25T16:32:10.016628+0000","last_clean":"2024-04-25T16:32:10.016628+0000","last_became_active":"2024-04-25T16:30:08.997107+0000","last_became_peered":"2024-04-25T16:30:08.997107+0000","last_unstale":"2024-04-25T16:32:10.016628+0000","last_undegraded":"2024-04-25T16:32:10.016628+0000","last_fullsized":"2024-04-25T16:32:10.016628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:25.856556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781991+0000","last_change":"2024-04-25T16:30:09.987947+0000","last_active":"2024-04-25T16:32:10.781991+0000","last_peered":"2024-04-25T16:32:10.781991+0000","last_clean":"2024-04-25T16:32:10.781991+0000","last_became_active":"2024-04-25T16:30:09.987575+0000","last_became_peered":"2024-04-25T16:30:09.987575+0000","last_unstale":"2024-04-25T16:32:10.781991+0000","last_undegraded":"2024-04-25T16:32:10.781991+0000","last_fullsized":"2024-04-25T16:32:10.781991+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:28.443249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211996+0000","last_change":"2024-04-25T16:30:14.006711+0000","last_active":"2024-04-25T16:32:10.211996+0000","last_peered":"2024-04-25T16:32:10.211996+0000","last_clean":"2024-04-25T16:32:10.211996+0000","last_became_active":"2024-04-25T16:30:14.005500+0000","last_became_peered":"2024-04-25T16:30:14.005500+0000","last_unstale":"2024-04-25T16:32:10.211996+0000","last_undegraded":"2024-04-25T16:32:10.211996+0000","last_fullsized":"2024-04-25T16:32:10.211996+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:58:26.288410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.034584+0000","last_change":"2024-04-25T16:32:10.034065+0000","last_active":"2024-04-25T16:32:10.034584+0000","last_peered":"2024-04-25T16:32:10.034584+0000","last_clean":"2024-04-25T16:32:10.034584+0000","last_became_active":"2024-04-25T16:32:10.022888+0000","last_became_peered":"2024-04-25T16:32:10.022888+0000","last_unstale":"2024-04-25T16:32:10.034584+0000","last_undegraded":"2024-04-25T16:32:10.034584+0000","last_fullsized":"2024-04-25T16:32:10.034584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:58:44.139220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530964+0000","last_change":"2024-04-25T16:30:09.988216+0000","last_active":"2024-04-25T16:32:10.530964+0000","last_peered":"2024-04-25T16:32:10.530964+0000","last_clean":"2024-04-25T16:32:10.530964+0000","last_became_active":"2024-04-25T16:30:09.987752+0000","last_became_peered":"2024-04-25T16:30:09.987752+0000","last_unstale":"2024-04-25T16:32:10.530964+0000","last_undegraded":"2024-04-25T16:32:10.530964+0000","last_fullsized":"2024-04-25T16:32:10.530964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:57:35.583000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211839+0000","last_change":"2024-04-25T16:30:14.001329+0000","last_active":"2024-04-25T16:32:10.211839+0000","last_peered":"2024-04-25T16:32:10.211839+0000","last_clean":"2024-04-25T16:32:10.211839+0000","last_became_active":"2024-04-25T16:30:14.000878+0000","last_became_peered":"2024-04-25T16:30:14.000878+0000","last_unstale":"2024-04-25T16:32:10.211839+0000","last_undegraded":"2024-04-25T16:32:10.211839+0000","last_fullsized":"2024-04-25T16:32:10.211839+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:07.946611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020332+0000","last_change":"2024-04-25T16:30:08.989065+0000","last_active":"2024-04-25T16:32:10.020332+0000","last_peered":"2024-04-25T16:32:10.020332+0000","last_clean":"2024-04-25T16:32:10.020332+0000","last_became_active":"2024-04-25T16:30:08.988709+0000","last_became_peered":"2024-04-25T16:30:08.988709+0000","last_unstale":"2024-04-25T16:32:10.020332+0000","last_undegraded":"2024-04-25T16:32:10.020332+0000","last_fullsized":"2024-04-25T16:32:10.020332+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:34.306581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205194+0000","last_change":"2024-04-25T16:30:09.999641+0000","last_active":"2024-04-25T16:32:10.205194+0000","last_peered":"2024-04-25T16:32:10.205194+0000","last_clean":"2024-04-25T16:32:10.205194+0000","last_became_active":"2024-04-25T16:30:09.999319+0000","last_became_peered":"2024-04-25T16:30:09.999319+0000","last_unstale":"2024-04-25T16:32:10.205194+0000","last_undegraded":"2024-04-25T16:32:10.205194+0000","last_fullsized":"2024-04-25T16:32:10.205194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:14.043201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017299+0000","last_change":"2024-04-25T16:30:13.999609+0000","last_active":"2024-04-25T16:32:10.017299+0000","last_peered":"2024-04-25T16:32:10.017299+0000","last_clean":"2024-04-25T16:32:10.017299+0000","last_became_active":"2024-04-25T16:30:13.999172+0000","last_became_peered":"2024-04-25T16:30:13.999172+0000","last_unstale":"2024-04-25T16:32:10.017299+0000","last_undegraded":"2024-04-25T16:32:10.017299+0000","last_fullsized":"2024-04-25T16:32:10.017299+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:49.878153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'8","reported_seq":158,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.240175+0000","last_change":"2024-04-25T16:30:08.988919+0000","last_active":"2024-04-25T16:32:31.240175+0000","last_peered":"2024-04-25T16:32:31.240175+0000","last_clean":"2024-04-25T16:32:31.240175+0000","last_became_active":"2024-04-25T16:30:08.988218+0000","last_became_peered":"2024-04-25T16:30:08.988218+0000","last_unstale":"2024-04-25T16:32:31.240175+0000","last_undegraded":"2024-04-25T16:32:31.240175+0000","last_fullsized":"2024-04-25T16:32:31.240175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:57:57.046055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020493+0000","last_change":"2024-04-25T16:30:09.992498+0000","last_active":"2024-04-25T16:32:10.020493+0000","last_peered":"2024-04-25T16:32:10.020493+0000","last_clean":"2024-04-25T16:32:10.020493+0000","last_became_active":"2024-04-25T16:30:09.992190+0000","last_became_peered":"2024-04-25T16:30:09.992190+0000","last_unstale":"2024-04-25T16:32:10.020493+0000","last_undegraded":"2024-04-25T16:32:10.020493+0000","last_fullsized":"2024-04-25T16:32:10.020493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:34.407963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017060+0000","last_change":"2024-04-25T16:30:14.007371+0000","last_active":"2024-04-25T16:32:10.017060+0000","last_peered":"2024-04-25T16:32:10.017060+0000","last_clean":"2024-04-25T16:32:10.017060+0000","last_became_active":"2024-04-25T16:30:14.006811+0000","last_became_peered":"2024-04-25T16:30:14.006811+0000","last_unstale":"2024-04-25T16:32:10.017060+0000","last_undegraded":"2024-04-25T16:32:10.017060+0000","last_fullsized":"2024-04-25T16:32:10.017060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:01.473142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014900+0000","last_change":"2024-04-25T16:30:08.984080+0000","last_active":"2024-04-25T16:32:10.014900+0000","last_peered":"2024-04-25T16:32:10.014900+0000","last_clean":"2024-04-25T16:32:10.014900+0000","last_became_active":"2024-04-25T16:30:08.983702+0000","last_became_peered":"2024-04-25T16:30:08.983702+0000","last_unstale":"2024-04-25T16:32:10.014900+0000","last_undegraded":"2024-04-25T16:32:10.014900+0000","last_fullsized":"2024-04-25T16:32:10.014900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:22.311986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205200+0000","last_change":"2024-04-25T16:31:09.187653+0000","last_active":"2024-04-25T16:32:10.205200+0000","last_peered":"2024-04-25T16:32:10.205200+0000","last_clean":"2024-04-25T16:32:10.205200+0000","last_became_active":"2024-04-25T16:31:09.187183+0000","last_became_peered":"2024-04-25T16:31:09.187183+0000","last_unstale":"2024-04-25T16:32:10.205200+0000","last_undegraded":"2024-04-25T16:32:10.205200+0000","last_fullsized":"2024-04-25T16:32:10.205200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:44:35.157126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021131+0000","last_change":"2024-04-25T16:30:14.003218+0000","last_active":"2024-04-25T16:32:10.021131+0000","last_peered":"2024-04-25T16:32:10.021131+0000","last_clean":"2024-04-25T16:32:10.021131+0000","last_became_active":"2024-04-25T16:30:14.002782+0000","last_became_peered":"2024-04-25T16:30:14.002782+0000","last_unstale":"2024-04-25T16:32:10.021131+0000","last_undegraded":"2024-04-25T16:32:10.021131+0000","last_fullsized":"2024-04-25T16:32:10.021131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013630+0000","last_change":"2024-04-25T16:30:08.993447+0000","last_active":"2024-04-25T16:32:10.013630+0000","last_peered":"2024-04-25T16:32:10.013630+0000","last_clean":"2024-04-25T16:32:10.013630+0000","last_became_active":"2024-04-25T16:30:08.993108+0000","last_became_peered":"2024-04-25T16:30:08.993108+0000","last_unstale":"2024-04-25T16:32:10.013630+0000","last_undegraded":"2024-04-25T16:32:10.013630+0000","last_fullsized":"2024-04-25T16:32:10.013630+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:00.511392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016351+0000","last_change":"2024-04-25T16:30:09.989863+0000","last_active":"2024-04-25T16:32:10.016351+0000","last_peered":"2024-04-25T16:32:10.016351+0000","last_clean":"2024-04-25T16:32:10.016351+0000","last_became_active":"2024-04-25T16:30:09.989520+0000","last_became_peered":"2024-04-25T16:30:09.989520+0000","last_unstale":"2024-04-25T16:32:10.016351+0000","last_undegraded":"2024-04-25T16:32:10.016351+0000","last_fullsized":"2024-04-25T16:32:10.016351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:54.178298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781912+0000","last_change":"2024-04-25T16:30:13.999904+0000","last_active":"2024-04-25T16:32:10.781912+0000","last_peered":"2024-04-25T16:32:10.781912+0000","last_clean":"2024-04-25T16:32:10.781912+0000","last_became_active":"2024-04-25T16:30:13.999468+0000","last_became_peered":"2024-04-25T16:30:13.999468+0000","last_unstale":"2024-04-25T16:32:10.781912+0000","last_undegraded":"2024-04-25T16:32:10.781912+0000","last_fullsized":"2024-04-25T16:32:10.781912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:09.653365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204676+0000","last_change":"2024-04-25T16:30:08.987550+0000","last_active":"2024-04-25T16:32:10.204676+0000","last_peered":"2024-04-25T16:32:10.204676+0000","last_clean":"2024-04-25T16:32:10.204676+0000","last_became_active":"2024-04-25T16:30:08.987231+0000","last_became_peered":"2024-04-25T16:30:08.987231+0000","last_unstale":"2024-04-25T16:32:10.204676+0000","last_undegraded":"2024-04-25T16:32:10.204676+0000","last_fullsized":"2024-04-25T16:32:10.204676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:01.110960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017634+0000","last_change":"2024-04-25T16:30:09.991985+0000","last_active":"2024-04-25T16:32:10.017634+0000","last_peered":"2024-04-25T16:32:10.017634+0000","last_clean":"2024-04-25T16:32:10.017634+0000","last_became_active":"2024-04-25T16:30:09.991645+0000","last_became_peered":"2024-04-25T16:30:09.991645+0000","last_unstale":"2024-04-25T16:32:10.017634+0000","last_undegraded":"2024-04-25T16:32:10.017634+0000","last_fullsized":"2024-04-25T16:32:10.017634+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:30.419416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016667+0000","last_change":"2024-04-25T16:30:13.999931+0000","last_active":"2024-04-25T16:32:10.016667+0000","last_peered":"2024-04-25T16:32:10.016667+0000","last_clean":"2024-04-25T16:32:10.016667+0000","last_became_active":"2024-04-25T16:30:13.997093+0000","last_became_peered":"2024-04-25T16:30:13.997093+0000","last_unstale":"2024-04-25T16:32:10.016667+0000","last_undegraded":"2024-04-25T16:32:10.016667+0000","last_fullsized":"2024-04-25T16:32:10.016667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:31.973370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'7","reported_seq":264,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:30.837890+0000","last_change":"2024-04-25T16:30:08.985721+0000","last_active":"2024-04-25T16:32:30.837890+0000","last_peered":"2024-04-25T16:32:30.837890+0000","last_clean":"2024-04-25T16:32:30.837890+0000","last_became_active":"2024-04-25T16:30:08.985097+0000","last_became_peered":"2024-04-25T16:30:08.985097+0000","last_unstale":"2024-04-25T16:32:30.837890+0000","last_undegraded":"2024-04-25T16:32:30.837890+0000","last_fullsized":"2024-04-25T16:32:30.837890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:00.713027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.531050+0000","last_change":"2024-04-25T16:30:09.992659+0000","last_active":"2024-04-25T16:32:10.531050+0000","last_peered":"2024-04-25T16:32:10.531050+0000","last_clean":"2024-04-25T16:32:10.531050+0000","last_became_active":"2024-04-25T16:30:09.985936+0000","last_became_peered":"2024-04-25T16:30:09.985936+0000","last_unstale":"2024-04-25T16:32:10.531050+0000","last_undegraded":"2024-04-25T16:32:10.531050+0000","last_fullsized":"2024-04-25T16:32:10.531050+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:10.808082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017564+0000","last_change":"2024-04-25T16:30:14.009352+0000","last_active":"2024-04-25T16:32:10.017564+0000","last_peered":"2024-04-25T16:32:10.017564+0000","last_clean":"2024-04-25T16:32:10.017564+0000","last_became_active":"2024-04-25T16:30:14.005250+0000","last_became_peered":"2024-04-25T16:30:14.005250+0000","last_unstale":"2024-04-25T16:32:10.017564+0000","last_undegraded":"2024-04-25T16:32:10.017564+0000","last_fullsized":"2024-04-25T16:32:10.017564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:23.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022369+0000","last_change":"2024-04-25T16:30:08.990547+0000","last_active":"2024-04-25T16:32:10.022369+0000","last_peered":"2024-04-25T16:32:10.022369+0000","last_clean":"2024-04-25T16:32:10.022369+0000","last_became_active":"2024-04-25T16:30:08.989927+0000","last_became_peered":"2024-04-25T16:30:08.989927+0000","last_unstale":"2024-04-25T16:32:10.022369+0000","last_undegraded":"2024-04-25T16:32:10.022369+0000","last_fullsized":"2024-04-25T16:32:10.022369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:05.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781967+0000","last_change":"2024-04-25T16:30:10.002459+0000","last_active":"2024-04-25T16:32:10.781967+0000","last_peered":"2024-04-25T16:32:10.781967+0000","last_clean":"2024-04-25T16:32:10.781967+0000","last_became_active":"2024-04-25T16:30:10.002005+0000","last_became_peered":"2024-04-25T16:30:10.002005+0000","last_unstale":"2024-04-25T16:32:10.781967+0000","last_undegraded":"2024-04-25T16:32:10.781967+0000","last_fullsized":"2024-04-25T16:32:10.781967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:01.220371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212172+0000","last_change":"2024-04-25T16:31:09.186538+0000","last_active":"2024-04-25T16:32:10.212172+0000","last_peered":"2024-04-25T16:32:10.212172+0000","last_clean":"2024-04-25T16:32:10.212172+0000","last_became_active":"2024-04-25T16:31:09.185930+0000","last_became_peered":"2024-04-25T16:31:09.185930+0000","last_unstale":"2024-04-25T16:32:10.212172+0000","last_undegraded":"2024-04-25T16:32:10.212172+0000","last_fullsized":"2024-04-25T16:32:10.212172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:14:38.930206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205179+0000","last_change":"2024-04-25T16:30:08.985264+0000","last_active":"2024-04-25T16:32:10.205179+0000","last_peered":"2024-04-25T16:32:10.205179+0000","last_clean":"2024-04-25T16:32:10.205179+0000","last_became_active":"2024-04-25T16:30:08.984901+0000","last_became_peered":"2024-04-25T16:30:08.984901+0000","last_unstale":"2024-04-25T16:32:10.205179+0000","last_undegraded":"2024-04-25T16:32:10.205179+0000","last_fullsized":"2024-04-25T16:32:10.205179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:15.271981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530865+0000","last_change":"2024-04-25T16:30:09.997002+0000","last_active":"2024-04-25T16:32:10.530865+0000","last_peered":"2024-04-25T16:32:10.530865+0000","last_clean":"2024-04-25T16:32:10.530865+0000","last_became_active":"2024-04-25T16:30:09.996636+0000","last_became_peered":"2024-04-25T16:30:09.996636+0000","last_unstale":"2024-04-25T16:32:10.530865+0000","last_undegraded":"2024-04-25T16:32:10.530865+0000","last_fullsized":"2024-04-25T16:32:10.530865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:32.522050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017760+0000","last_change":"2024-04-25T16:30:14.006403+0000","last_active":"2024-04-25T16:32:10.017760+0000","last_peered":"2024-04-25T16:32:10.017760+0000","last_clean":"2024-04-25T16:32:10.017760+0000","last_became_active":"2024-04-25T16:30:14.004816+0000","last_became_peered":"2024-04-25T16:30:14.004816+0000","last_unstale":"2024-04-25T16:32:10.017760+0000","last_undegraded":"2024-04-25T16:32:10.017760+0000","last_fullsized":"2024-04-25T16:32:10.017760+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:51:10.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":305,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:30.846875+0000","last_change":"2024-04-25T16:30:08.986119+0000","last_active":"2024-04-25T16:32:30.846875+0000","last_peered":"2024-04-25T16:32:30.846875+0000","last_clean":"2024-04-25T16:32:30.846875+0000","last_became_active":"2024-04-25T16:30:08.985806+0000","last_became_peered":"2024-04-25T16:30:08.985806+0000","last_unstale":"2024-04-25T16:32:30.846875+0000","last_undegraded":"2024-04-25T16:32:30.846875+0000","last_fullsized":"2024-04-25T16:32:30.846875+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:17.125886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64098,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":233,"num_read_kb":179,"num_write":42,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021027+0000","last_change":"2024-04-25T16:30:09.996891+0000","last_active":"2024-04-25T16:32:10.021027+0000","last_peered":"2024-04-25T16:32:10.021027+0000","last_clean":"2024-04-25T16:32:10.021027+0000","last_became_active":"2024-04-25T16:30:09.995851+0000","last_became_peered":"2024-04-25T16:30:09.995851+0000","last_unstale":"2024-04-25T16:32:10.021027+0000","last_undegraded":"2024-04-25T16:32:10.021027+0000","last_fullsized":"2024-04-25T16:32:10.021027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:42.602854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205467+0000","last_change":"2024-04-25T16:30:13.998277+0000","last_active":"2024-04-25T16:32:10.205467+0000","last_peered":"2024-04-25T16:32:10.205467+0000","last_clean":"2024-04-25T16:32:10.205467+0000","last_became_active":"2024-04-25T16:30:13.997313+0000","last_became_peered":"2024-04-25T16:30:13.997313+0000","last_unstale":"2024-04-25T16:32:10.205467+0000","last_undegraded":"2024-04-25T16:32:10.205467+0000","last_fullsized":"2024-04-25T16:32:10.205467+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:56.379806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017045+0000","last_change":"2024-04-25T16:30:08.986377+0000","last_active":"2024-04-25T16:32:10.017045+0000","last_peered":"2024-04-25T16:32:10.017045+0000","last_clean":"2024-04-25T16:32:10.017045+0000","last_became_active":"2024-04-25T16:30:08.985892+0000","last_became_peered":"2024-04-25T16:30:08.985892+0000","last_unstale":"2024-04-25T16:32:10.017045+0000","last_undegraded":"2024-04-25T16:32:10.017045+0000","last_fullsized":"2024-04-25T16:32:10.017045+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:53.651789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013775+0000","last_change":"2024-04-25T16:31:09.010551+0000","last_active":"2024-04-25T16:32:10.013775+0000","last_peered":"2024-04-25T16:32:10.013775+0000","last_clean":"2024-04-25T16:32:10.013775+0000","last_became_active":"2024-04-25T16:31:09.010141+0000","last_became_peered":"2024-04-25T16:31:09.010141+0000","last_unstale":"2024-04-25T16:32:10.013775+0000","last_undegraded":"2024-04-25T16:32:10.013775+0000","last_fullsized":"2024-04-25T16:32:10.013775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:54.650080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022080+0000","last_change":"2024-04-25T16:30:14.006064+0000","last_active":"2024-04-25T16:32:10.022080+0000","last_peered":"2024-04-25T16:32:10.022080+0000","last_clean":"2024-04-25T16:32:10.022080+0000","last_became_active":"2024-04-25T16:30:14.005545+0000","last_became_peered":"2024-04-25T16:30:14.005545+0000","last_unstale":"2024-04-25T16:32:10.022080+0000","last_undegraded":"2024-04-25T16:32:10.022080+0000","last_fullsized":"2024-04-25T16:32:10.022080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:36.611008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205079+0000","last_change":"2024-04-25T16:30:08.988413+0000","last_active":"2024-04-25T16:32:10.205079+0000","last_peered":"2024-04-25T16:32:10.205079+0000","last_clean":"2024-04-25T16:32:10.205079+0000","last_became_active":"2024-04-25T16:30:08.988075+0000","last_became_peered":"2024-04-25T16:30:08.988075+0000","last_unstale":"2024-04-25T16:32:10.205079+0000","last_undegraded":"2024-04-25T16:32:10.205079+0000","last_fullsized":"2024-04-25T16:32:10.205079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:16.635876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022110+0000","last_change":"2024-04-25T16:30:09.997697+0000","last_active":"2024-04-25T16:32:10.022110+0000","last_peered":"2024-04-25T16:32:10.022110+0000","last_clean":"2024-04-25T16:32:10.022110+0000","last_became_active":"2024-04-25T16:30:09.997086+0000","last_became_peered":"2024-04-25T16:30:09.997086+0000","last_unstale":"2024-04-25T16:32:10.022110+0000","last_undegraded":"2024-04-25T16:32:10.022110+0000","last_fullsized":"2024-04-25T16:32:10.022110+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:35.345902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017958+0000","last_change":"2024-04-25T16:30:14.010486+0000","last_active":"2024-04-25T16:32:10.017958+0000","last_peered":"2024-04-25T16:32:10.017958+0000","last_clean":"2024-04-25T16:32:10.017958+0000","last_became_active":"2024-04-25T16:30:14.009273+0000","last_became_peered":"2024-04-25T16:30:14.009273+0000","last_unstale":"2024-04-25T16:32:10.017958+0000","last_undegraded":"2024-04-25T16:32:10.017958+0000","last_fullsized":"2024-04-25T16:32:10.017958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:11.863575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'8","reported_seq":159,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:32.245066+0000","last_change":"2024-04-25T16:30:08.986661+0000","last_active":"2024-04-25T16:32:32.245066+0000","last_peered":"2024-04-25T16:32:32.245066+0000","last_clean":"2024-04-25T16:32:32.245066+0000","last_became_active":"2024-04-25T16:30:08.986306+0000","last_became_peered":"2024-04-25T16:30:08.986306+0000","last_unstale":"2024-04-25T16:32:32.245066+0000","last_undegraded":"2024-04-25T16:32:32.245066+0000","last_fullsized":"2024-04-25T16:32:32.245066+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:44.363693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021934+0000","last_change":"2024-04-25T16:30:09.993443+0000","last_active":"2024-04-25T16:32:10.021934+0000","last_peered":"2024-04-25T16:32:10.021934+0000","last_clean":"2024-04-25T16:32:10.021934+0000","last_became_active":"2024-04-25T16:30:09.993031+0000","last_became_peered":"2024-04-25T16:30:09.993031+0000","last_unstale":"2024-04-25T16:32:10.021934+0000","last_undegraded":"2024-04-25T16:32:10.021934+0000","last_fullsized":"2024-04-25T16:32:10.021934+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:22.908713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014172+0000","last_change":"2024-04-25T16:30:14.003183+0000","last_active":"2024-04-25T16:32:10.014172+0000","last_peered":"2024-04-25T16:32:10.014172+0000","last_clean":"2024-04-25T16:32:10.014172+0000","last_became_active":"2024-04-25T16:30:14.002182+0000","last_became_peered":"2024-04-25T16:30:14.002182+0000","last_unstale":"2024-04-25T16:32:10.014172+0000","last_undegraded":"2024-04-25T16:32:10.014172+0000","last_fullsized":"2024-04-25T16:32:10.014172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:08.616556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020096+0000","last_change":"2024-04-25T16:30:08.995860+0000","last_active":"2024-04-25T16:32:10.020096+0000","last_peered":"2024-04-25T16:32:10.020096+0000","last_clean":"2024-04-25T16:32:10.020096+0000","last_became_active":"2024-04-25T16:30:08.995487+0000","last_became_peered":"2024-04-25T16:30:08.995487+0000","last_unstale":"2024-04-25T16:32:10.020096+0000","last_undegraded":"2024-04-25T16:32:10.020096+0000","last_fullsized":"2024-04-25T16:32:10.020096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:48.206502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204879+0000","last_change":"2024-04-25T16:30:09.998820+0000","last_active":"2024-04-25T16:32:10.204879+0000","last_peered":"2024-04-25T16:32:10.204879+0000","last_clean":"2024-04-25T16:32:10.204879+0000","last_became_active":"2024-04-25T16:30:09.998448+0000","last_became_peered":"2024-04-25T16:30:09.998448+0000","last_unstale":"2024-04-25T16:32:10.204879+0000","last_undegraded":"2024-04-25T16:32:10.204879+0000","last_fullsized":"2024-04-25T16:32:10.204879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:54.361250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204753+0000","last_change":"2024-04-25T16:30:13.992379+0000","last_active":"2024-04-25T16:32:10.204753+0000","last_peered":"2024-04-25T16:32:10.204753+0000","last_clean":"2024-04-25T16:32:10.204753+0000","last_became_active":"2024-04-25T16:30:13.991940+0000","last_became_peered":"2024-04-25T16:30:13.991940+0000","last_unstale":"2024-04-25T16:32:10.204753+0000","last_undegraded":"2024-04-25T16:32:10.204753+0000","last_fullsized":"2024-04-25T16:32:10.204753+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:59.152384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:27.942459+0000","last_change":"2024-04-25T16:30:08.987701+0000","last_active":"2024-04-25T16:32:27.942459+0000","last_peered":"2024-04-25T16:32:27.942459+0000","last_clean":"2024-04-25T16:32:27.942459+0000","last_became_active":"2024-04-25T16:30:08.987376+0000","last_became_peered":"2024-04-25T16:30:08.987376+0000","last_unstale":"2024-04-25T16:32:27.942459+0000","last_undegraded":"2024-04-25T16:32:27.942459+0000","last_fullsized":"2024-04-25T16:32:27.942459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:21.810100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021777+0000","last_change":"2024-04-25T16:30:09.993005+0000","last_active":"2024-04-25T16:32:10.021777+0000","last_peered":"2024-04-25T16:32:10.021777+0000","last_clean":"2024-04-25T16:32:10.021777+0000","last_became_active":"2024-04-25T16:30:09.992619+0000","last_became_peered":"2024-04-25T16:30:09.992619+0000","last_unstale":"2024-04-25T16:32:10.021777+0000","last_undegraded":"2024-04-25T16:32:10.021777+0000","last_fullsized":"2024-04-25T16:32:10.021777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:39:40.998241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020416+0000","last_change":"2024-04-25T16:30:14.147895+0000","last_active":"2024-04-25T16:32:10.020416+0000","last_peered":"2024-04-25T16:32:10.020416+0000","last_clean":"2024-04-25T16:32:10.020416+0000","last_became_active":"2024-04-25T16:30:14.147298+0000","last_became_peered":"2024-04-25T16:30:14.147298+0000","last_unstale":"2024-04-25T16:32:10.020416+0000","last_undegraded":"2024-04-25T16:32:10.020416+0000","last_fullsized":"2024-04-25T16:32:10.020416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:06.269587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018035+0000","last_change":"2024-04-25T16:30:08.982683+0000","last_active":"2024-04-25T16:32:10.018035+0000","last_peered":"2024-04-25T16:32:10.018035+0000","last_clean":"2024-04-25T16:32:10.018035+0000","last_became_active":"2024-04-25T16:30:08.982236+0000","last_became_peered":"2024-04-25T16:30:08.982236+0000","last_unstale":"2024-04-25T16:32:10.018035+0000","last_undegraded":"2024-04-25T16:32:10.018035+0000","last_fullsized":"2024-04-25T16:32:10.018035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:48.781791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.015356+0000","last_change":"2024-04-25T16:30:09.986734+0000","last_active":"2024-04-25T16:32:10.015356+0000","last_peered":"2024-04-25T16:32:10.015356+0000","last_clean":"2024-04-25T16:32:10.015356+0000","last_became_active":"2024-04-25T16:30:09.986337+0000","last_became_peered":"2024-04-25T16:30:09.986337+0000","last_unstale":"2024-04-25T16:32:10.015356+0000","last_undegraded":"2024-04-25T16:32:10.015356+0000","last_fullsized":"2024-04-25T16:32:10.015356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.678009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013878+0000","last_change":"2024-04-25T16:30:14.003324+0000","last_active":"2024-04-25T16:32:10.013878+0000","last_peered":"2024-04-25T16:32:10.013878+0000","last_clean":"2024-04-25T16:32:10.013878+0000","last_became_active":"2024-04-25T16:30:14.002904+0000","last_became_peered":"2024-04-25T16:30:14.002904+0000","last_unstale":"2024-04-25T16:32:10.013878+0000","last_undegraded":"2024-04-25T16:32:10.013878+0000","last_fullsized":"2024-04-25T16:32:10.013878+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:28.750102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'6","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.232485+0000","last_change":"2024-04-25T16:30:08.993469+0000","last_active":"2024-04-25T16:32:31.232485+0000","last_peered":"2024-04-25T16:32:31.232485+0000","last_clean":"2024-04-25T16:32:31.232485+0000","last_became_active":"2024-04-25T16:30:08.993126+0000","last_became_peered":"2024-04-25T16:30:08.993126+0000","last_unstale":"2024-04-25T16:32:31.232485+0000","last_undegraded":"2024-04-25T16:32:31.232485+0000","last_fullsized":"2024-04-25T16:32:31.232485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:10.274645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014980+0000","last_change":"2024-04-25T16:30:10.000597+0000","last_active":"2024-04-25T16:32:10.014980+0000","last_peered":"2024-04-25T16:32:10.014980+0000","last_clean":"2024-04-25T16:32:10.014980+0000","last_became_active":"2024-04-25T16:30:09.998892+0000","last_became_peered":"2024-04-25T16:30:09.998892+0000","last_unstale":"2024-04-25T16:32:10.014980+0000","last_undegraded":"2024-04-25T16:32:10.014980+0000","last_fullsized":"2024-04-25T16:32:10.014980+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:09.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013930+0000","last_change":"2024-04-25T16:30:14.147240+0000","last_active":"2024-04-25T16:32:10.013930+0000","last_peered":"2024-04-25T16:32:10.013930+0000","last_clean":"2024-04-25T16:32:10.013930+0000","last_became_active":"2024-04-25T16:30:14.146793+0000","last_became_peered":"2024-04-25T16:30:14.146793+0000","last_unstale":"2024-04-25T16:32:10.013930+0000","last_undegraded":"2024-04-25T16:32:10.013930+0000","last_fullsized":"2024-04-25T16:32:10.013930+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:55.713386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204864+0000","last_change":"2024-04-25T16:30:09.994595+0000","last_active":"2024-04-25T16:32:10.204864+0000","last_peered":"2024-04-25T16:32:10.204864+0000","last_clean":"2024-04-25T16:32:10.204864+0000","last_became_active":"2024-04-25T16:30:09.994305+0000","last_became_peered":"2024-04-25T16:30:09.994305+0000","last_unstale":"2024-04-25T16:32:10.204864+0000","last_undegraded":"2024-04-25T16:32:10.204864+0000","last_fullsized":"2024-04-25T16:32:10.204864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:27.839500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'7","reported_seq":156,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:32.244380+0000","last_change":"2024-04-25T16:30:08.984203+0000","last_active":"2024-04-25T16:32:32.244380+0000","last_peered":"2024-04-25T16:32:32.244380+0000","last_clean":"2024-04-25T16:32:32.244380+0000","last_became_active":"2024-04-25T16:30:08.982883+0000","last_became_peered":"2024-04-25T16:30:08.982883+0000","last_unstale":"2024-04-25T16:32:32.244380+0000","last_undegraded":"2024-04-25T16:32:32.244380+0000","last_fullsized":"2024-04-25T16:32:32.244380+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:18.447624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022086+0000","last_change":"2024-04-25T16:31:09.017505+0000","last_active":"2024-04-25T16:32:10.022086+0000","last_peered":"2024-04-25T16:32:10.022086+0000","last_clean":"2024-04-25T16:32:10.022086+0000","last_became_active":"2024-04-25T16:31:09.011812+0000","last_became_peered":"2024-04-25T16:31:09.011812+0000","last_unstale":"2024-04-25T16:32:10.022086+0000","last_undegraded":"2024-04-25T16:32:10.022086+0000","last_fullsized":"2024-04-25T16:32:10.022086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:02.404908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016285+0000","last_change":"2024-04-25T16:30:09.990998+0000","last_active":"2024-04-25T16:32:10.016285+0000","last_peered":"2024-04-25T16:32:10.016285+0000","last_clean":"2024-04-25T16:32:10.016285+0000","last_became_active":"2024-04-25T16:30:09.990644+0000","last_became_peered":"2024-04-25T16:30:09.990644+0000","last_unstale":"2024-04-25T16:32:10.016285+0000","last_undegraded":"2024-04-25T16:32:10.016285+0000","last_fullsized":"2024-04-25T16:32:10.016285+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:42.175520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212092+0000","last_change":"2024-04-25T16:30:08.988315+0000","last_active":"2024-04-25T16:32:10.212092+0000","last_peered":"2024-04-25T16:32:10.212092+0000","last_clean":"2024-04-25T16:32:10.212092+0000","last_became_active":"2024-04-25T16:30:08.987930+0000","last_became_peered":"2024-04-25T16:30:08.987930+0000","last_unstale":"2024-04-25T16:32:10.212092+0000","last_undegraded":"2024-04-25T16:32:10.212092+0000","last_fullsized":"2024-04-25T16:32:10.212092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:44.884602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":106,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021552+0000","last_change":"2024-04-25T16:28:58.219510+0000","last_active":"2024-04-25T16:32:10.021552+0000","last_peered":"2024-04-25T16:32:10.021552+0000","last_clean":"2024-04-25T16:32:10.021552+0000","last_became_active":"2024-04-25T16:28:58.213888+0000","last_became_peered":"2024-04-25T16:28:58.213888+0000","last_unstale":"2024-04-25T16:32:10.021552+0000","last_undegraded":"2024-04-25T16:32:10.021552+0000","last_fullsized":"2024-04-25T16:32:10.021552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_clean_scrub_stamp":"2024-04-25T16:25:33.939186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:15.378406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205344+0000","last_change":"2024-04-25T16:31:09.012831+0000","last_active":"2024-04-25T16:32:10.205344+0000","last_peered":"2024-04-25T16:32:10.205344+0000","last_clean":"2024-04-25T16:32:10.205344+0000","last_became_active":"2024-04-25T16:31:09.012375+0000","last_became_peered":"2024-04-25T16:31:09.012375+0000","last_unstale":"2024-04-25T16:32:10.205344+0000","last_undegraded":"2024-04-25T16:32:10.205344+0000","last_fullsized":"2024-04-25T16:32:10.205344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:02.214734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013551+0000","last_change":"2024-04-25T16:30:10.005210+0000","last_active":"2024-04-25T16:32:10.013551+0000","last_peered":"2024-04-25T16:32:10.013551+0000","last_clean":"2024-04-25T16:32:10.013551+0000","last_became_active":"2024-04-25T16:30:10.004936+0000","last_became_peered":"2024-04-25T16:30:10.004936+0000","last_unstale":"2024-04-25T16:32:10.013551+0000","last_undegraded":"2024-04-25T16:32:10.013551+0000","last_fullsized":"2024-04-25T16:32:10.013551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:53.292675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021680+0000","last_change":"2024-04-25T16:30:08.990162+0000","last_active":"2024-04-25T16:32:10.021680+0000","last_peered":"2024-04-25T16:32:10.021680+0000","last_clean":"2024-04-25T16:32:10.021680+0000","last_became_active":"2024-04-25T16:30:08.989613+0000","last_became_peered":"2024-04-25T16:30:08.989613+0000","last_unstale":"2024-04-25T16:32:10.021680+0000","last_undegraded":"2024-04-25T16:32:10.021680+0000","last_fullsized":"2024-04-25T16:32:10.021680+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:41.595381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021251+0000","last_change":"2024-04-25T16:30:14.001850+0000","last_active":"2024-04-25T16:32:10.021251+0000","last_peered":"2024-04-25T16:32:10.021251+0000","last_clean":"2024-04-25T16:32:10.021251+0000","last_became_active":"2024-04-25T16:30:14.001348+0000","last_became_peered":"2024-04-25T16:30:14.001348+0000","last_unstale":"2024-04-25T16:32:10.021251+0000","last_undegraded":"2024-04-25T16:32:10.021251+0000","last_fullsized":"2024-04-25T16:32:10.021251+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:56:41.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021095+0000","last_change":"2024-04-25T16:30:08.995833+0000","last_active":"2024-04-25T16:32:10.021095+0000","last_peered":"2024-04-25T16:32:10.021095+0000","last_clean":"2024-04-25T16:32:10.021095+0000","last_became_active":"2024-04-25T16:30:08.995545+0000","last_became_peered":"2024-04-25T16:30:08.995545+0000","last_unstale":"2024-04-25T16:32:10.021095+0000","last_undegraded":"2024-04-25T16:32:10.021095+0000","last_fullsized":"2024-04-25T16:32:10.021095+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:31.452219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014096+0000","last_change":"2024-04-25T16:30:09.990225+0000","last_active":"2024-04-25T16:32:10.014096+0000","last_peered":"2024-04-25T16:32:10.014096+0000","last_clean":"2024-04-25T16:32:10.014096+0000","last_became_active":"2024-04-25T16:30:09.989859+0000","last_became_peered":"2024-04-25T16:30:09.989859+0000","last_unstale":"2024-04-25T16:32:10.014096+0000","last_undegraded":"2024-04-25T16:32:10.014096+0000","last_fullsized":"2024-04-25T16:32:10.014096+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:40.589395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016177+0000","last_change":"2024-04-25T16:30:14.147223+0000","last_active":"2024-04-25T16:32:10.016177+0000","last_peered":"2024-04-25T16:32:10.016177+0000","last_clean":"2024-04-25T16:32:10.016177+0000","last_became_active":"2024-04-25T16:30:14.146821+0000","last_became_peered":"2024-04-25T16:30:14.146821+0000","last_unstale":"2024-04-25T16:32:10.016177+0000","last_undegraded":"2024-04-25T16:32:10.016177+0000","last_fullsized":"2024-04-25T16:32:10.016177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:01.289165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'16","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:33.364360+0000","last_change":"2024-04-25T16:32:10.029459+0000","last_active":"2024-04-25T16:32:33.364360+0000","last_peered":"2024-04-25T16:32:33.364360+0000","last_clean":"2024-04-25T16:32:33.364360+0000","last_became_active":"2024-04-25T16:32:10.022574+0000","last_became_peered":"2024-04-25T16:32:10.022574+0000","last_unstale":"2024-04-25T16:32:33.364360+0000","last_undegraded":"2024-04-25T16:32:33.364360+0000","last_fullsized":"2024-04-25T16:32:33.364360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:41.675443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016438+0000","last_change":"2024-04-25T16:30:09.986088+0000","last_active":"2024-04-25T16:32:10.016438+0000","last_peered":"2024-04-25T16:32:10.016438+0000","last_clean":"2024-04-25T16:32:10.016438+0000","last_became_active":"2024-04-25T16:30:09.985256+0000","last_became_peered":"2024-04-25T16:30:09.985256+0000","last_unstale":"2024-04-25T16:32:10.016438+0000","last_undegraded":"2024-04-25T16:32:10.016438+0000","last_fullsized":"2024-04-25T16:32:10.016438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:29.197558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016928+0000","last_change":"2024-04-25T16:30:14.000699+0000","last_active":"2024-04-25T16:32:10.016928+0000","last_peered":"2024-04-25T16:32:10.016928+0000","last_clean":"2024-04-25T16:32:10.016928+0000","last_became_active":"2024-04-25T16:30:14.000255+0000","last_became_peered":"2024-04-25T16:30:14.000255+0000","last_unstale":"2024-04-25T16:32:10.016928+0000","last_undegraded":"2024-04-25T16:32:10.016928+0000","last_fullsized":"2024-04-25T16:32:10.016928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:33.436693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212550+0000","last_change":"2024-04-25T16:30:08.981926+0000","last_active":"2024-04-25T16:32:10.212550+0000","last_peered":"2024-04-25T16:32:10.212550+0000","last_clean":"2024-04-25T16:32:10.212550+0000","last_became_active":"2024-04-25T16:30:08.981590+0000","last_became_peered":"2024-04-25T16:30:08.981590+0000","last_unstale":"2024-04-25T16:32:10.212550+0000","last_undegraded":"2024-04-25T16:32:10.212550+0000","last_fullsized":"2024-04-25T16:32:10.212550+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:57.818774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781964+0000","last_change":"2024-04-25T16:30:10.003998+0000","last_active":"2024-04-25T16:32:10.781964+0000","last_peered":"2024-04-25T16:32:10.781964+0000","last_clean":"2024-04-25T16:32:10.781964+0000","last_became_active":"2024-04-25T16:30:10.003626+0000","last_became_peered":"2024-04-25T16:30:10.003626+0000","last_unstale":"2024-04-25T16:32:10.781964+0000","last_undegraded":"2024-04-25T16:32:10.781964+0000","last_fullsized":"2024-04-25T16:32:10.781964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:10.686092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212401+0000","last_change":"2024-04-25T16:30:14.149278+0000","last_active":"2024-04-25T16:32:10.212401+0000","last_peered":"2024-04-25T16:32:10.212401+0000","last_clean":"2024-04-25T16:32:10.212401+0000","last_became_active":"2024-04-25T16:30:14.148654+0000","last_became_peered":"2024-04-25T16:30:14.148654+0000","last_unstale":"2024-04-25T16:32:10.212401+0000","last_undegraded":"2024-04-25T16:32:10.212401+0000","last_fullsized":"2024-04-25T16:32:10.212401+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:24.823222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530250+0000","last_change":"2024-04-25T16:30:08.981612+0000","last_active":"2024-04-25T16:32:10.530250+0000","last_peered":"2024-04-25T16:32:10.530250+0000","last_clean":"2024-04-25T16:32:10.530250+0000","last_became_active":"2024-04-25T16:30:08.981222+0000","last_became_peered":"2024-04-25T16:30:08.981222+0000","last_unstale":"2024-04-25T16:32:10.530250+0000","last_undegraded":"2024-04-25T16:32:10.530250+0000","last_fullsized":"2024-04-25T16:32:10.530250+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:14.914658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.019805+0000","last_change":"2024-04-25T16:30:09.989592+0000","last_active":"2024-04-25T16:32:10.019805+0000","last_peered":"2024-04-25T16:32:10.019805+0000","last_clean":"2024-04-25T16:32:10.019805+0000","last_became_active":"2024-04-25T16:30:09.989222+0000","last_became_peered":"2024-04-25T16:30:09.989222+0000","last_unstale":"2024-04-25T16:32:10.019805+0000","last_undegraded":"2024-04-25T16:32:10.019805+0000","last_fullsized":"2024-04-25T16:32:10.019805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:52.403239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018128+0000","last_change":"2024-04-25T16:30:14.010480+0000","last_active":"2024-04-25T16:32:10.018128+0000","last_peered":"2024-04-25T16:32:10.018128+0000","last_clean":"2024-04-25T16:32:10.018128+0000","last_became_active":"2024-04-25T16:30:14.009023+0000","last_became_peered":"2024-04-25T16:30:14.009023+0000","last_unstale":"2024-04-25T16:32:10.018128+0000","last_undegraded":"2024-04-25T16:32:10.018128+0000","last_fullsized":"2024-04-25T16:32:10.018128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:10.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017158+0000","last_change":"2024-04-25T16:30:08.986849+0000","last_active":"2024-04-25T16:32:10.017158+0000","last_peered":"2024-04-25T16:32:10.017158+0000","last_clean":"2024-04-25T16:32:10.017158+0000","last_became_active":"2024-04-25T16:30:08.986479+0000","last_became_peered":"2024-04-25T16:30:08.986479+0000","last_unstale":"2024-04-25T16:32:10.017158+0000","last_undegraded":"2024-04-25T16:32:10.017158+0000","last_fullsized":"2024-04-25T16:32:10.017158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022094+0000","last_change":"2024-04-25T16:30:09.990927+0000","last_active":"2024-04-25T16:32:10.022094+0000","last_peered":"2024-04-25T16:32:10.022094+0000","last_clean":"2024-04-25T16:32:10.022094+0000","last_became_active":"2024-04-25T16:30:09.990482+0000","last_became_peered":"2024-04-25T16:30:09.990482+0000","last_unstale":"2024-04-25T16:32:10.022094+0000","last_undegraded":"2024-04-25T16:32:10.022094+0000","last_fullsized":"2024-04-25T16:32:10.022094+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:18.649721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212337+0000","last_change":"2024-04-25T16:30:14.006159+0000","last_active":"2024-04-25T16:32:10.212337+0000","last_peered":"2024-04-25T16:32:10.212337+0000","last_clean":"2024-04-25T16:32:10.212337+0000","last_became_active":"2024-04-25T16:30:14.005413+0000","last_became_peered":"2024-04-25T16:30:14.005413+0000","last_unstale":"2024-04-25T16:32:10.212337+0000","last_undegraded":"2024-04-25T16:32:10.212337+0000","last_fullsized":"2024-04-25T16:32:10.212337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:52.934385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.026829+0000","last_change":"2024-04-25T16:30:08.986207+0000","last_active":"2024-04-25T16:32:31.026829+0000","last_peered":"2024-04-25T16:32:31.026829+0000","last_clean":"2024-04-25T16:32:31.026829+0000","last_became_active":"2024-04-25T16:30:08.985857+0000","last_became_peered":"2024-04-25T16:30:08.985857+0000","last_unstale":"2024-04-25T16:32:31.026829+0000","last_undegraded":"2024-04-25T16:32:31.026829+0000","last_fullsized":"2024-04-25T16:32:31.026829+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:45.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016835+0000","last_change":"2024-04-25T16:30:09.993566+0000","last_active":"2024-04-25T16:32:10.016835+0000","last_peered":"2024-04-25T16:32:10.016835+0000","last_clean":"2024-04-25T16:32:10.016835+0000","last_became_active":"2024-04-25T16:30:09.992250+0000","last_became_peered":"2024-04-25T16:30:09.992250+0000","last_unstale":"2024-04-25T16:32:10.016835+0000","last_undegraded":"2024-04-25T16:32:10.016835+0000","last_fullsized":"2024-04-25T16:32:10.016835+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:33.279300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212478+0000","last_change":"2024-04-25T16:30:14.147443+0000","last_active":"2024-04-25T16:32:10.212478+0000","last_peered":"2024-04-25T16:32:10.212478+0000","last_clean":"2024-04-25T16:32:10.212478+0000","last_became_active":"2024-04-25T16:30:14.146838+0000","last_became_peered":"2024-04-25T16:30:14.146838+0000","last_unstale":"2024-04-25T16:32:10.212478+0000","last_undegraded":"2024-04-25T16:32:10.212478+0000","last_fullsized":"2024-04-25T16:32:10.212478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:51.532685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020627+0000","last_change":"2024-04-25T16:31:09.015526+0000","last_active":"2024-04-25T16:32:10.020627+0000","last_peered":"2024-04-25T16:32:10.020627+0000","last_clean":"2024-04-25T16:32:10.020627+0000","last_became_active":"2024-04-25T16:31:09.007972+0000","last_became_peered":"2024-04-25T16:31:09.007972+0000","last_unstale":"2024-04-25T16:32:10.020627+0000","last_undegraded":"2024-04-25T16:32:10.020627+0000","last_fullsized":"2024-04-25T16:32:10.020627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:56.464426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022010+0000","last_change":"2024-04-25T16:30:09.998486+0000","last_active":"2024-04-25T16:32:10.022010+0000","last_peered":"2024-04-25T16:32:10.022010+0000","last_clean":"2024-04-25T16:32:10.022010+0000","last_became_active":"2024-04-25T16:30:09.998154+0000","last_became_peered":"2024-04-25T16:30:09.998154+0000","last_unstale":"2024-04-25T16:32:10.022010+0000","last_undegraded":"2024-04-25T16:32:10.022010+0000","last_fullsized":"2024-04-25T16:32:10.022010+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:55.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530492+0000","last_change":"2024-04-25T16:30:13.995864+0000","last_active":"2024-04-25T16:32:10.530492+0000","last_peered":"2024-04-25T16:32:10.530492+0000","last_clean":"2024-04-25T16:32:10.530492+0000","last_became_active":"2024-04-25T16:30:13.995326+0000","last_became_peered":"2024-04-25T16:30:13.995326+0000","last_unstale":"2024-04-25T16:32:10.530492+0000","last_undegraded":"2024-04-25T16:32:10.530492+0000","last_fullsized":"2024-04-25T16:32:10.530492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:11.718826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":156,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.265799+0000","last_change":"2024-04-25T16:30:08.984189+0000","last_active":"2024-04-25T16:32:31.265799+0000","last_peered":"2024-04-25T16:32:31.265799+0000","last_clean":"2024-04-25T16:32:31.265799+0000","last_became_active":"2024-04-25T16:30:08.983821+0000","last_became_peered":"2024-04-25T16:30:08.983821+0000","last_unstale":"2024-04-25T16:32:31.265799+0000","last_undegraded":"2024-04-25T16:32:31.265799+0000","last_fullsized":"2024-04-25T16:32:31.265799+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:51.315397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022285+0000","last_change":"2024-04-25T16:30:09.998763+0000","last_active":"2024-04-25T16:32:10.022285+0000","last_peered":"2024-04-25T16:32:10.022285+0000","last_clean":"2024-04-25T16:32:10.022285+0000","last_became_active":"2024-04-25T16:30:09.998439+0000","last_became_peered":"2024-04-25T16:30:09.998439+0000","last_unstale":"2024-04-25T16:32:10.022285+0000","last_undegraded":"2024-04-25T16:32:10.022285+0000","last_fullsized":"2024-04-25T16:32:10.022285+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:14.279883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781635+0000","last_change":"2024-04-25T16:30:14.005427+0000","last_active":"2024-04-25T16:32:10.781635+0000","last_peered":"2024-04-25T16:32:10.781635+0000","last_clean":"2024-04-25T16:32:10.781635+0000","last_became_active":"2024-04-25T16:30:14.005006+0000","last_became_peered":"2024-04-25T16:30:14.005006+0000","last_unstale":"2024-04-25T16:32:10.781635+0000","last_undegraded":"2024-04-25T16:32:10.781635+0000","last_fullsized":"2024-04-25T16:32:10.781635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:49.563365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205387+0000","last_change":"2024-04-25T16:30:08.985557+0000","last_active":"2024-04-25T16:32:10.205387+0000","last_peered":"2024-04-25T16:32:10.205387+0000","last_clean":"2024-04-25T16:32:10.205387+0000","last_became_active":"2024-04-25T16:30:08.985215+0000","last_became_peered":"2024-04-25T16:30:08.985215+0000","last_unstale":"2024-04-25T16:32:10.205387+0000","last_undegraded":"2024-04-25T16:32:10.205387+0000","last_fullsized":"2024-04-25T16:32:10.205387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:51.504089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022076+0000","last_change":"2024-04-25T16:30:09.991336+0000","last_active":"2024-04-25T16:32:10.022076+0000","last_peered":"2024-04-25T16:32:10.022076+0000","last_clean":"2024-04-25T16:32:10.022076+0000","last_became_active":"2024-04-25T16:30:09.990819+0000","last_became_peered":"2024-04-25T16:30:09.990819+0000","last_unstale":"2024-04-25T16:32:10.022076+0000","last_undegraded":"2024-04-25T16:32:10.022076+0000","last_fullsized":"2024-04-25T16:32:10.022076+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:45.988061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204595+0000","last_change":"2024-04-25T16:30:13.998463+0000","last_active":"2024-04-25T16:32:10.204595+0000","last_peered":"2024-04-25T16:32:10.204595+0000","last_clean":"2024-04-25T16:32:10.204595+0000","last_became_active":"2024-04-25T16:30:13.997952+0000","last_became_peered":"2024-04-25T16:30:13.997952+0000","last_unstale":"2024-04-25T16:32:10.204595+0000","last_undegraded":"2024-04-25T16:32:10.204595+0000","last_fullsized":"2024-04-25T16:32:10.204595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:58.507942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.536390+0000","last_change":"2024-04-25T16:32:10.535950+0000","last_active":"2024-04-25T16:32:10.536390+0000","last_peered":"2024-04-25T16:32:10.536390+0000","last_clean":"2024-04-25T16:32:10.536390+0000","last_became_active":"2024-04-25T16:32:10.531823+0000","last_became_peered":"2024-04-25T16:32:10.531823+0000","last_unstale":"2024-04-25T16:32:10.536390+0000","last_undegraded":"2024-04-25T16:32:10.536390+0000","last_fullsized":"2024-04-25T16:32:10.536390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:36.471975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022474+0000","last_change":"2024-04-25T16:30:09.988566+0000","last_active":"2024-04-25T16:32:10.022474+0000","last_peered":"2024-04-25T16:32:10.022474+0000","last_clean":"2024-04-25T16:32:10.022474+0000","last_became_active":"2024-04-25T16:30:09.988164+0000","last_became_peered":"2024-04-25T16:30:09.988164+0000","last_unstale":"2024-04-25T16:32:10.022474+0000","last_undegraded":"2024-04-25T16:32:10.022474+0000","last_fullsized":"2024-04-25T16:32:10.022474+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:40.476766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022056+0000","last_change":"2024-04-25T16:30:09.996349+0000","last_active":"2024-04-25T16:32:10.022056+0000","last_peered":"2024-04-25T16:32:10.022056+0000","last_clean":"2024-04-25T16:32:10.022056+0000","last_became_active":"2024-04-25T16:30:09.994974+0000","last_became_peered":"2024-04-25T16:30:09.994974+0000","last_unstale":"2024-04-25T16:32:10.022056+0000","last_undegraded":"2024-04-25T16:32:10.022056+0000","last_fullsized":"2024-04-25T16:32:10.022056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:46.929057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017091+0000","last_change":"2024-04-25T16:30:09.997837+0000","last_active":"2024-04-25T16:32:10.017091+0000","last_peered":"2024-04-25T16:32:10.017091+0000","last_clean":"2024-04-25T16:32:10.017091+0000","last_became_active":"2024-04-25T16:30:09.997361+0000","last_became_peered":"2024-04-25T16:30:09.997361+0000","last_unstale":"2024-04-25T16:32:10.017091+0000","last_undegraded":"2024-04-25T16:32:10.017091+0000","last_fullsized":"2024-04-25T16:32:10.017091+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:35.698528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016888+0000","last_change":"2024-04-25T16:30:09.991794+0000","last_active":"2024-04-25T16:32:10.016888+0000","last_peered":"2024-04-25T16:32:10.016888+0000","last_clean":"2024-04-25T16:32:10.016888+0000","last_became_active":"2024-04-25T16:30:09.991440+0000","last_became_peered":"2024-04-25T16:30:09.991440+0000","last_unstale":"2024-04-25T16:32:10.016888+0000","last_undegraded":"2024-04-25T16:32:10.016888+0000","last_fullsized":"2024-04-25T16:32:10.016888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:11.322254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781869+0000","last_change":"2024-04-25T16:30:10.002364+0000","last_active":"2024-04-25T16:32:10.781869+0000","last_peered":"2024-04-25T16:32:10.781869+0000","last_clean":"2024-04-25T16:32:10.781869+0000","last_became_active":"2024-04-25T16:30:10.001957+0000","last_became_peered":"2024-04-25T16:30:10.001957+0000","last_unstale":"2024-04-25T16:32:10.781869+0000","last_undegraded":"2024-04-25T16:32:10.781869+0000","last_fullsized":"2024-04-25T16:32:10.781869+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:19.150175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014045+0000","last_change":"2024-04-25T16:30:10.004150+0000","last_active":"2024-04-25T16:32:10.014045+0000","last_peered":"2024-04-25T16:32:10.014045+0000","last_clean":"2024-04-25T16:32:10.014045+0000","last_became_active":"2024-04-25T16:30:10.003733+0000","last_became_peered":"2024-04-25T16:30:10.003733+0000","last_unstale":"2024-04-25T16:32:10.014045+0000","last_undegraded":"2024-04-25T16:32:10.014045+0000","last_fullsized":"2024-04-25T16:32:10.014045+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:11.850366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021171+0000","last_change":"2024-04-25T16:30:09.991056+0000","last_active":"2024-04-25T16:32:10.021171+0000","last_peered":"2024-04-25T16:32:10.021171+0000","last_clean":"2024-04-25T16:32:10.021171+0000","last_became_active":"2024-04-25T16:30:09.988708+0000","last_became_peered":"2024-04-25T16:30:09.988708+0000","last_unstale":"2024-04-25T16:32:10.021171+0000","last_undegraded":"2024-04-25T16:32:10.021171+0000","last_fullsized":"2024-04-25T16:32:10.021171+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:13.062269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017612+0000","last_change":"2024-04-25T16:30:09.996571+0000","last_active":"2024-04-25T16:32:10.017612+0000","last_peered":"2024-04-25T16:32:10.017612+0000","last_clean":"2024-04-25T16:32:10.017612+0000","last_became_active":"2024-04-25T16:30:09.996206+0000","last_became_peered":"2024-04-25T16:30:09.996206+0000","last_unstale":"2024-04-25T16:32:10.017612+0000","last_undegraded":"2024-04-25T16:32:10.017612+0000","last_fullsized":"2024-04-25T16:32:10.017612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:07.107949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016603+0000","last_change":"2024-04-25T16:30:09.995067+0000","last_active":"2024-04-25T16:32:10.016603+0000","last_peered":"2024-04-25T16:32:10.016603+0000","last_clean":"2024-04-25T16:32:10.016603+0000","last_became_active":"2024-04-25T16:30:09.994678+0000","last_became_peered":"2024-04-25T16:30:09.994678+0000","last_unstale":"2024-04-25T16:32:10.016603+0000","last_undegraded":"2024-04-25T16:32:10.016603+0000","last_fullsized":"2024-04-25T16:32:10.016603+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:09.000869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204798+0000","last_change":"2024-04-25T16:30:09.996821+0000","last_active":"2024-04-25T16:32:10.204798+0000","last_peered":"2024-04-25T16:32:10.204798+0000","last_clean":"2024-04-25T16:32:10.204798+0000","last_became_active":"2024-04-25T16:30:09.996464+0000","last_became_peered":"2024-04-25T16:30:09.996464+0000","last_unstale":"2024-04-25T16:32:10.204798+0000","last_undegraded":"2024-04-25T16:32:10.204798+0000","last_fullsized":"2024-04-25T16:32:10.204798+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:54.961544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017023+0000","last_change":"2024-04-25T16:30:09.991733+0000","last_active":"2024-04-25T16:32:10.017023+0000","last_peered":"2024-04-25T16:32:10.017023+0000","last_clean":"2024-04-25T16:32:10.017023+0000","last_became_active":"2024-04-25T16:30:09.991401+0000","last_became_peered":"2024-04-25T16:30:09.991401+0000","last_unstale":"2024-04-25T16:32:10.017023+0000","last_undegraded":"2024-04-25T16:32:10.017023+0000","last_fullsized":"2024-04-25T16:32:10.017023+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:14.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530708+0000","last_change":"2024-04-25T16:30:09.999087+0000","last_active":"2024-04-25T16:32:10.530708+0000","last_peered":"2024-04-25T16:32:10.530708+0000","last_clean":"2024-04-25T16:32:10.530708+0000","last_became_active":"2024-04-25T16:30:09.998725+0000","last_became_peered":"2024-04-25T16:30:09.998725+0000","last_unstale":"2024-04-25T16:32:10.530708+0000","last_undegraded":"2024-04-25T16:32:10.530708+0000","last_fullsized":"2024-04-25T16:32:10.530708+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:19.155171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013940+0000","last_change":"2024-04-25T16:30:09.995880+0000","last_active":"2024-04-25T16:32:10.013940+0000","last_peered":"2024-04-25T16:32:10.013940+0000","last_clean":"2024-04-25T16:32:10.013940+0000","last_became_active":"2024-04-25T16:30:09.995514+0000","last_became_peered":"2024-04-25T16:30:09.995514+0000","last_unstale":"2024-04-25T16:32:10.013940+0000","last_undegraded":"2024-04-25T16:32:10.013940+0000","last_fullsized":"2024-04-25T16:32:10.013940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022174+0000","last_change":"2024-04-25T16:30:09.998326+0000","last_active":"2024-04-25T16:32:10.022174+0000","last_peered":"2024-04-25T16:32:10.022174+0000","last_clean":"2024-04-25T16:32:10.022174+0000","last_became_active":"2024-04-25T16:30:09.995803+0000","last_became_peered":"2024-04-25T16:30:09.995803+0000","last_unstale":"2024-04-25T16:32:10.022174+0000","last_undegraded":"2024-04-25T16:32:10.022174+0000","last_fullsized":"2024-04-25T16:32:10.022174+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:46.983947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73556,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1186,"num_read_kb":672,"num_write":273,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":225764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645354,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":475742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841583,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1671168,"data_stored":1130298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037812,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234041,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":475557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1074206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":471176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":536304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":473023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":537549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":470391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":475782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1064174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:32:37.273 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 -- 172.21.15.98:0/259043080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e01029c0 msgr2=0x7fc9e01036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 --2- 172.21.15.98:0/259043080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e01029c0 0x7fc9e01036c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0009900 tx=0x7fc9d002a410 comp rx=0 tx=0).stop 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 -- 172.21.15.98:0/259043080 shutdown_connections 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 --2- 172.21.15.98:0/259043080 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e0108900 0x7fc9e010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 --2- 172.21.15.98:0/259043080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e0103c00 0x7fc9e0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 --2- 172.21.15.98:0/259043080 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e01029c0 0x7fc9e01036c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 -- 172.21.15.98:0/259043080 >> 172.21.15.98:0/259043080 conn(0x7fc9e00fdbd0 msgr2=0x7fc9e0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 -- 172.21.15.98:0/259043080 shutdown_connections 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.416+0000 7fc9e5884640 1 -- 172.21.15.98:0/259043080 wait complete. 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9e5884640 1 Processor -- start 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9e5884640 1 -- start start 2024-04-25T16:32:37.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9e5884640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e01029c0 0x7fc9e006a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9e5884640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e0103c00 0x7fc9e019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9e5884640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e0108900 0x7fc9e01a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9e5884640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e010e370 con 0x7fc9e0108900 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9e5884640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e010e1f0 con 0x7fc9e01029c0 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9e5884640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc9e010e4f0 con 0x7fc9e0103c00 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e0108900 0x7fc9e01a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9df7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e0108900 0x7fc9e01a1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37124/0 (socket says 172.21.15.98:37124) 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9df7fe640 1 -- 172.21.15.98:0/890310485 learned_addr learned my addr 172.21.15.98:0/890310485 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.417+0000 7fc9deffd640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e01029c0 0x7fc9e006a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9d7fff640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e0103c00 0x7fc9e019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9df7fe640 1 -- 172.21.15.98:0/890310485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e0103c00 msgr2=0x7fc9e019ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9df7fe640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e0103c00 0x7fc9e019ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9df7fe640 1 -- 172.21.15.98:0/890310485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e01029c0 msgr2=0x7fc9e006a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9df7fe640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e01029c0 0x7fc9e006a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9df7fe640 1 -- 172.21.15.98:0/890310485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9e01a1c80 con 0x7fc9e0108900 2024-04-25T16:32:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9df7fe640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e0108900 0x7fc9e01a1460 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800cad0 tx=0x7fc9d800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9e4882640 1 -- 172.21.15.98:0/890310485 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d800ae20 con 0x7fc9e0108900 2024-04-25T16:32:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9e4882640 1 -- 172.21.15.98:0/890310485 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc9d8005020 con 0x7fc9e0108900 2024-04-25T16:32:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9e01a1f70 con 0x7fc9e0108900 2024-04-25T16:32:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9e4882640 1 -- 172.21.15.98:0/890310485 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9d800f6b0 con 0x7fc9e0108900 2024-04-25T16:32:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.418+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9e01a23d0 con 0x7fc9e0108900 2024-04-25T16:32:37.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.419+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9e0107710 con 0x7fc9e0108900 2024-04-25T16:32:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.421+0000 7fc9e4882640 1 -- 172.21.15.98:0/890310485 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fc9d80051d0 con 0x7fc9e0108900 2024-04-25T16:32:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.422+0000 7fc9e4882640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9cc0afb80 0x7fc9cc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.422+0000 7fc9deffd640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9cc0afb80 0x7fc9cc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.422+0000 7fc9e4882640 1 -- 172.21.15.98:0/890310485 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fc9d80df3a0 con 0x7fc9e0108900 2024-04-25T16:32:37.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.422+0000 7fc9deffd640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9cc0afb80 0x7fc9cc0b2090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0005760 tx=0x7fc9d0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:37.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.425+0000 7fc9e4882640 1 -- 172.21.15.98:0/890310485 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9d8014030 con 0x7fc9e0108900 2024-04-25T16:32:37.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:37 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1326802175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:37.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:37 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3236520517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:37.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:37 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4169282731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:37 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1326802175' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:37 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3236520517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:37.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:37 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4169282731' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:37.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.601+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc9e0062fe0 con 0x7fc9cc0afb80 2024-04-25T16:32:37.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.610+0000 7fc9e4882640 1 -- 172.21.15.98:0/890310485 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340525 (secure 0 0 0) 0x7fc9e0062fe0 con 0x7fc9cc0afb80 2024-04-25T16:32:37.613 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:37.615 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T16:32:37.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9cc0afb80 msgr2=0x7fc9cc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9cc0afb80 0x7fc9cc0b2090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc9d0005760 tx=0x7fc9d0000bc0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e0108900 msgr2=0x7fc9e01a1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e0108900 0x7fc9e01a1460 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc9d800cad0 tx=0x7fc9d800cf90 comp rx=0 tx=0).stop 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 shutdown_connections 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc9cc0afb80 0x7fc9cc0b2090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc9e0108900 0x7fc9e01a1460 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc9e0103c00 0x7fc9e019ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 --2- 172.21.15.98:0/890310485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9e01029c0 0x7fc9e006a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.613+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 >> 172.21.15.98:0/890310485 conn(0x7fc9e00fdbd0 msgr2=0x7fc9e01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.614+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 shutdown_connections 2024-04-25T16:32:37.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.614+0000 7fc9e5884640 1 -- 172.21.15.98:0/890310485 wait complete. 2024-04-25T16:32:37.624 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-04-25T16:32:35.964692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664158,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1262,"num_read_kb":736,"num_write":398,"num_write_kb":2316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2951957,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":829,"ondisk_log_size":829,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":395960,"kb_used_data":13520,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124545864,"statfs":{"total":1151940427776,"available":1151534964736,"internally_reserved":0,"allocated":13844480,"data_stored":7663477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2465,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":109,"num_read_kb":59,"num_write":22,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003530"},"pg_stats":[{"pgid":"3.3f","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021251+0000","last_change":"2024-04-25T16:30:09.990700+0000","last_active":"2024-04-25T16:32:10.021251+0000","last_peered":"2024-04-25T16:32:10.021251+0000","last_clean":"2024-04-25T16:32:10.021251+0000","last_became_active":"2024-04-25T16:30:09.990225+0000","last_became_peered":"2024-04-25T16:30:09.990225+0000","last_unstale":"2024-04-25T16:32:10.021251+0000","last_undegraded":"2024-04-25T16:32:10.021251+0000","last_fullsized":"2024-04-25T16:32:10.021251+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:33.897277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.782026+0000","last_change":"2024-04-25T16:30:09.989618+0000","last_active":"2024-04-25T16:32:10.782026+0000","last_peered":"2024-04-25T16:32:10.782026+0000","last_clean":"2024-04-25T16:32:10.782026+0000","last_became_active":"2024-04-25T16:30:09.989270+0000","last_became_peered":"2024-04-25T16:30:09.989270+0000","last_unstale":"2024-04-25T16:32:10.782026+0000","last_undegraded":"2024-04-25T16:32:10.782026+0000","last_fullsized":"2024-04-25T16:32:10.782026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:03.952233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781848+0000","last_change":"2024-04-25T16:30:09.992442+0000","last_active":"2024-04-25T16:32:10.781848+0000","last_peered":"2024-04-25T16:32:10.781848+0000","last_clean":"2024-04-25T16:32:10.781848+0000","last_became_active":"2024-04-25T16:30:09.992115+0000","last_became_peered":"2024-04-25T16:30:09.992115+0000","last_unstale":"2024-04-25T16:32:10.781848+0000","last_undegraded":"2024-04-25T16:32:10.781848+0000","last_fullsized":"2024-04-25T16:32:10.781848+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:00.668427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017939+0000","last_change":"2024-04-25T16:31:09.006824+0000","last_active":"2024-04-25T16:32:10.017939+0000","last_peered":"2024-04-25T16:32:10.017939+0000","last_clean":"2024-04-25T16:32:10.017939+0000","last_became_active":"2024-04-25T16:31:09.005560+0000","last_became_peered":"2024-04-25T16:31:09.005560+0000","last_unstale":"2024-04-25T16:32:10.017939+0000","last_undegraded":"2024-04-25T16:32:10.017939+0000","last_fullsized":"2024-04-25T16:32:10.017939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:41.979642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020683+0000","last_change":"2024-04-25T16:30:09.991372+0000","last_active":"2024-04-25T16:32:10.020683+0000","last_peered":"2024-04-25T16:32:10.020683+0000","last_clean":"2024-04-25T16:32:10.020683+0000","last_became_active":"2024-04-25T16:30:09.991053+0000","last_became_peered":"2024-04-25T16:30:09.991053+0000","last_unstale":"2024-04-25T16:32:10.020683+0000","last_undegraded":"2024-04-25T16:32:10.020683+0000","last_fullsized":"2024-04-25T16:32:10.020683+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:28.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014900+0000","last_change":"2024-04-25T16:30:09.997563+0000","last_active":"2024-04-25T16:32:10.014900+0000","last_peered":"2024-04-25T16:32:10.014900+0000","last_clean":"2024-04-25T16:32:10.014900+0000","last_became_active":"2024-04-25T16:30:09.997158+0000","last_became_peered":"2024-04-25T16:30:09.997158+0000","last_unstale":"2024-04-25T16:32:10.014900+0000","last_undegraded":"2024-04-25T16:32:10.014900+0000","last_fullsized":"2024-04-25T16:32:10.014900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:21.441957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021712+0000","last_change":"2024-04-25T16:31:09.009888+0000","last_active":"2024-04-25T16:32:10.021712+0000","last_peered":"2024-04-25T16:32:10.021712+0000","last_clean":"2024-04-25T16:32:10.021712+0000","last_became_active":"2024-04-25T16:31:09.009201+0000","last_became_peered":"2024-04-25T16:31:09.009201+0000","last_unstale":"2024-04-25T16:32:10.021712+0000","last_undegraded":"2024-04-25T16:32:10.021712+0000","last_fullsized":"2024-04-25T16:32:10.021712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:52.171410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016068+0000","last_change":"2024-04-25T16:30:09.994820+0000","last_active":"2024-04-25T16:32:10.016068+0000","last_peered":"2024-04-25T16:32:10.016068+0000","last_clean":"2024-04-25T16:32:10.016068+0000","last_became_active":"2024-04-25T16:30:09.994494+0000","last_became_peered":"2024-04-25T16:30:09.994494+0000","last_unstale":"2024-04-25T16:32:10.016068+0000","last_undegraded":"2024-04-25T16:32:10.016068+0000","last_fullsized":"2024-04-25T16:32:10.016068+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:39.373266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530879+0000","last_change":"2024-04-25T16:30:09.994870+0000","last_active":"2024-04-25T16:32:10.530879+0000","last_peered":"2024-04-25T16:32:10.530879+0000","last_clean":"2024-04-25T16:32:10.530879+0000","last_became_active":"2024-04-25T16:30:09.994521+0000","last_became_peered":"2024-04-25T16:30:09.994521+0000","last_unstale":"2024-04-25T16:32:10.530879+0000","last_undegraded":"2024-04-25T16:32:10.530879+0000","last_fullsized":"2024-04-25T16:32:10.530879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:18.093904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.015411+0000","last_change":"2024-04-25T16:31:09.010148+0000","last_active":"2024-04-25T16:32:10.015411+0000","last_peered":"2024-04-25T16:32:10.015411+0000","last_clean":"2024-04-25T16:32:10.015411+0000","last_became_active":"2024-04-25T16:31:09.009655+0000","last_became_peered":"2024-04-25T16:31:09.009655+0000","last_unstale":"2024-04-25T16:32:10.015411+0000","last_undegraded":"2024-04-25T16:32:10.015411+0000","last_fullsized":"2024-04-25T16:32:10.015411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:18.637202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022213+0000","last_change":"2024-04-25T16:30:09.999519+0000","last_active":"2024-04-25T16:32:10.022213+0000","last_peered":"2024-04-25T16:32:10.022213+0000","last_clean":"2024-04-25T16:32:10.022213+0000","last_became_active":"2024-04-25T16:30:09.999158+0000","last_became_peered":"2024-04-25T16:30:09.999158+0000","last_unstale":"2024-04-25T16:32:10.022213+0000","last_undegraded":"2024-04-25T16:32:10.022213+0000","last_fullsized":"2024-04-25T16:32:10.022213+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:38:06.686541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212538+0000","last_change":"2024-04-25T16:30:10.005455+0000","last_active":"2024-04-25T16:32:10.212538+0000","last_peered":"2024-04-25T16:32:10.212538+0000","last_clean":"2024-04-25T16:32:10.212538+0000","last_became_active":"2024-04-25T16:30:10.004891+0000","last_became_peered":"2024-04-25T16:30:10.004891+0000","last_unstale":"2024-04-25T16:32:10.212538+0000","last_undegraded":"2024-04-25T16:32:10.212538+0000","last_fullsized":"2024-04-25T16:32:10.212538+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.598321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014090+0000","last_change":"2024-04-25T16:30:09.992870+0000","last_active":"2024-04-25T16:32:10.014090+0000","last_peered":"2024-04-25T16:32:10.014090+0000","last_clean":"2024-04-25T16:32:10.014090+0000","last_became_active":"2024-04-25T16:30:09.992399+0000","last_became_peered":"2024-04-25T16:30:09.992399+0000","last_unstale":"2024-04-25T16:32:10.014090+0000","last_undegraded":"2024-04-25T16:32:10.014090+0000","last_fullsized":"2024-04-25T16:32:10.014090+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.127287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781631+0000","last_change":"2024-04-25T16:30:09.988707+0000","last_active":"2024-04-25T16:32:10.781631+0000","last_peered":"2024-04-25T16:32:10.781631+0000","last_clean":"2024-04-25T16:32:10.781631+0000","last_became_active":"2024-04-25T16:30:09.988386+0000","last_became_peered":"2024-04-25T16:30:09.988386+0000","last_unstale":"2024-04-25T16:32:10.781631+0000","last_undegraded":"2024-04-25T16:32:10.781631+0000","last_fullsized":"2024-04-25T16:32:10.781631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:03:15.973591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781712+0000","last_change":"2024-04-25T16:30:10.002780+0000","last_active":"2024-04-25T16:32:10.781712+0000","last_peered":"2024-04-25T16:32:10.781712+0000","last_clean":"2024-04-25T16:32:10.781712+0000","last_became_active":"2024-04-25T16:30:10.002055+0000","last_became_peered":"2024-04-25T16:30:10.002055+0000","last_unstale":"2024-04-25T16:32:10.781712+0000","last_undegraded":"2024-04-25T16:32:10.781712+0000","last_fullsized":"2024-04-25T16:32:10.781712+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:23.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530394+0000","last_change":"2024-04-25T16:30:09.998270+0000","last_active":"2024-04-25T16:32:10.530394+0000","last_peered":"2024-04-25T16:32:10.530394+0000","last_clean":"2024-04-25T16:32:10.530394+0000","last_became_active":"2024-04-25T16:30:09.997903+0000","last_became_peered":"2024-04-25T16:30:09.997903+0000","last_unstale":"2024-04-25T16:32:10.530394+0000","last_undegraded":"2024-04-25T16:32:10.530394+0000","last_fullsized":"2024-04-25T16:32:10.530394+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:14.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021350+0000","last_change":"2024-04-25T16:30:09.988894+0000","last_active":"2024-04-25T16:32:10.021350+0000","last_peered":"2024-04-25T16:32:10.021350+0000","last_clean":"2024-04-25T16:32:10.021350+0000","last_became_active":"2024-04-25T16:30:09.988508+0000","last_became_peered":"2024-04-25T16:30:09.988508+0000","last_unstale":"2024-04-25T16:32:10.021350+0000","last_undegraded":"2024-04-25T16:32:10.021350+0000","last_fullsized":"2024-04-25T16:32:10.021350+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:48.226193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018080+0000","last_change":"2024-04-25T16:30:09.989910+0000","last_active":"2024-04-25T16:32:10.018080+0000","last_peered":"2024-04-25T16:32:10.018080+0000","last_clean":"2024-04-25T16:32:10.018080+0000","last_became_active":"2024-04-25T16:30:09.989560+0000","last_became_peered":"2024-04-25T16:30:09.989560+0000","last_unstale":"2024-04-25T16:32:10.018080+0000","last_undegraded":"2024-04-25T16:32:10.018080+0000","last_fullsized":"2024-04-25T16:32:10.018080+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:12.834034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017567+0000","last_change":"2024-04-25T16:30:09.987930+0000","last_active":"2024-04-25T16:32:10.017567+0000","last_peered":"2024-04-25T16:32:10.017567+0000","last_clean":"2024-04-25T16:32:10.017567+0000","last_became_active":"2024-04-25T16:30:09.987595+0000","last_became_peered":"2024-04-25T16:30:09.987595+0000","last_unstale":"2024-04-25T16:32:10.017567+0000","last_undegraded":"2024-04-25T16:32:10.017567+0000","last_fullsized":"2024-04-25T16:32:10.017567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:04.778172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016791+0000","last_change":"2024-04-25T16:30:14.006360+0000","last_active":"2024-04-25T16:32:10.016791+0000","last_peered":"2024-04-25T16:32:10.016791+0000","last_clean":"2024-04-25T16:32:10.016791+0000","last_became_active":"2024-04-25T16:30:14.005894+0000","last_became_peered":"2024-04-25T16:30:14.005894+0000","last_unstale":"2024-04-25T16:32:10.016791+0000","last_undegraded":"2024-04-25T16:32:10.016791+0000","last_fullsized":"2024-04-25T16:32:10.016791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:33.150381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'14","reported_seq":164,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:33.234677+0000","last_change":"2024-04-25T16:30:08.992779+0000","last_active":"2024-04-25T16:32:33.234677+0000","last_peered":"2024-04-25T16:32:33.234677+0000","last_clean":"2024-04-25T16:32:33.234677+0000","last_became_active":"2024-04-25T16:30:08.992458+0000","last_became_peered":"2024-04-25T16:30:08.992458+0000","last_unstale":"2024-04-25T16:32:33.234677+0000","last_undegraded":"2024-04-25T16:32:33.234677+0000","last_fullsized":"2024-04-25T16:32:33.234677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:21.430855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016381+0000","last_change":"2024-04-25T16:30:09.995353+0000","last_active":"2024-04-25T16:32:10.016381+0000","last_peered":"2024-04-25T16:32:10.016381+0000","last_clean":"2024-04-25T16:32:10.016381+0000","last_became_active":"2024-04-25T16:30:09.995011+0000","last_became_peered":"2024-04-25T16:30:09.995011+0000","last_unstale":"2024-04-25T16:32:10.016381+0000","last_undegraded":"2024-04-25T16:32:10.016381+0000","last_fullsized":"2024-04-25T16:32:10.016381+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:33.367268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211820+0000","last_change":"2024-04-25T16:31:09.187252+0000","last_active":"2024-04-25T16:32:10.211820+0000","last_peered":"2024-04-25T16:32:10.211820+0000","last_clean":"2024-04-25T16:32:10.211820+0000","last_became_active":"2024-04-25T16:31:09.186689+0000","last_became_peered":"2024-04-25T16:31:09.186689+0000","last_unstale":"2024-04-25T16:32:10.211820+0000","last_undegraded":"2024-04-25T16:32:10.211820+0000","last_fullsized":"2024-04-25T16:32:10.211820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:18.287476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016628+0000","last_change":"2024-04-25T16:30:08.997663+0000","last_active":"2024-04-25T16:32:10.016628+0000","last_peered":"2024-04-25T16:32:10.016628+0000","last_clean":"2024-04-25T16:32:10.016628+0000","last_became_active":"2024-04-25T16:30:08.997107+0000","last_became_peered":"2024-04-25T16:30:08.997107+0000","last_unstale":"2024-04-25T16:32:10.016628+0000","last_undegraded":"2024-04-25T16:32:10.016628+0000","last_fullsized":"2024-04-25T16:32:10.016628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:25.856556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781991+0000","last_change":"2024-04-25T16:30:09.987947+0000","last_active":"2024-04-25T16:32:10.781991+0000","last_peered":"2024-04-25T16:32:10.781991+0000","last_clean":"2024-04-25T16:32:10.781991+0000","last_became_active":"2024-04-25T16:30:09.987575+0000","last_became_peered":"2024-04-25T16:30:09.987575+0000","last_unstale":"2024-04-25T16:32:10.781991+0000","last_undegraded":"2024-04-25T16:32:10.781991+0000","last_fullsized":"2024-04-25T16:32:10.781991+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:28.443249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211996+0000","last_change":"2024-04-25T16:30:14.006711+0000","last_active":"2024-04-25T16:32:10.211996+0000","last_peered":"2024-04-25T16:32:10.211996+0000","last_clean":"2024-04-25T16:32:10.211996+0000","last_became_active":"2024-04-25T16:30:14.005500+0000","last_became_peered":"2024-04-25T16:30:14.005500+0000","last_unstale":"2024-04-25T16:32:10.211996+0000","last_undegraded":"2024-04-25T16:32:10.211996+0000","last_fullsized":"2024-04-25T16:32:10.211996+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:58:26.288410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.034584+0000","last_change":"2024-04-25T16:32:10.034065+0000","last_active":"2024-04-25T16:32:10.034584+0000","last_peered":"2024-04-25T16:32:10.034584+0000","last_clean":"2024-04-25T16:32:10.034584+0000","last_became_active":"2024-04-25T16:32:10.022888+0000","last_became_peered":"2024-04-25T16:32:10.022888+0000","last_unstale":"2024-04-25T16:32:10.034584+0000","last_undegraded":"2024-04-25T16:32:10.034584+0000","last_fullsized":"2024-04-25T16:32:10.034584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:58:44.139220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530964+0000","last_change":"2024-04-25T16:30:09.988216+0000","last_active":"2024-04-25T16:32:10.530964+0000","last_peered":"2024-04-25T16:32:10.530964+0000","last_clean":"2024-04-25T16:32:10.530964+0000","last_became_active":"2024-04-25T16:30:09.987752+0000","last_became_peered":"2024-04-25T16:30:09.987752+0000","last_unstale":"2024-04-25T16:32:10.530964+0000","last_undegraded":"2024-04-25T16:32:10.530964+0000","last_fullsized":"2024-04-25T16:32:10.530964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:57:35.583000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211839+0000","last_change":"2024-04-25T16:30:14.001329+0000","last_active":"2024-04-25T16:32:10.211839+0000","last_peered":"2024-04-25T16:32:10.211839+0000","last_clean":"2024-04-25T16:32:10.211839+0000","last_became_active":"2024-04-25T16:30:14.000878+0000","last_became_peered":"2024-04-25T16:30:14.000878+0000","last_unstale":"2024-04-25T16:32:10.211839+0000","last_undegraded":"2024-04-25T16:32:10.211839+0000","last_fullsized":"2024-04-25T16:32:10.211839+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:07.946611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020332+0000","last_change":"2024-04-25T16:30:08.989065+0000","last_active":"2024-04-25T16:32:10.020332+0000","last_peered":"2024-04-25T16:32:10.020332+0000","last_clean":"2024-04-25T16:32:10.020332+0000","last_became_active":"2024-04-25T16:30:08.988709+0000","last_became_peered":"2024-04-25T16:30:08.988709+0000","last_unstale":"2024-04-25T16:32:10.020332+0000","last_undegraded":"2024-04-25T16:32:10.020332+0000","last_fullsized":"2024-04-25T16:32:10.020332+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:34.306581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205194+0000","last_change":"2024-04-25T16:30:09.999641+0000","last_active":"2024-04-25T16:32:10.205194+0000","last_peered":"2024-04-25T16:32:10.205194+0000","last_clean":"2024-04-25T16:32:10.205194+0000","last_became_active":"2024-04-25T16:30:09.999319+0000","last_became_peered":"2024-04-25T16:30:09.999319+0000","last_unstale":"2024-04-25T16:32:10.205194+0000","last_undegraded":"2024-04-25T16:32:10.205194+0000","last_fullsized":"2024-04-25T16:32:10.205194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:14.043201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017299+0000","last_change":"2024-04-25T16:30:13.999609+0000","last_active":"2024-04-25T16:32:10.017299+0000","last_peered":"2024-04-25T16:32:10.017299+0000","last_clean":"2024-04-25T16:32:10.017299+0000","last_became_active":"2024-04-25T16:30:13.999172+0000","last_became_peered":"2024-04-25T16:30:13.999172+0000","last_unstale":"2024-04-25T16:32:10.017299+0000","last_undegraded":"2024-04-25T16:32:10.017299+0000","last_fullsized":"2024-04-25T16:32:10.017299+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:49.878153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'8","reported_seq":158,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.240175+0000","last_change":"2024-04-25T16:30:08.988919+0000","last_active":"2024-04-25T16:32:31.240175+0000","last_peered":"2024-04-25T16:32:31.240175+0000","last_clean":"2024-04-25T16:32:31.240175+0000","last_became_active":"2024-04-25T16:30:08.988218+0000","last_became_peered":"2024-04-25T16:30:08.988218+0000","last_unstale":"2024-04-25T16:32:31.240175+0000","last_undegraded":"2024-04-25T16:32:31.240175+0000","last_fullsized":"2024-04-25T16:32:31.240175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:57:57.046055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020493+0000","last_change":"2024-04-25T16:30:09.992498+0000","last_active":"2024-04-25T16:32:10.020493+0000","last_peered":"2024-04-25T16:32:10.020493+0000","last_clean":"2024-04-25T16:32:10.020493+0000","last_became_active":"2024-04-25T16:30:09.992190+0000","last_became_peered":"2024-04-25T16:30:09.992190+0000","last_unstale":"2024-04-25T16:32:10.020493+0000","last_undegraded":"2024-04-25T16:32:10.020493+0000","last_fullsized":"2024-04-25T16:32:10.020493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:34.407963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017060+0000","last_change":"2024-04-25T16:30:14.007371+0000","last_active":"2024-04-25T16:32:10.017060+0000","last_peered":"2024-04-25T16:32:10.017060+0000","last_clean":"2024-04-25T16:32:10.017060+0000","last_became_active":"2024-04-25T16:30:14.006811+0000","last_became_peered":"2024-04-25T16:30:14.006811+0000","last_unstale":"2024-04-25T16:32:10.017060+0000","last_undegraded":"2024-04-25T16:32:10.017060+0000","last_fullsized":"2024-04-25T16:32:10.017060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:01.473142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014900+0000","last_change":"2024-04-25T16:30:08.984080+0000","last_active":"2024-04-25T16:32:10.014900+0000","last_peered":"2024-04-25T16:32:10.014900+0000","last_clean":"2024-04-25T16:32:10.014900+0000","last_became_active":"2024-04-25T16:30:08.983702+0000","last_became_peered":"2024-04-25T16:30:08.983702+0000","last_unstale":"2024-04-25T16:32:10.014900+0000","last_undegraded":"2024-04-25T16:32:10.014900+0000","last_fullsized":"2024-04-25T16:32:10.014900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:22.311986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205200+0000","last_change":"2024-04-25T16:31:09.187653+0000","last_active":"2024-04-25T16:32:10.205200+0000","last_peered":"2024-04-25T16:32:10.205200+0000","last_clean":"2024-04-25T16:32:10.205200+0000","last_became_active":"2024-04-25T16:31:09.187183+0000","last_became_peered":"2024-04-25T16:31:09.187183+0000","last_unstale":"2024-04-25T16:32:10.205200+0000","last_undegraded":"2024-04-25T16:32:10.205200+0000","last_fullsized":"2024-04-25T16:32:10.205200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:44:35.157126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021131+0000","last_change":"2024-04-25T16:30:14.003218+0000","last_active":"2024-04-25T16:32:10.021131+0000","last_peered":"2024-04-25T16:32:10.021131+0000","last_clean":"2024-04-25T16:32:10.021131+0000","last_became_active":"2024-04-25T16:30:14.002782+0000","last_became_peered":"2024-04-25T16:30:14.002782+0000","last_unstale":"2024-04-25T16:32:10.021131+0000","last_undegraded":"2024-04-25T16:32:10.021131+0000","last_fullsized":"2024-04-25T16:32:10.021131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013630+0000","last_change":"2024-04-25T16:30:08.993447+0000","last_active":"2024-04-25T16:32:10.013630+0000","last_peered":"2024-04-25T16:32:10.013630+0000","last_clean":"2024-04-25T16:32:10.013630+0000","last_became_active":"2024-04-25T16:30:08.993108+0000","last_became_peered":"2024-04-25T16:30:08.993108+0000","last_unstale":"2024-04-25T16:32:10.013630+0000","last_undegraded":"2024-04-25T16:32:10.013630+0000","last_fullsized":"2024-04-25T16:32:10.013630+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:00.511392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016351+0000","last_change":"2024-04-25T16:30:09.989863+0000","last_active":"2024-04-25T16:32:10.016351+0000","last_peered":"2024-04-25T16:32:10.016351+0000","last_clean":"2024-04-25T16:32:10.016351+0000","last_became_active":"2024-04-25T16:30:09.989520+0000","last_became_peered":"2024-04-25T16:30:09.989520+0000","last_unstale":"2024-04-25T16:32:10.016351+0000","last_undegraded":"2024-04-25T16:32:10.016351+0000","last_fullsized":"2024-04-25T16:32:10.016351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:54.178298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781912+0000","last_change":"2024-04-25T16:30:13.999904+0000","last_active":"2024-04-25T16:32:10.781912+0000","last_peered":"2024-04-25T16:32:10.781912+0000","last_clean":"2024-04-25T16:32:10.781912+0000","last_became_active":"2024-04-25T16:30:13.999468+0000","last_became_peered":"2024-04-25T16:30:13.999468+0000","last_unstale":"2024-04-25T16:32:10.781912+0000","last_undegraded":"2024-04-25T16:32:10.781912+0000","last_fullsized":"2024-04-25T16:32:10.781912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:09.653365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204676+0000","last_change":"2024-04-25T16:30:08.987550+0000","last_active":"2024-04-25T16:32:10.204676+0000","last_peered":"2024-04-25T16:32:10.204676+0000","last_clean":"2024-04-25T16:32:10.204676+0000","last_became_active":"2024-04-25T16:30:08.987231+0000","last_became_peered":"2024-04-25T16:30:08.987231+0000","last_unstale":"2024-04-25T16:32:10.204676+0000","last_undegraded":"2024-04-25T16:32:10.204676+0000","last_fullsized":"2024-04-25T16:32:10.204676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:01.110960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017634+0000","last_change":"2024-04-25T16:30:09.991985+0000","last_active":"2024-04-25T16:32:10.017634+0000","last_peered":"2024-04-25T16:32:10.017634+0000","last_clean":"2024-04-25T16:32:10.017634+0000","last_became_active":"2024-04-25T16:30:09.991645+0000","last_became_peered":"2024-04-25T16:30:09.991645+0000","last_unstale":"2024-04-25T16:32:10.017634+0000","last_undegraded":"2024-04-25T16:32:10.017634+0000","last_fullsized":"2024-04-25T16:32:10.017634+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:30.419416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016667+0000","last_change":"2024-04-25T16:30:13.999931+0000","last_active":"2024-04-25T16:32:10.016667+0000","last_peered":"2024-04-25T16:32:10.016667+0000","last_clean":"2024-04-25T16:32:10.016667+0000","last_became_active":"2024-04-25T16:30:13.997093+0000","last_became_peered":"2024-04-25T16:30:13.997093+0000","last_unstale":"2024-04-25T16:32:10.016667+0000","last_undegraded":"2024-04-25T16:32:10.016667+0000","last_fullsized":"2024-04-25T16:32:10.016667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:31.973370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'7","reported_seq":264,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:30.837890+0000","last_change":"2024-04-25T16:30:08.985721+0000","last_active":"2024-04-25T16:32:30.837890+0000","last_peered":"2024-04-25T16:32:30.837890+0000","last_clean":"2024-04-25T16:32:30.837890+0000","last_became_active":"2024-04-25T16:30:08.985097+0000","last_became_peered":"2024-04-25T16:30:08.985097+0000","last_unstale":"2024-04-25T16:32:30.837890+0000","last_undegraded":"2024-04-25T16:32:30.837890+0000","last_fullsized":"2024-04-25T16:32:30.837890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:00.713027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.531050+0000","last_change":"2024-04-25T16:30:09.992659+0000","last_active":"2024-04-25T16:32:10.531050+0000","last_peered":"2024-04-25T16:32:10.531050+0000","last_clean":"2024-04-25T16:32:10.531050+0000","last_became_active":"2024-04-25T16:30:09.985936+0000","last_became_peered":"2024-04-25T16:30:09.985936+0000","last_unstale":"2024-04-25T16:32:10.531050+0000","last_undegraded":"2024-04-25T16:32:10.531050+0000","last_fullsized":"2024-04-25T16:32:10.531050+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:10.808082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017564+0000","last_change":"2024-04-25T16:30:14.009352+0000","last_active":"2024-04-25T16:32:10.017564+0000","last_peered":"2024-04-25T16:32:10.017564+0000","last_clean":"2024-04-25T16:32:10.017564+0000","last_became_active":"2024-04-25T16:30:14.005250+0000","last_became_peered":"2024-04-25T16:30:14.005250+0000","last_unstale":"2024-04-25T16:32:10.017564+0000","last_undegraded":"2024-04-25T16:32:10.017564+0000","last_fullsized":"2024-04-25T16:32:10.017564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:23.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022369+0000","last_change":"2024-04-25T16:30:08.990547+0000","last_active":"2024-04-25T16:32:10.022369+0000","last_peered":"2024-04-25T16:32:10.022369+0000","last_clean":"2024-04-25T16:32:10.022369+0000","last_became_active":"2024-04-25T16:30:08.989927+0000","last_became_peered":"2024-04-25T16:30:08.989927+0000","last_unstale":"2024-04-25T16:32:10.022369+0000","last_undegraded":"2024-04-25T16:32:10.022369+0000","last_fullsized":"2024-04-25T16:32:10.022369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:05.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781967+0000","last_change":"2024-04-25T16:30:10.002459+0000","last_active":"2024-04-25T16:32:10.781967+0000","last_peered":"2024-04-25T16:32:10.781967+0000","last_clean":"2024-04-25T16:32:10.781967+0000","last_became_active":"2024-04-25T16:30:10.002005+0000","last_became_peered":"2024-04-25T16:30:10.002005+0000","last_unstale":"2024-04-25T16:32:10.781967+0000","last_undegraded":"2024-04-25T16:32:10.781967+0000","last_fullsized":"2024-04-25T16:32:10.781967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:01.220371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212172+0000","last_change":"2024-04-25T16:31:09.186538+0000","last_active":"2024-04-25T16:32:10.212172+0000","last_peered":"2024-04-25T16:32:10.212172+0000","last_clean":"2024-04-25T16:32:10.212172+0000","last_became_active":"2024-04-25T16:31:09.185930+0000","last_became_peered":"2024-04-25T16:31:09.185930+0000","last_unstale":"2024-04-25T16:32:10.212172+0000","last_undegraded":"2024-04-25T16:32:10.212172+0000","last_fullsized":"2024-04-25T16:32:10.212172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:14:38.930206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205179+0000","last_change":"2024-04-25T16:30:08.985264+0000","last_active":"2024-04-25T16:32:10.205179+0000","last_peered":"2024-04-25T16:32:10.205179+0000","last_clean":"2024-04-25T16:32:10.205179+0000","last_became_active":"2024-04-25T16:30:08.984901+0000","last_became_peered":"2024-04-25T16:30:08.984901+0000","last_unstale":"2024-04-25T16:32:10.205179+0000","last_undegraded":"2024-04-25T16:32:10.205179+0000","last_fullsized":"2024-04-25T16:32:10.205179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:15.271981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530865+0000","last_change":"2024-04-25T16:30:09.997002+0000","last_active":"2024-04-25T16:32:10.530865+0000","last_peered":"2024-04-25T16:32:10.530865+0000","last_clean":"2024-04-25T16:32:10.530865+0000","last_became_active":"2024-04-25T16:30:09.996636+0000","last_became_peered":"2024-04-25T16:30:09.996636+0000","last_unstale":"2024-04-25T16:32:10.530865+0000","last_undegraded":"2024-04-25T16:32:10.530865+0000","last_fullsized":"2024-04-25T16:32:10.530865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:32.522050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017760+0000","last_change":"2024-04-25T16:30:14.006403+0000","last_active":"2024-04-25T16:32:10.017760+0000","last_peered":"2024-04-25T16:32:10.017760+0000","last_clean":"2024-04-25T16:32:10.017760+0000","last_became_active":"2024-04-25T16:30:14.004816+0000","last_became_peered":"2024-04-25T16:30:14.004816+0000","last_unstale":"2024-04-25T16:32:10.017760+0000","last_undegraded":"2024-04-25T16:32:10.017760+0000","last_fullsized":"2024-04-25T16:32:10.017760+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:51:10.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":305,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:30.846875+0000","last_change":"2024-04-25T16:30:08.986119+0000","last_active":"2024-04-25T16:32:30.846875+0000","last_peered":"2024-04-25T16:32:30.846875+0000","last_clean":"2024-04-25T16:32:30.846875+0000","last_became_active":"2024-04-25T16:30:08.985806+0000","last_became_peered":"2024-04-25T16:30:08.985806+0000","last_unstale":"2024-04-25T16:32:30.846875+0000","last_undegraded":"2024-04-25T16:32:30.846875+0000","last_fullsized":"2024-04-25T16:32:30.846875+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:17.125886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64098,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":233,"num_read_kb":179,"num_write":42,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021027+0000","last_change":"2024-04-25T16:30:09.996891+0000","last_active":"2024-04-25T16:32:10.021027+0000","last_peered":"2024-04-25T16:32:10.021027+0000","last_clean":"2024-04-25T16:32:10.021027+0000","last_became_active":"2024-04-25T16:30:09.995851+0000","last_became_peered":"2024-04-25T16:30:09.995851+0000","last_unstale":"2024-04-25T16:32:10.021027+0000","last_undegraded":"2024-04-25T16:32:10.021027+0000","last_fullsized":"2024-04-25T16:32:10.021027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:42.602854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205467+0000","last_change":"2024-04-25T16:30:13.998277+0000","last_active":"2024-04-25T16:32:10.205467+0000","last_peered":"2024-04-25T16:32:10.205467+0000","last_clean":"2024-04-25T16:32:10.205467+0000","last_became_active":"2024-04-25T16:30:13.997313+0000","last_became_peered":"2024-04-25T16:30:13.997313+0000","last_unstale":"2024-04-25T16:32:10.205467+0000","last_undegraded":"2024-04-25T16:32:10.205467+0000","last_fullsized":"2024-04-25T16:32:10.205467+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:56.379806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017045+0000","last_change":"2024-04-25T16:30:08.986377+0000","last_active":"2024-04-25T16:32:10.017045+0000","last_peered":"2024-04-25T16:32:10.017045+0000","last_clean":"2024-04-25T16:32:10.017045+0000","last_became_active":"2024-04-25T16:30:08.985892+0000","last_became_peered":"2024-04-25T16:30:08.985892+0000","last_unstale":"2024-04-25T16:32:10.017045+0000","last_undegraded":"2024-04-25T16:32:10.017045+0000","last_fullsized":"2024-04-25T16:32:10.017045+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:53.651789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013775+0000","last_change":"2024-04-25T16:31:09.010551+0000","last_active":"2024-04-25T16:32:10.013775+0000","last_peered":"2024-04-25T16:32:10.013775+0000","last_clean":"2024-04-25T16:32:10.013775+0000","last_became_active":"2024-04-25T16:31:09.010141+0000","last_became_peered":"2024-04-25T16:31:09.010141+0000","last_unstale":"2024-04-25T16:32:10.013775+0000","last_undegraded":"2024-04-25T16:32:10.013775+0000","last_fullsized":"2024-04-25T16:32:10.013775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:54.650080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022080+0000","last_change":"2024-04-25T16:30:14.006064+0000","last_active":"2024-04-25T16:32:10.022080+0000","last_peered":"2024-04-25T16:32:10.022080+0000","last_clean":"2024-04-25T16:32:10.022080+0000","last_became_active":"2024-04-25T16:30:14.005545+0000","last_became_peered":"2024-04-25T16:30:14.005545+0000","last_unstale":"2024-04-25T16:32:10.022080+0000","last_undegraded":"2024-04-25T16:32:10.022080+0000","last_fullsized":"2024-04-25T16:32:10.022080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:36.611008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205079+0000","last_change":"2024-04-25T16:30:08.988413+0000","last_active":"2024-04-25T16:32:10.205079+0000","last_peered":"2024-04-25T16:32:10.205079+0000","last_clean":"2024-04-25T16:32:10.205079+0000","last_became_active":"2024-04-25T16:30:08.988075+0000","last_became_peered":"2024-04-25T16:30:08.988075+0000","last_unstale":"2024-04-25T16:32:10.205079+0000","last_undegraded":"2024-04-25T16:32:10.205079+0000","last_fullsized":"2024-04-25T16:32:10.205079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:16.635876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022110+0000","last_change":"2024-04-25T16:30:09.997697+0000","last_active":"2024-04-25T16:32:10.022110+0000","last_peered":"2024-04-25T16:32:10.022110+0000","last_clean":"2024-04-25T16:32:10.022110+0000","last_became_active":"2024-04-25T16:30:09.997086+0000","last_became_peered":"2024-04-25T16:30:09.997086+0000","last_unstale":"2024-04-25T16:32:10.022110+0000","last_undegraded":"2024-04-25T16:32:10.022110+0000","last_fullsized":"2024-04-25T16:32:10.022110+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:35.345902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017958+0000","last_change":"2024-04-25T16:30:14.010486+0000","last_active":"2024-04-25T16:32:10.017958+0000","last_peered":"2024-04-25T16:32:10.017958+0000","last_clean":"2024-04-25T16:32:10.017958+0000","last_became_active":"2024-04-25T16:30:14.009273+0000","last_became_peered":"2024-04-25T16:30:14.009273+0000","last_unstale":"2024-04-25T16:32:10.017958+0000","last_undegraded":"2024-04-25T16:32:10.017958+0000","last_fullsized":"2024-04-25T16:32:10.017958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:11.863575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'8","reported_seq":159,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:32.245066+0000","last_change":"2024-04-25T16:30:08.986661+0000","last_active":"2024-04-25T16:32:32.245066+0000","last_peered":"2024-04-25T16:32:32.245066+0000","last_clean":"2024-04-25T16:32:32.245066+0000","last_became_active":"2024-04-25T16:30:08.986306+0000","last_became_peered":"2024-04-25T16:30:08.986306+0000","last_unstale":"2024-04-25T16:32:32.245066+0000","last_undegraded":"2024-04-25T16:32:32.245066+0000","last_fullsized":"2024-04-25T16:32:32.245066+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:44.363693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021934+0000","last_change":"2024-04-25T16:30:09.993443+0000","last_active":"2024-04-25T16:32:10.021934+0000","last_peered":"2024-04-25T16:32:10.021934+0000","last_clean":"2024-04-25T16:32:10.021934+0000","last_became_active":"2024-04-25T16:30:09.993031+0000","last_became_peered":"2024-04-25T16:30:09.993031+0000","last_unstale":"2024-04-25T16:32:10.021934+0000","last_undegraded":"2024-04-25T16:32:10.021934+0000","last_fullsized":"2024-04-25T16:32:10.021934+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:22.908713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014172+0000","last_change":"2024-04-25T16:30:14.003183+0000","last_active":"2024-04-25T16:32:10.014172+0000","last_peered":"2024-04-25T16:32:10.014172+0000","last_clean":"2024-04-25T16:32:10.014172+0000","last_became_active":"2024-04-25T16:30:14.002182+0000","last_became_peered":"2024-04-25T16:30:14.002182+0000","last_unstale":"2024-04-25T16:32:10.014172+0000","last_undegraded":"2024-04-25T16:32:10.014172+0000","last_fullsized":"2024-04-25T16:32:10.014172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:08.616556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020096+0000","last_change":"2024-04-25T16:30:08.995860+0000","last_active":"2024-04-25T16:32:10.020096+0000","last_peered":"2024-04-25T16:32:10.020096+0000","last_clean":"2024-04-25T16:32:10.020096+0000","last_became_active":"2024-04-25T16:30:08.995487+0000","last_became_peered":"2024-04-25T16:30:08.995487+0000","last_unstale":"2024-04-25T16:32:10.020096+0000","last_undegraded":"2024-04-25T16:32:10.020096+0000","last_fullsized":"2024-04-25T16:32:10.020096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:48.206502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204879+0000","last_change":"2024-04-25T16:30:09.998820+0000","last_active":"2024-04-25T16:32:10.204879+0000","last_peered":"2024-04-25T16:32:10.204879+0000","last_clean":"2024-04-25T16:32:10.204879+0000","last_became_active":"2024-04-25T16:30:09.998448+0000","last_became_peered":"2024-04-25T16:30:09.998448+0000","last_unstale":"2024-04-25T16:32:10.204879+0000","last_undegraded":"2024-04-25T16:32:10.204879+0000","last_fullsized":"2024-04-25T16:32:10.204879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:54.361250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204753+0000","last_change":"2024-04-25T16:30:13.992379+0000","last_active":"2024-04-25T16:32:10.204753+0000","last_peered":"2024-04-25T16:32:10.204753+0000","last_clean":"2024-04-25T16:32:10.204753+0000","last_became_active":"2024-04-25T16:30:13.991940+0000","last_became_peered":"2024-04-25T16:30:13.991940+0000","last_unstale":"2024-04-25T16:32:10.204753+0000","last_undegraded":"2024-04-25T16:32:10.204753+0000","last_fullsized":"2024-04-25T16:32:10.204753+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:59.152384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:27.942459+0000","last_change":"2024-04-25T16:30:08.987701+0000","last_active":"2024-04-25T16:32:27.942459+0000","last_peered":"2024-04-25T16:32:27.942459+0000","last_clean":"2024-04-25T16:32:27.942459+0000","last_became_active":"2024-04-25T16:30:08.987376+0000","last_became_peered":"2024-04-25T16:30:08.987376+0000","last_unstale":"2024-04-25T16:32:27.942459+0000","last_undegraded":"2024-04-25T16:32:27.942459+0000","last_fullsized":"2024-04-25T16:32:27.942459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:21.810100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021777+0000","last_change":"2024-04-25T16:30:09.993005+0000","last_active":"2024-04-25T16:32:10.021777+0000","last_peered":"2024-04-25T16:32:10.021777+0000","last_clean":"2024-04-25T16:32:10.021777+0000","last_became_active":"2024-04-25T16:30:09.992619+0000","last_became_peered":"2024-04-25T16:30:09.992619+0000","last_unstale":"2024-04-25T16:32:10.021777+0000","last_undegraded":"2024-04-25T16:32:10.021777+0000","last_fullsized":"2024-04-25T16:32:10.021777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:39:40.998241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020416+0000","last_change":"2024-04-25T16:30:14.147895+0000","last_active":"2024-04-25T16:32:10.020416+0000","last_peered":"2024-04-25T16:32:10.020416+0000","last_clean":"2024-04-25T16:32:10.020416+0000","last_became_active":"2024-04-25T16:30:14.147298+0000","last_became_peered":"2024-04-25T16:30:14.147298+0000","last_unstale":"2024-04-25T16:32:10.020416+0000","last_undegraded":"2024-04-25T16:32:10.020416+0000","last_fullsized":"2024-04-25T16:32:10.020416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:06.269587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018035+0000","last_change":"2024-04-25T16:30:08.982683+0000","last_active":"2024-04-25T16:32:10.018035+0000","last_peered":"2024-04-25T16:32:10.018035+0000","last_clean":"2024-04-25T16:32:10.018035+0000","last_became_active":"2024-04-25T16:30:08.982236+0000","last_became_peered":"2024-04-25T16:30:08.982236+0000","last_unstale":"2024-04-25T16:32:10.018035+0000","last_undegraded":"2024-04-25T16:32:10.018035+0000","last_fullsized":"2024-04-25T16:32:10.018035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:48.781791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.015356+0000","last_change":"2024-04-25T16:30:09.986734+0000","last_active":"2024-04-25T16:32:10.015356+0000","last_peered":"2024-04-25T16:32:10.015356+0000","last_clean":"2024-04-25T16:32:10.015356+0000","last_became_active":"2024-04-25T16:30:09.986337+0000","last_became_peered":"2024-04-25T16:30:09.986337+0000","last_unstale":"2024-04-25T16:32:10.015356+0000","last_undegraded":"2024-04-25T16:32:10.015356+0000","last_fullsized":"2024-04-25T16:32:10.015356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.678009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013878+0000","last_change":"2024-04-25T16:30:14.003324+0000","last_active":"2024-04-25T16:32:10.013878+0000","last_peered":"2024-04-25T16:32:10.013878+0000","last_clean":"2024-04-25T16:32:10.013878+0000","last_became_active":"2024-04-25T16:30:14.002904+0000","last_became_peered":"2024-04-25T16:30:14.002904+0000","last_unstale":"2024-04-25T16:32:10.013878+0000","last_undegraded":"2024-04-25T16:32:10.013878+0000","last_fullsized":"2024-04-25T16:32:10.013878+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:28.750102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'6","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.232485+0000","last_change":"2024-04-25T16:30:08.993469+0000","last_active":"2024-04-25T16:32:31.232485+0000","last_peered":"2024-04-25T16:32:31.232485+0000","last_clean":"2024-04-25T16:32:31.232485+0000","last_became_active":"2024-04-25T16:30:08.993126+0000","last_became_peered":"2024-04-25T16:30:08.993126+0000","last_unstale":"2024-04-25T16:32:31.232485+0000","last_undegraded":"2024-04-25T16:32:31.232485+0000","last_fullsized":"2024-04-25T16:32:31.232485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:10.274645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014980+0000","last_change":"2024-04-25T16:30:10.000597+0000","last_active":"2024-04-25T16:32:10.014980+0000","last_peered":"2024-04-25T16:32:10.014980+0000","last_clean":"2024-04-25T16:32:10.014980+0000","last_became_active":"2024-04-25T16:30:09.998892+0000","last_became_peered":"2024-04-25T16:30:09.998892+0000","last_unstale":"2024-04-25T16:32:10.014980+0000","last_undegraded":"2024-04-25T16:32:10.014980+0000","last_fullsized":"2024-04-25T16:32:10.014980+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:09.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013930+0000","last_change":"2024-04-25T16:30:14.147240+0000","last_active":"2024-04-25T16:32:10.013930+0000","last_peered":"2024-04-25T16:32:10.013930+0000","last_clean":"2024-04-25T16:32:10.013930+0000","last_became_active":"2024-04-25T16:30:14.146793+0000","last_became_peered":"2024-04-25T16:30:14.146793+0000","last_unstale":"2024-04-25T16:32:10.013930+0000","last_undegraded":"2024-04-25T16:32:10.013930+0000","last_fullsized":"2024-04-25T16:32:10.013930+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:55.713386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204864+0000","last_change":"2024-04-25T16:30:09.994595+0000","last_active":"2024-04-25T16:32:10.204864+0000","last_peered":"2024-04-25T16:32:10.204864+0000","last_clean":"2024-04-25T16:32:10.204864+0000","last_became_active":"2024-04-25T16:30:09.994305+0000","last_became_peered":"2024-04-25T16:30:09.994305+0000","last_unstale":"2024-04-25T16:32:10.204864+0000","last_undegraded":"2024-04-25T16:32:10.204864+0000","last_fullsized":"2024-04-25T16:32:10.204864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:27.839500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'7","reported_seq":156,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:32.244380+0000","last_change":"2024-04-25T16:30:08.984203+0000","last_active":"2024-04-25T16:32:32.244380+0000","last_peered":"2024-04-25T16:32:32.244380+0000","last_clean":"2024-04-25T16:32:32.244380+0000","last_became_active":"2024-04-25T16:30:08.982883+0000","last_became_peered":"2024-04-25T16:30:08.982883+0000","last_unstale":"2024-04-25T16:32:32.244380+0000","last_undegraded":"2024-04-25T16:32:32.244380+0000","last_fullsized":"2024-04-25T16:32:32.244380+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:18.447624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022086+0000","last_change":"2024-04-25T16:31:09.017505+0000","last_active":"2024-04-25T16:32:10.022086+0000","last_peered":"2024-04-25T16:32:10.022086+0000","last_clean":"2024-04-25T16:32:10.022086+0000","last_became_active":"2024-04-25T16:31:09.011812+0000","last_became_peered":"2024-04-25T16:31:09.011812+0000","last_unstale":"2024-04-25T16:32:10.022086+0000","last_undegraded":"2024-04-25T16:32:10.022086+0000","last_fullsized":"2024-04-25T16:32:10.022086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:02.404908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016285+0000","last_change":"2024-04-25T16:30:09.990998+0000","last_active":"2024-04-25T16:32:10.016285+0000","last_peered":"2024-04-25T16:32:10.016285+0000","last_clean":"2024-04-25T16:32:10.016285+0000","last_became_active":"2024-04-25T16:30:09.990644+0000","last_became_peered":"2024-04-25T16:30:09.990644+0000","last_unstale":"2024-04-25T16:32:10.016285+0000","last_undegraded":"2024-04-25T16:32:10.016285+0000","last_fullsized":"2024-04-25T16:32:10.016285+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:42.175520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212092+0000","last_change":"2024-04-25T16:30:08.988315+0000","last_active":"2024-04-25T16:32:10.212092+0000","last_peered":"2024-04-25T16:32:10.212092+0000","last_clean":"2024-04-25T16:32:10.212092+0000","last_became_active":"2024-04-25T16:30:08.987930+0000","last_became_peered":"2024-04-25T16:30:08.987930+0000","last_unstale":"2024-04-25T16:32:10.212092+0000","last_undegraded":"2024-04-25T16:32:10.212092+0000","last_fullsized":"2024-04-25T16:32:10.212092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:44.884602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":106,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021552+0000","last_change":"2024-04-25T16:28:58.219510+0000","last_active":"2024-04-25T16:32:10.021552+0000","last_peered":"2024-04-25T16:32:10.021552+0000","last_clean":"2024-04-25T16:32:10.021552+0000","last_became_active":"2024-04-25T16:28:58.213888+0000","last_became_peered":"2024-04-25T16:28:58.213888+0000","last_unstale":"2024-04-25T16:32:10.021552+0000","last_undegraded":"2024-04-25T16:32:10.021552+0000","last_fullsized":"2024-04-25T16:32:10.021552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_clean_scrub_stamp":"2024-04-25T16:25:33.939186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:15.378406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205344+0000","last_change":"2024-04-25T16:31:09.012831+0000","last_active":"2024-04-25T16:32:10.205344+0000","last_peered":"2024-04-25T16:32:10.205344+0000","last_clean":"2024-04-25T16:32:10.205344+0000","last_became_active":"2024-04-25T16:31:09.012375+0000","last_became_peered":"2024-04-25T16:31:09.012375+0000","last_unstale":"2024-04-25T16:32:10.205344+0000","last_undegraded":"2024-04-25T16:32:10.205344+0000","last_fullsized":"2024-04-25T16:32:10.205344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:02.214734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013551+0000","last_change":"2024-04-25T16:30:10.005210+0000","last_active":"2024-04-25T16:32:10.013551+0000","last_peered":"2024-04-25T16:32:10.013551+0000","last_clean":"2024-04-25T16:32:10.013551+0000","last_became_active":"2024-04-25T16:30:10.004936+0000","last_became_peered":"2024-04-25T16:30:10.004936+0000","last_unstale":"2024-04-25T16:32:10.013551+0000","last_undegraded":"2024-04-25T16:32:10.013551+0000","last_fullsized":"2024-04-25T16:32:10.013551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:53.292675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021680+0000","last_change":"2024-04-25T16:30:08.990162+0000","last_active":"2024-04-25T16:32:10.021680+0000","last_peered":"2024-04-25T16:32:10.021680+0000","last_clean":"2024-04-25T16:32:10.021680+0000","last_became_active":"2024-04-25T16:30:08.989613+0000","last_became_peered":"2024-04-25T16:30:08.989613+0000","last_unstale":"2024-04-25T16:32:10.021680+0000","last_undegraded":"2024-04-25T16:32:10.021680+0000","last_fullsized":"2024-04-25T16:32:10.021680+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:41.595381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021251+0000","last_change":"2024-04-25T16:30:14.001850+0000","last_active":"2024-04-25T16:32:10.021251+0000","last_peered":"2024-04-25T16:32:10.021251+0000","last_clean":"2024-04-25T16:32:10.021251+0000","last_became_active":"2024-04-25T16:30:14.001348+0000","last_became_peered":"2024-04-25T16:30:14.001348+0000","last_unstale":"2024-04-25T16:32:10.021251+0000","last_undegraded":"2024-04-25T16:32:10.021251+0000","last_fullsized":"2024-04-25T16:32:10.021251+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:56:41.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021095+0000","last_change":"2024-04-25T16:30:08.995833+0000","last_active":"2024-04-25T16:32:10.021095+0000","last_peered":"2024-04-25T16:32:10.021095+0000","last_clean":"2024-04-25T16:32:10.021095+0000","last_became_active":"2024-04-25T16:30:08.995545+0000","last_became_peered":"2024-04-25T16:30:08.995545+0000","last_unstale":"2024-04-25T16:32:10.021095+0000","last_undegraded":"2024-04-25T16:32:10.021095+0000","last_fullsized":"2024-04-25T16:32:10.021095+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:31.452219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014096+0000","last_change":"2024-04-25T16:30:09.990225+0000","last_active":"2024-04-25T16:32:10.014096+0000","last_peered":"2024-04-25T16:32:10.014096+0000","last_clean":"2024-04-25T16:32:10.014096+0000","last_became_active":"2024-04-25T16:30:09.989859+0000","last_became_peered":"2024-04-25T16:30:09.989859+0000","last_unstale":"2024-04-25T16:32:10.014096+0000","last_undegraded":"2024-04-25T16:32:10.014096+0000","last_fullsized":"2024-04-25T16:32:10.014096+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:40.589395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016177+0000","last_change":"2024-04-25T16:30:14.147223+0000","last_active":"2024-04-25T16:32:10.016177+0000","last_peered":"2024-04-25T16:32:10.016177+0000","last_clean":"2024-04-25T16:32:10.016177+0000","last_became_active":"2024-04-25T16:30:14.146821+0000","last_became_peered":"2024-04-25T16:30:14.146821+0000","last_unstale":"2024-04-25T16:32:10.016177+0000","last_undegraded":"2024-04-25T16:32:10.016177+0000","last_fullsized":"2024-04-25T16:32:10.016177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:01.289165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'16","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:33.364360+0000","last_change":"2024-04-25T16:32:10.029459+0000","last_active":"2024-04-25T16:32:33.364360+0000","last_peered":"2024-04-25T16:32:33.364360+0000","last_clean":"2024-04-25T16:32:33.364360+0000","last_became_active":"2024-04-25T16:32:10.022574+0000","last_became_peered":"2024-04-25T16:32:10.022574+0000","last_unstale":"2024-04-25T16:32:33.364360+0000","last_undegraded":"2024-04-25T16:32:33.364360+0000","last_fullsized":"2024-04-25T16:32:33.364360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:41.675443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016438+0000","last_change":"2024-04-25T16:30:09.986088+0000","last_active":"2024-04-25T16:32:10.016438+0000","last_peered":"2024-04-25T16:32:10.016438+0000","last_clean":"2024-04-25T16:32:10.016438+0000","last_became_active":"2024-04-25T16:30:09.985256+0000","last_became_peered":"2024-04-25T16:30:09.985256+0000","last_unstale":"2024-04-25T16:32:10.016438+0000","last_undegraded":"2024-04-25T16:32:10.016438+0000","last_fullsized":"2024-04-25T16:32:10.016438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:29.197558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016928+0000","last_change":"2024-04-25T16:30:14.000699+0000","last_active":"2024-04-25T16:32:10.016928+0000","last_peered":"2024-04-25T16:32:10.016928+0000","last_clean":"2024-04-25T16:32:10.016928+0000","last_became_active":"2024-04-25T16:30:14.000255+0000","last_became_peered":"2024-04-25T16:30:14.000255+0000","last_unstale":"2024-04-25T16:32:10.016928+0000","last_undegraded":"2024-04-25T16:32:10.016928+0000","last_fullsized":"2024-04-25T16:32:10.016928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:33.436693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212550+0000","last_change":"2024-04-25T16:30:08.981926+0000","last_active":"2024-04-25T16:32:10.212550+0000","last_peered":"2024-04-25T16:32:10.212550+0000","last_clean":"2024-04-25T16:32:10.212550+0000","last_became_active":"2024-04-25T16:30:08.981590+0000","last_became_peered":"2024-04-25T16:30:08.981590+0000","last_unstale":"2024-04-25T16:32:10.212550+0000","last_undegraded":"2024-04-25T16:32:10.212550+0000","last_fullsized":"2024-04-25T16:32:10.212550+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:57.818774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781964+0000","last_change":"2024-04-25T16:30:10.003998+0000","last_active":"2024-04-25T16:32:10.781964+0000","last_peered":"2024-04-25T16:32:10.781964+0000","last_clean":"2024-04-25T16:32:10.781964+0000","last_became_active":"2024-04-25T16:30:10.003626+0000","last_became_peered":"2024-04-25T16:30:10.003626+0000","last_unstale":"2024-04-25T16:32:10.781964+0000","last_undegraded":"2024-04-25T16:32:10.781964+0000","last_fullsized":"2024-04-25T16:32:10.781964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:10.686092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212401+0000","last_change":"2024-04-25T16:30:14.149278+0000","last_active":"2024-04-25T16:32:10.212401+0000","last_peered":"2024-04-25T16:32:10.212401+0000","last_clean":"2024-04-25T16:32:10.212401+0000","last_became_active":"2024-04-25T16:30:14.148654+0000","last_became_peered":"2024-04-25T16:30:14.148654+0000","last_unstale":"2024-04-25T16:32:10.212401+0000","last_undegraded":"2024-04-25T16:32:10.212401+0000","last_fullsized":"2024-04-25T16:32:10.212401+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:24.823222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530250+0000","last_change":"2024-04-25T16:30:08.981612+0000","last_active":"2024-04-25T16:32:10.530250+0000","last_peered":"2024-04-25T16:32:10.530250+0000","last_clean":"2024-04-25T16:32:10.530250+0000","last_became_active":"2024-04-25T16:30:08.981222+0000","last_became_peered":"2024-04-25T16:30:08.981222+0000","last_unstale":"2024-04-25T16:32:10.530250+0000","last_undegraded":"2024-04-25T16:32:10.530250+0000","last_fullsized":"2024-04-25T16:32:10.530250+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:14.914658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.019805+0000","last_change":"2024-04-25T16:30:09.989592+0000","last_active":"2024-04-25T16:32:10.019805+0000","last_peered":"2024-04-25T16:32:10.019805+0000","last_clean":"2024-04-25T16:32:10.019805+0000","last_became_active":"2024-04-25T16:30:09.989222+0000","last_became_peered":"2024-04-25T16:30:09.989222+0000","last_unstale":"2024-04-25T16:32:10.019805+0000","last_undegraded":"2024-04-25T16:32:10.019805+0000","last_fullsized":"2024-04-25T16:32:10.019805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:52.403239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018128+0000","last_change":"2024-04-25T16:30:14.010480+0000","last_active":"2024-04-25T16:32:10.018128+0000","last_peered":"2024-04-25T16:32:10.018128+0000","last_clean":"2024-04-25T16:32:10.018128+0000","last_became_active":"2024-04-25T16:30:14.009023+0000","last_became_peered":"2024-04-25T16:30:14.009023+0000","last_unstale":"2024-04-25T16:32:10.018128+0000","last_undegraded":"2024-04-25T16:32:10.018128+0000","last_fullsized":"2024-04-25T16:32:10.018128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:10.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017158+0000","last_change":"2024-04-25T16:30:08.986849+0000","last_active":"2024-04-25T16:32:10.017158+0000","last_peered":"2024-04-25T16:32:10.017158+0000","last_clean":"2024-04-25T16:32:10.017158+0000","last_became_active":"2024-04-25T16:30:08.986479+0000","last_became_peered":"2024-04-25T16:30:08.986479+0000","last_unstale":"2024-04-25T16:32:10.017158+0000","last_undegraded":"2024-04-25T16:32:10.017158+0000","last_fullsized":"2024-04-25T16:32:10.017158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022094+0000","last_change":"2024-04-25T16:30:09.990927+0000","last_active":"2024-04-25T16:32:10.022094+0000","last_peered":"2024-04-25T16:32:10.022094+0000","last_clean":"2024-04-25T16:32:10.022094+0000","last_became_active":"2024-04-25T16:30:09.990482+0000","last_became_peered":"2024-04-25T16:30:09.990482+0000","last_unstale":"2024-04-25T16:32:10.022094+0000","last_undegraded":"2024-04-25T16:32:10.022094+0000","last_fullsized":"2024-04-25T16:32:10.022094+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:18.649721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212337+0000","last_change":"2024-04-25T16:30:14.006159+0000","last_active":"2024-04-25T16:32:10.212337+0000","last_peered":"2024-04-25T16:32:10.212337+0000","last_clean":"2024-04-25T16:32:10.212337+0000","last_became_active":"2024-04-25T16:30:14.005413+0000","last_became_peered":"2024-04-25T16:30:14.005413+0000","last_unstale":"2024-04-25T16:32:10.212337+0000","last_undegraded":"2024-04-25T16:32:10.212337+0000","last_fullsized":"2024-04-25T16:32:10.212337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:52.934385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.026829+0000","last_change":"2024-04-25T16:30:08.986207+0000","last_active":"2024-04-25T16:32:31.026829+0000","last_peered":"2024-04-25T16:32:31.026829+0000","last_clean":"2024-04-25T16:32:31.026829+0000","last_became_active":"2024-04-25T16:30:08.985857+0000","last_became_peered":"2024-04-25T16:30:08.985857+0000","last_unstale":"2024-04-25T16:32:31.026829+0000","last_undegraded":"2024-04-25T16:32:31.026829+0000","last_fullsized":"2024-04-25T16:32:31.026829+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:45.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016835+0000","last_change":"2024-04-25T16:30:09.993566+0000","last_active":"2024-04-25T16:32:10.016835+0000","last_peered":"2024-04-25T16:32:10.016835+0000","last_clean":"2024-04-25T16:32:10.016835+0000","last_became_active":"2024-04-25T16:30:09.992250+0000","last_became_peered":"2024-04-25T16:30:09.992250+0000","last_unstale":"2024-04-25T16:32:10.016835+0000","last_undegraded":"2024-04-25T16:32:10.016835+0000","last_fullsized":"2024-04-25T16:32:10.016835+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:33.279300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212478+0000","last_change":"2024-04-25T16:30:14.147443+0000","last_active":"2024-04-25T16:32:10.212478+0000","last_peered":"2024-04-25T16:32:10.212478+0000","last_clean":"2024-04-25T16:32:10.212478+0000","last_became_active":"2024-04-25T16:30:14.146838+0000","last_became_peered":"2024-04-25T16:30:14.146838+0000","last_unstale":"2024-04-25T16:32:10.212478+0000","last_undegraded":"2024-04-25T16:32:10.212478+0000","last_fullsized":"2024-04-25T16:32:10.212478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:51.532685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020627+0000","last_change":"2024-04-25T16:31:09.015526+0000","last_active":"2024-04-25T16:32:10.020627+0000","last_peered":"2024-04-25T16:32:10.020627+0000","last_clean":"2024-04-25T16:32:10.020627+0000","last_became_active":"2024-04-25T16:31:09.007972+0000","last_became_peered":"2024-04-25T16:31:09.007972+0000","last_unstale":"2024-04-25T16:32:10.020627+0000","last_undegraded":"2024-04-25T16:32:10.020627+0000","last_fullsized":"2024-04-25T16:32:10.020627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:56.464426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022010+0000","last_change":"2024-04-25T16:30:09.998486+0000","last_active":"2024-04-25T16:32:10.022010+0000","last_peered":"2024-04-25T16:32:10.022010+0000","last_clean":"2024-04-25T16:32:10.022010+0000","last_became_active":"2024-04-25T16:30:09.998154+0000","last_became_peered":"2024-04-25T16:30:09.998154+0000","last_unstale":"2024-04-25T16:32:10.022010+0000","last_undegraded":"2024-04-25T16:32:10.022010+0000","last_fullsized":"2024-04-25T16:32:10.022010+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:55.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530492+0000","last_change":"2024-04-25T16:30:13.995864+0000","last_active":"2024-04-25T16:32:10.530492+0000","last_peered":"2024-04-25T16:32:10.530492+0000","last_clean":"2024-04-25T16:32:10.530492+0000","last_became_active":"2024-04-25T16:30:13.995326+0000","last_became_peered":"2024-04-25T16:30:13.995326+0000","last_unstale":"2024-04-25T16:32:10.530492+0000","last_undegraded":"2024-04-25T16:32:10.530492+0000","last_fullsized":"2024-04-25T16:32:10.530492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:11.718826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":156,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.265799+0000","last_change":"2024-04-25T16:30:08.984189+0000","last_active":"2024-04-25T16:32:31.265799+0000","last_peered":"2024-04-25T16:32:31.265799+0000","last_clean":"2024-04-25T16:32:31.265799+0000","last_became_active":"2024-04-25T16:30:08.983821+0000","last_became_peered":"2024-04-25T16:30:08.983821+0000","last_unstale":"2024-04-25T16:32:31.265799+0000","last_undegraded":"2024-04-25T16:32:31.265799+0000","last_fullsized":"2024-04-25T16:32:31.265799+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:51.315397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022285+0000","last_change":"2024-04-25T16:30:09.998763+0000","last_active":"2024-04-25T16:32:10.022285+0000","last_peered":"2024-04-25T16:32:10.022285+0000","last_clean":"2024-04-25T16:32:10.022285+0000","last_became_active":"2024-04-25T16:30:09.998439+0000","last_became_peered":"2024-04-25T16:30:09.998439+0000","last_unstale":"2024-04-25T16:32:10.022285+0000","last_undegraded":"2024-04-25T16:32:10.022285+0000","last_fullsized":"2024-04-25T16:32:10.022285+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:14.279883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781635+0000","last_change":"2024-04-25T16:30:14.005427+0000","last_active":"2024-04-25T16:32:10.781635+0000","last_peered":"2024-04-25T16:32:10.781635+0000","last_clean":"2024-04-25T16:32:10.781635+0000","last_became_active":"2024-04-25T16:30:14.005006+0000","last_became_peered":"2024-04-25T16:30:14.005006+0000","last_unstale":"2024-04-25T16:32:10.781635+0000","last_undegraded":"2024-04-25T16:32:10.781635+0000","last_fullsized":"2024-04-25T16:32:10.781635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:49.563365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205387+0000","last_change":"2024-04-25T16:30:08.985557+0000","last_active":"2024-04-25T16:32:10.205387+0000","last_peered":"2024-04-25T16:32:10.205387+0000","last_clean":"2024-04-25T16:32:10.205387+0000","last_became_active":"2024-04-25T16:30:08.985215+0000","last_became_peered":"2024-04-25T16:30:08.985215+0000","last_unstale":"2024-04-25T16:32:10.205387+0000","last_undegraded":"2024-04-25T16:32:10.205387+0000","last_fullsized":"2024-04-25T16:32:10.205387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:51.504089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022076+0000","last_change":"2024-04-25T16:30:09.991336+0000","last_active":"2024-04-25T16:32:10.022076+0000","last_peered":"2024-04-25T16:32:10.022076+0000","last_clean":"2024-04-25T16:32:10.022076+0000","last_became_active":"2024-04-25T16:30:09.990819+0000","last_became_peered":"2024-04-25T16:30:09.990819+0000","last_unstale":"2024-04-25T16:32:10.022076+0000","last_undegraded":"2024-04-25T16:32:10.022076+0000","last_fullsized":"2024-04-25T16:32:10.022076+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:45.988061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204595+0000","last_change":"2024-04-25T16:30:13.998463+0000","last_active":"2024-04-25T16:32:10.204595+0000","last_peered":"2024-04-25T16:32:10.204595+0000","last_clean":"2024-04-25T16:32:10.204595+0000","last_became_active":"2024-04-25T16:30:13.997952+0000","last_became_peered":"2024-04-25T16:30:13.997952+0000","last_unstale":"2024-04-25T16:32:10.204595+0000","last_undegraded":"2024-04-25T16:32:10.204595+0000","last_fullsized":"2024-04-25T16:32:10.204595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:58.507942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.536390+0000","last_change":"2024-04-25T16:32:10.535950+0000","last_active":"2024-04-25T16:32:10.536390+0000","last_peered":"2024-04-25T16:32:10.536390+0000","last_clean":"2024-04-25T16:32:10.536390+0000","last_became_active":"2024-04-25T16:32:10.531823+0000","last_became_peered":"2024-04-25T16:32:10.531823+0000","last_unstale":"2024-04-25T16:32:10.536390+0000","last_undegraded":"2024-04-25T16:32:10.536390+0000","last_fullsized":"2024-04-25T16:32:10.536390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:36.471975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022474+0000","last_change":"2024-04-25T16:30:09.988566+0000","last_active":"2024-04-25T16:32:10.022474+0000","last_peered":"2024-04-25T16:32:10.022474+0000","last_clean":"2024-04-25T16:32:10.022474+0000","last_became_active":"2024-04-25T16:30:09.988164+0000","last_became_peered":"2024-04-25T16:30:09.988164+0000","last_unstale":"2024-04-25T16:32:10.022474+0000","last_undegraded":"2024-04-25T16:32:10.022474+0000","last_fullsized":"2024-04-25T16:32:10.022474+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:40.476766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022056+0000","last_change":"2024-04-25T16:30:09.996349+0000","last_active":"2024-04-25T16:32:10.022056+0000","last_peered":"2024-04-25T16:32:10.022056+0000","last_clean":"2024-04-25T16:32:10.022056+0000","last_became_active":"2024-04-25T16:30:09.994974+0000","last_became_peered":"2024-04-25T16:30:09.994974+0000","last_unstale":"2024-04-25T16:32:10.022056+0000","last_undegraded":"2024-04-25T16:32:10.022056+0000","last_fullsized":"2024-04-25T16:32:10.022056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:46.929057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017091+0000","last_change":"2024-04-25T16:30:09.997837+0000","last_active":"2024-04-25T16:32:10.017091+0000","last_peered":"2024-04-25T16:32:10.017091+0000","last_clean":"2024-04-25T16:32:10.017091+0000","last_became_active":"2024-04-25T16:30:09.997361+0000","last_became_peered":"2024-04-25T16:30:09.997361+0000","last_unstale":"2024-04-25T16:32:10.017091+0000","last_undegraded":"2024-04-25T16:32:10.017091+0000","last_fullsized":"2024-04-25T16:32:10.017091+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:35.698528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016888+0000","last_change":"2024-04-25T16:30:09.991794+0000","last_active":"2024-04-25T16:32:10.016888+0000","last_peered":"2024-04-25T16:32:10.016888+0000","last_clean":"2024-04-25T16:32:10.016888+0000","last_became_active":"2024-04-25T16:30:09.991440+0000","last_became_peered":"2024-04-25T16:30:09.991440+0000","last_unstale":"2024-04-25T16:32:10.016888+0000","last_undegraded":"2024-04-25T16:32:10.016888+0000","last_fullsized":"2024-04-25T16:32:10.016888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:11.322254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781869+0000","last_change":"2024-04-25T16:30:10.002364+0000","last_active":"2024-04-25T16:32:10.781869+0000","last_peered":"2024-04-25T16:32:10.781869+0000","last_clean":"2024-04-25T16:32:10.781869+0000","last_became_active":"2024-04-25T16:30:10.001957+0000","last_became_peered":"2024-04-25T16:30:10.001957+0000","last_unstale":"2024-04-25T16:32:10.781869+0000","last_undegraded":"2024-04-25T16:32:10.781869+0000","last_fullsized":"2024-04-25T16:32:10.781869+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:19.150175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014045+0000","last_change":"2024-04-25T16:30:10.004150+0000","last_active":"2024-04-25T16:32:10.014045+0000","last_peered":"2024-04-25T16:32:10.014045+0000","last_clean":"2024-04-25T16:32:10.014045+0000","last_became_active":"2024-04-25T16:30:10.003733+0000","last_became_peered":"2024-04-25T16:30:10.003733+0000","last_unstale":"2024-04-25T16:32:10.014045+0000","last_undegraded":"2024-04-25T16:32:10.014045+0000","last_fullsized":"2024-04-25T16:32:10.014045+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:11.850366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021171+0000","last_change":"2024-04-25T16:30:09.991056+0000","last_active":"2024-04-25T16:32:10.021171+0000","last_peered":"2024-04-25T16:32:10.021171+0000","last_clean":"2024-04-25T16:32:10.021171+0000","last_became_active":"2024-04-25T16:30:09.988708+0000","last_became_peered":"2024-04-25T16:30:09.988708+0000","last_unstale":"2024-04-25T16:32:10.021171+0000","last_undegraded":"2024-04-25T16:32:10.021171+0000","last_fullsized":"2024-04-25T16:32:10.021171+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:13.062269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017612+0000","last_change":"2024-04-25T16:30:09.996571+0000","last_active":"2024-04-25T16:32:10.017612+0000","last_peered":"2024-04-25T16:32:10.017612+0000","last_clean":"2024-04-25T16:32:10.017612+0000","last_became_active":"2024-04-25T16:30:09.996206+0000","last_became_peered":"2024-04-25T16:30:09.996206+0000","last_unstale":"2024-04-25T16:32:10.017612+0000","last_undegraded":"2024-04-25T16:32:10.017612+0000","last_fullsized":"2024-04-25T16:32:10.017612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:07.107949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016603+0000","last_change":"2024-04-25T16:30:09.995067+0000","last_active":"2024-04-25T16:32:10.016603+0000","last_peered":"2024-04-25T16:32:10.016603+0000","last_clean":"2024-04-25T16:32:10.016603+0000","last_became_active":"2024-04-25T16:30:09.994678+0000","last_became_peered":"2024-04-25T16:30:09.994678+0000","last_unstale":"2024-04-25T16:32:10.016603+0000","last_undegraded":"2024-04-25T16:32:10.016603+0000","last_fullsized":"2024-04-25T16:32:10.016603+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:09.000869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204798+0000","last_change":"2024-04-25T16:30:09.996821+0000","last_active":"2024-04-25T16:32:10.204798+0000","last_peered":"2024-04-25T16:32:10.204798+0000","last_clean":"2024-04-25T16:32:10.204798+0000","last_became_active":"2024-04-25T16:30:09.996464+0000","last_became_peered":"2024-04-25T16:30:09.996464+0000","last_unstale":"2024-04-25T16:32:10.204798+0000","last_undegraded":"2024-04-25T16:32:10.204798+0000","last_fullsized":"2024-04-25T16:32:10.204798+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:54.961544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017023+0000","last_change":"2024-04-25T16:30:09.991733+0000","last_active":"2024-04-25T16:32:10.017023+0000","last_peered":"2024-04-25T16:32:10.017023+0000","last_clean":"2024-04-25T16:32:10.017023+0000","last_became_active":"2024-04-25T16:30:09.991401+0000","last_became_peered":"2024-04-25T16:30:09.991401+0000","last_unstale":"2024-04-25T16:32:10.017023+0000","last_undegraded":"2024-04-25T16:32:10.017023+0000","last_fullsized":"2024-04-25T16:32:10.017023+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:14.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530708+0000","last_change":"2024-04-25T16:30:09.999087+0000","last_active":"2024-04-25T16:32:10.530708+0000","last_peered":"2024-04-25T16:32:10.530708+0000","last_clean":"2024-04-25T16:32:10.530708+0000","last_became_active":"2024-04-25T16:30:09.998725+0000","last_became_peered":"2024-04-25T16:30:09.998725+0000","last_unstale":"2024-04-25T16:32:10.530708+0000","last_undegraded":"2024-04-25T16:32:10.530708+0000","last_fullsized":"2024-04-25T16:32:10.530708+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:19.155171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013940+0000","last_change":"2024-04-25T16:30:09.995880+0000","last_active":"2024-04-25T16:32:10.013940+0000","last_peered":"2024-04-25T16:32:10.013940+0000","last_clean":"2024-04-25T16:32:10.013940+0000","last_became_active":"2024-04-25T16:30:09.995514+0000","last_became_peered":"2024-04-25T16:30:09.995514+0000","last_unstale":"2024-04-25T16:32:10.013940+0000","last_undegraded":"2024-04-25T16:32:10.013940+0000","last_fullsized":"2024-04-25T16:32:10.013940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022174+0000","last_change":"2024-04-25T16:30:09.998326+0000","last_active":"2024-04-25T16:32:10.022174+0000","last_peered":"2024-04-25T16:32:10.022174+0000","last_clean":"2024-04-25T16:32:10.022174+0000","last_became_active":"2024-04-25T16:30:09.995803+0000","last_became_peered":"2024-04-25T16:30:09.995803+0000","last_unstale":"2024-04-25T16:32:10.022174+0000","last_undegraded":"2024-04-25T16:32:10.022174+0000","last_fullsized":"2024-04-25T16:32:10.022174+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:46.983947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73556,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1186,"num_read_kb":672,"num_write":273,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":225764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645354,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":475742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841583,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1671168,"data_stored":1130298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037812,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234041,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":475557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1074206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":471176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":536304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":473023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":537549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":470391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":475782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1064174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:32:37.629 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T16:32:37.629 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:32:37.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 -- 172.21.15.98:0/738244733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae41029e0 msgr2=0x7f2ae41036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/738244733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae41029e0 0x7f2ae41036e0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4009900 tx=0x7f2ad402a410 comp rx=0 tx=0).stop 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 -- 172.21.15.98:0/738244733 shutdown_connections 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/738244733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ae4108920 0x7f2ae410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/738244733 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4103c20 0x7f2ae4104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/738244733 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae41029e0 0x7f2ae41036e0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 -- 172.21.15.98:0/738244733 >> 172.21.15.98:0/738244733 conn(0x7f2ae40fdbd0 msgr2=0x7f2ae4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 -- 172.21.15.98:0/738244733 shutdown_connections 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.772+0000 7f2ae8c15640 1 -- 172.21.15.98:0/738244733 wait complete. 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.773+0000 7f2ae8c15640 1 Processor -- start 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.773+0000 7f2ae8c15640 1 -- start start 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.773+0000 7f2ae8c15640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ae41029e0 0x7f2ae419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.773+0000 7f2ae8c15640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae4103c20 0x7f2ae419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.773+0000 7f2ae8c15640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4108920 0x7f2ae41a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.773+0000 7f2ae8c15640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae410f480 con 0x7f2ae4103c20 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.773+0000 7f2ae8c15640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae410f300 con 0x7f2ae41029e0 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.773+0000 7f2ae8c15640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae410f600 con 0x7f2ae4108920 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae4103c20 0x7f2ae419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae4103c20 0x7f2ae419a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37136/0 (socket says 172.21.15.98:37136) 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 -- 172.21.15.98:0/721968466 learned_addr learned my addr 172.21.15.98:0/721968466 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2ae259c640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ae41029e0 0x7f2ae419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2ae2d9d640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4108920 0x7f2ae41a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 -- 172.21.15.98:0/721968466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4108920 msgr2=0x7f2ae41a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4108920 0x7f2ae41a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 -- 172.21.15.98:0/721968466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ae41029e0 msgr2=0x7f2ae419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ae41029e0 0x7f2ae419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 -- 172.21.15.98:0/721968466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae41a1d40 con 0x7f2ae4103c20 2024-04-25T16:32:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adbfff640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae4103c20 0x7f2ae419a740 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00d950 tx=0x7f2acc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adb7fe640 1 -- 172.21.15.98:0/721968466 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc017030 con 0x7f2ae4103c20 2024-04-25T16:32:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae41a1fd0 con 0x7f2ae4103c20 2024-04-25T16:32:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adb7fe640 1 -- 172.21.15.98:0/721968466 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2acc004020 con 0x7f2ae4103c20 2024-04-25T16:32:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2adb7fe640 1 -- 172.21.15.98:0/721968466 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0055d0 con 0x7f2ae4103c20 2024-04-25T16:32:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.774+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae41a24e0 con 0x7f2ae4103c20 2024-04-25T16:32:37.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.775+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae406a190 con 0x7f2ae4103c20 2024-04-25T16:32:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.777+0000 7f2adb7fe640 1 -- 172.21.15.98:0/721968466 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f2acc009a30 con 0x7f2ae4103c20 2024-04-25T16:32:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.777+0000 7f2adb7fe640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ad00afb80 0x7f2ad00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.777+0000 7f2ae259c640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ad00afb80 0x7f2ad00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.777+0000 7f2adb7fe640 1 -- 172.21.15.98:0/721968466 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f2acc0df290 con 0x7f2ae4103c20 2024-04-25T16:32:37.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.778+0000 7f2ae259c640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ad00afb80 0x7f2ad00b2090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ad40057c0 tx=0x7f2ad4000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:37.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.781+0000 7f2adb7fe640 1 -- 172.21.15.98:0/721968466 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2acc0a7960 con 0x7f2ae4103c20 2024-04-25T16:32:37.956 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.957+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 --> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2ae4062fe0 con 0x7f2ad00afb80 2024-04-25T16:32:37.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.966+0000 7f2adb7fe640 1 -- 172.21.15.98:0/721968466 <== mgr.14150 v2:172.21.15.98:6800/4283487583 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340525 (secure 0 0 0) 0x7f2ae4062fe0 con 0x7f2ad00afb80 2024-04-25T16:32:37.969 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:37.971 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-04-25T16:32:37.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.969+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ad00afb80 msgr2=0x7f2ad00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.969+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ad00afb80 0x7f2ad00b2090 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2ad40057c0 tx=0x7f2ad4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae4103c20 msgr2=0x7f2ae419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae4103c20 0x7f2ae419a740 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00d950 tx=0x7f2acc00de10 comp rx=0 tx=0).stop 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 shutdown_connections 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ad00afb80 0x7f2ad00b2090 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ae4108920 0x7f2ae41a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ae4103c20 0x7f2ae419a740 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 --2- 172.21.15.98:0/721968466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ae41029e0 0x7f2ae419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 >> 172.21.15.98:0/721968466 conn(0x7f2ae40fdbd0 msgr2=0x7f2ae4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 shutdown_connections 2024-04-25T16:32:37.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:37.970+0000 7f2ae8c15640 1 -- 172.21.15.98:0/721968466 wait complete. 2024-04-25T16:32:37.980 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-04-25T16:32:35.964692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664158,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1262,"num_read_kb":736,"num_write":398,"num_write_kb":2316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":2951957,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":829,"ondisk_log_size":829,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":395960,"kb_used_data":13520,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124545864,"statfs":{"total":1151940427776,"available":1151534964736,"internally_reserved":0,"allocated":13844480,"data_stored":7663477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19142,"internal_metadata":391492922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2465,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":109,"num_read_kb":59,"num_write":22,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003530"},"pg_stats":[{"pgid":"3.3f","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021251+0000","last_change":"2024-04-25T16:30:09.990700+0000","last_active":"2024-04-25T16:32:10.021251+0000","last_peered":"2024-04-25T16:32:10.021251+0000","last_clean":"2024-04-25T16:32:10.021251+0000","last_became_active":"2024-04-25T16:30:09.990225+0000","last_became_peered":"2024-04-25T16:30:09.990225+0000","last_unstale":"2024-04-25T16:32:10.021251+0000","last_undegraded":"2024-04-25T16:32:10.021251+0000","last_fullsized":"2024-04-25T16:32:10.021251+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:46:33.897277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.782026+0000","last_change":"2024-04-25T16:30:09.989618+0000","last_active":"2024-04-25T16:32:10.782026+0000","last_peered":"2024-04-25T16:32:10.782026+0000","last_clean":"2024-04-25T16:32:10.782026+0000","last_became_active":"2024-04-25T16:30:09.989270+0000","last_became_peered":"2024-04-25T16:30:09.989270+0000","last_unstale":"2024-04-25T16:32:10.782026+0000","last_undegraded":"2024-04-25T16:32:10.782026+0000","last_fullsized":"2024-04-25T16:32:10.782026+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:03.952233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781848+0000","last_change":"2024-04-25T16:30:09.992442+0000","last_active":"2024-04-25T16:32:10.781848+0000","last_peered":"2024-04-25T16:32:10.781848+0000","last_clean":"2024-04-25T16:32:10.781848+0000","last_became_active":"2024-04-25T16:30:09.992115+0000","last_became_peered":"2024-04-25T16:30:09.992115+0000","last_unstale":"2024-04-25T16:32:10.781848+0000","last_undegraded":"2024-04-25T16:32:10.781848+0000","last_fullsized":"2024-04-25T16:32:10.781848+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:29:00.668427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"83'8","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017939+0000","last_change":"2024-04-25T16:31:09.006824+0000","last_active":"2024-04-25T16:32:10.017939+0000","last_peered":"2024-04-25T16:32:10.017939+0000","last_clean":"2024-04-25T16:32:10.017939+0000","last_became_active":"2024-04-25T16:31:09.005560+0000","last_became_peered":"2024-04-25T16:31:09.005560+0000","last_unstale":"2024-04-25T16:32:10.017939+0000","last_undegraded":"2024-04-25T16:32:10.017939+0000","last_fullsized":"2024-04-25T16:32:10.017939+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:32:41.979642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,2],"acting":[9,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.3b","version":"83'4","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020683+0000","last_change":"2024-04-25T16:30:09.991372+0000","last_active":"2024-04-25T16:32:10.020683+0000","last_peered":"2024-04-25T16:32:10.020683+0000","last_clean":"2024-04-25T16:32:10.020683+0000","last_became_active":"2024-04-25T16:30:09.991053+0000","last_became_peered":"2024-04-25T16:30:09.991053+0000","last_unstale":"2024-04-25T16:32:10.020683+0000","last_undegraded":"2024-04-25T16:32:10.020683+0000","last_fullsized":"2024-04-25T16:32:10.020683+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:28.049306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014900+0000","last_change":"2024-04-25T16:30:09.997563+0000","last_active":"2024-04-25T16:32:10.014900+0000","last_peered":"2024-04-25T16:32:10.014900+0000","last_clean":"2024-04-25T16:32:10.014900+0000","last_became_active":"2024-04-25T16:30:09.997158+0000","last_became_peered":"2024-04-25T16:30:09.997158+0000","last_unstale":"2024-04-25T16:32:10.014900+0000","last_undegraded":"2024-04-25T16:32:10.014900+0000","last_fullsized":"2024-04-25T16:32:10.014900+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:21.441957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"83'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021712+0000","last_change":"2024-04-25T16:31:09.009888+0000","last_active":"2024-04-25T16:32:10.021712+0000","last_peered":"2024-04-25T16:32:10.021712+0000","last_clean":"2024-04-25T16:32:10.021712+0000","last_became_active":"2024-04-25T16:31:09.009201+0000","last_became_peered":"2024-04-25T16:31:09.009201+0000","last_unstale":"2024-04-25T16:32:10.021712+0000","last_undegraded":"2024-04-25T16:32:10.021712+0000","last_fullsized":"2024-04-25T16:32:10.021712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:38:52.171410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016068+0000","last_change":"2024-04-25T16:30:09.994820+0000","last_active":"2024-04-25T16:32:10.016068+0000","last_peered":"2024-04-25T16:32:10.016068+0000","last_clean":"2024-04-25T16:32:10.016068+0000","last_became_active":"2024-04-25T16:30:09.994494+0000","last_became_peered":"2024-04-25T16:30:09.994494+0000","last_unstale":"2024-04-25T16:32:10.016068+0000","last_undegraded":"2024-04-25T16:32:10.016068+0000","last_fullsized":"2024-04-25T16:32:10.016068+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:00:39.373266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"83'11","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530879+0000","last_change":"2024-04-25T16:30:09.994870+0000","last_active":"2024-04-25T16:32:10.530879+0000","last_peered":"2024-04-25T16:32:10.530879+0000","last_clean":"2024-04-25T16:32:10.530879+0000","last_became_active":"2024-04-25T16:30:09.994521+0000","last_became_peered":"2024-04-25T16:30:09.994521+0000","last_unstale":"2024-04-25T16:32:10.530879+0000","last_undegraded":"2024-04-25T16:32:10.530879+0000","last_fullsized":"2024-04-25T16:32:10.530879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:18.093904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"83'16","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.015411+0000","last_change":"2024-04-25T16:31:09.010148+0000","last_active":"2024-04-25T16:32:10.015411+0000","last_peered":"2024-04-25T16:32:10.015411+0000","last_clean":"2024-04-25T16:32:10.015411+0000","last_became_active":"2024-04-25T16:31:09.009655+0000","last_became_peered":"2024-04-25T16:31:09.009655+0000","last_unstale":"2024-04-25T16:32:10.015411+0000","last_undegraded":"2024-04-25T16:32:10.015411+0000","last_fullsized":"2024-04-25T16:32:10.015411+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:18.637202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022213+0000","last_change":"2024-04-25T16:30:09.999519+0000","last_active":"2024-04-25T16:32:10.022213+0000","last_peered":"2024-04-25T16:32:10.022213+0000","last_clean":"2024-04-25T16:32:10.022213+0000","last_became_active":"2024-04-25T16:30:09.999158+0000","last_became_peered":"2024-04-25T16:30:09.999158+0000","last_unstale":"2024-04-25T16:32:10.022213+0000","last_undegraded":"2024-04-25T16:32:10.022213+0000","last_fullsized":"2024-04-25T16:32:10.022213+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:38:06.686541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"83'7","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212538+0000","last_change":"2024-04-25T16:30:10.005455+0000","last_active":"2024-04-25T16:32:10.212538+0000","last_peered":"2024-04-25T16:32:10.212538+0000","last_clean":"2024-04-25T16:32:10.212538+0000","last_became_active":"2024-04-25T16:30:10.004891+0000","last_became_peered":"2024-04-25T16:30:10.004891+0000","last_unstale":"2024-04-25T16:32:10.212538+0000","last_undegraded":"2024-04-25T16:32:10.212538+0000","last_fullsized":"2024-04-25T16:32:10.212538+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:29:25.598321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"83'10","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014090+0000","last_change":"2024-04-25T16:30:09.992870+0000","last_active":"2024-04-25T16:32:10.014090+0000","last_peered":"2024-04-25T16:32:10.014090+0000","last_clean":"2024-04-25T16:32:10.014090+0000","last_became_active":"2024-04-25T16:30:09.992399+0000","last_became_peered":"2024-04-25T16:30:09.992399+0000","last_unstale":"2024-04-25T16:32:10.014090+0000","last_undegraded":"2024-04-25T16:32:10.014090+0000","last_fullsized":"2024-04-25T16:32:10.014090+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.127287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781631+0000","last_change":"2024-04-25T16:30:09.988707+0000","last_active":"2024-04-25T16:32:10.781631+0000","last_peered":"2024-04-25T16:32:10.781631+0000","last_clean":"2024-04-25T16:32:10.781631+0000","last_became_active":"2024-04-25T16:30:09.988386+0000","last_became_peered":"2024-04-25T16:30:09.988386+0000","last_unstale":"2024-04-25T16:32:10.781631+0000","last_undegraded":"2024-04-25T16:32:10.781631+0000","last_fullsized":"2024-04-25T16:32:10.781631+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:03:15.973591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781712+0000","last_change":"2024-04-25T16:30:10.002780+0000","last_active":"2024-04-25T16:32:10.781712+0000","last_peered":"2024-04-25T16:32:10.781712+0000","last_clean":"2024-04-25T16:32:10.781712+0000","last_became_active":"2024-04-25T16:30:10.002055+0000","last_became_peered":"2024-04-25T16:30:10.002055+0000","last_unstale":"2024-04-25T16:32:10.781712+0000","last_undegraded":"2024-04-25T16:32:10.781712+0000","last_fullsized":"2024-04-25T16:32:10.781712+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:23.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530394+0000","last_change":"2024-04-25T16:30:09.998270+0000","last_active":"2024-04-25T16:32:10.530394+0000","last_peered":"2024-04-25T16:32:10.530394+0000","last_clean":"2024-04-25T16:32:10.530394+0000","last_became_active":"2024-04-25T16:30:09.997903+0000","last_became_peered":"2024-04-25T16:30:09.997903+0000","last_unstale":"2024-04-25T16:32:10.530394+0000","last_undegraded":"2024-04-25T16:32:10.530394+0000","last_fullsized":"2024-04-25T16:32:10.530394+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:19:14.887641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021350+0000","last_change":"2024-04-25T16:30:09.988894+0000","last_active":"2024-04-25T16:32:10.021350+0000","last_peered":"2024-04-25T16:32:10.021350+0000","last_clean":"2024-04-25T16:32:10.021350+0000","last_became_active":"2024-04-25T16:30:09.988508+0000","last_became_peered":"2024-04-25T16:30:09.988508+0000","last_unstale":"2024-04-25T16:32:10.021350+0000","last_undegraded":"2024-04-25T16:32:10.021350+0000","last_fullsized":"2024-04-25T16:32:10.021350+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:18:48.226193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"83'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018080+0000","last_change":"2024-04-25T16:30:09.989910+0000","last_active":"2024-04-25T16:32:10.018080+0000","last_peered":"2024-04-25T16:32:10.018080+0000","last_clean":"2024-04-25T16:32:10.018080+0000","last_became_active":"2024-04-25T16:30:09.989560+0000","last_became_peered":"2024-04-25T16:30:09.989560+0000","last_unstale":"2024-04-25T16:32:10.018080+0000","last_undegraded":"2024-04-25T16:32:10.018080+0000","last_fullsized":"2024-04-25T16:32:10.018080+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:53:12.834034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017567+0000","last_change":"2024-04-25T16:30:09.987930+0000","last_active":"2024-04-25T16:32:10.017567+0000","last_peered":"2024-04-25T16:32:10.017567+0000","last_clean":"2024-04-25T16:32:10.017567+0000","last_became_active":"2024-04-25T16:30:09.987595+0000","last_became_peered":"2024-04-25T16:30:09.987595+0000","last_unstale":"2024-04-25T16:32:10.017567+0000","last_undegraded":"2024-04-25T16:32:10.017567+0000","last_fullsized":"2024-04-25T16:32:10.017567+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:04.778172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016791+0000","last_change":"2024-04-25T16:30:14.006360+0000","last_active":"2024-04-25T16:32:10.016791+0000","last_peered":"2024-04-25T16:32:10.016791+0000","last_clean":"2024-04-25T16:32:10.016791+0000","last_became_active":"2024-04-25T16:30:14.005894+0000","last_became_peered":"2024-04-25T16:30:14.005894+0000","last_unstale":"2024-04-25T16:32:10.016791+0000","last_undegraded":"2024-04-25T16:32:10.016791+0000","last_fullsized":"2024-04-25T16:32:10.016791+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:33.150381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'14","reported_seq":164,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:33.234677+0000","last_change":"2024-04-25T16:30:08.992779+0000","last_active":"2024-04-25T16:32:33.234677+0000","last_peered":"2024-04-25T16:32:33.234677+0000","last_clean":"2024-04-25T16:32:33.234677+0000","last_became_active":"2024-04-25T16:30:08.992458+0000","last_became_peered":"2024-04-25T16:30:08.992458+0000","last_unstale":"2024-04-25T16:32:33.234677+0000","last_undegraded":"2024-04-25T16:32:33.234677+0000","last_fullsized":"2024-04-25T16:32:33.234677+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:21.430855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":118,"num_read_kb":118,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016381+0000","last_change":"2024-04-25T16:30:09.995353+0000","last_active":"2024-04-25T16:32:10.016381+0000","last_peered":"2024-04-25T16:32:10.016381+0000","last_clean":"2024-04-25T16:32:10.016381+0000","last_became_active":"2024-04-25T16:30:09.995011+0000","last_became_peered":"2024-04-25T16:30:09.995011+0000","last_unstale":"2024-04-25T16:32:10.016381+0000","last_undegraded":"2024-04-25T16:32:10.016381+0000","last_fullsized":"2024-04-25T16:32:10.016381+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:36:33.367268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211820+0000","last_change":"2024-04-25T16:31:09.187252+0000","last_active":"2024-04-25T16:32:10.211820+0000","last_peered":"2024-04-25T16:32:10.211820+0000","last_clean":"2024-04-25T16:32:10.211820+0000","last_became_active":"2024-04-25T16:31:09.186689+0000","last_became_peered":"2024-04-25T16:31:09.186689+0000","last_unstale":"2024-04-25T16:32:10.211820+0000","last_undegraded":"2024-04-25T16:32:10.211820+0000","last_fullsized":"2024-04-25T16:32:10.211820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:18.287476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016628+0000","last_change":"2024-04-25T16:30:08.997663+0000","last_active":"2024-04-25T16:32:10.016628+0000","last_peered":"2024-04-25T16:32:10.016628+0000","last_clean":"2024-04-25T16:32:10.016628+0000","last_became_active":"2024-04-25T16:30:08.997107+0000","last_became_peered":"2024-04-25T16:30:08.997107+0000","last_unstale":"2024-04-25T16:32:10.016628+0000","last_undegraded":"2024-04-25T16:32:10.016628+0000","last_fullsized":"2024-04-25T16:32:10.016628+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:27:25.856556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781991+0000","last_change":"2024-04-25T16:30:09.987947+0000","last_active":"2024-04-25T16:32:10.781991+0000","last_peered":"2024-04-25T16:32:10.781991+0000","last_clean":"2024-04-25T16:32:10.781991+0000","last_became_active":"2024-04-25T16:30:09.987575+0000","last_became_peered":"2024-04-25T16:30:09.987575+0000","last_unstale":"2024-04-25T16:32:10.781991+0000","last_undegraded":"2024-04-25T16:32:10.781991+0000","last_fullsized":"2024-04-25T16:32:10.781991+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:31:28.443249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211996+0000","last_change":"2024-04-25T16:30:14.006711+0000","last_active":"2024-04-25T16:32:10.211996+0000","last_peered":"2024-04-25T16:32:10.211996+0000","last_clean":"2024-04-25T16:32:10.211996+0000","last_became_active":"2024-04-25T16:30:14.005500+0000","last_became_peered":"2024-04-25T16:30:14.005500+0000","last_unstale":"2024-04-25T16:32:10.211996+0000","last_undegraded":"2024-04-25T16:32:10.211996+0000","last_fullsized":"2024-04-25T16:32:10.211996+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:58:26.288410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.034584+0000","last_change":"2024-04-25T16:32:10.034065+0000","last_active":"2024-04-25T16:32:10.034584+0000","last_peered":"2024-04-25T16:32:10.034584+0000","last_clean":"2024-04-25T16:32:10.034584+0000","last_became_active":"2024-04-25T16:32:10.022888+0000","last_became_peered":"2024-04-25T16:32:10.022888+0000","last_unstale":"2024-04-25T16:32:10.034584+0000","last_undegraded":"2024-04-25T16:32:10.034584+0000","last_fullsized":"2024-04-25T16:32:10.034584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:58:44.139220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530964+0000","last_change":"2024-04-25T16:30:09.988216+0000","last_active":"2024-04-25T16:32:10.530964+0000","last_peered":"2024-04-25T16:32:10.530964+0000","last_clean":"2024-04-25T16:32:10.530964+0000","last_became_active":"2024-04-25T16:30:09.987752+0000","last_became_peered":"2024-04-25T16:30:09.987752+0000","last_unstale":"2024-04-25T16:32:10.530964+0000","last_undegraded":"2024-04-25T16:32:10.530964+0000","last_fullsized":"2024-04-25T16:32:10.530964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:57:35.583000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.211839+0000","last_change":"2024-04-25T16:30:14.001329+0000","last_active":"2024-04-25T16:32:10.211839+0000","last_peered":"2024-04-25T16:32:10.211839+0000","last_clean":"2024-04-25T16:32:10.211839+0000","last_became_active":"2024-04-25T16:30:14.000878+0000","last_became_peered":"2024-04-25T16:30:14.000878+0000","last_unstale":"2024-04-25T16:32:10.211839+0000","last_undegraded":"2024-04-25T16:32:10.211839+0000","last_fullsized":"2024-04-25T16:32:10.211839+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:48:07.946611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020332+0000","last_change":"2024-04-25T16:30:08.989065+0000","last_active":"2024-04-25T16:32:10.020332+0000","last_peered":"2024-04-25T16:32:10.020332+0000","last_clean":"2024-04-25T16:32:10.020332+0000","last_became_active":"2024-04-25T16:30:08.988709+0000","last_became_peered":"2024-04-25T16:30:08.988709+0000","last_unstale":"2024-04-25T16:32:10.020332+0000","last_undegraded":"2024-04-25T16:32:10.020332+0000","last_fullsized":"2024-04-25T16:32:10.020332+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:34.306581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"83'15","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205194+0000","last_change":"2024-04-25T16:30:09.999641+0000","last_active":"2024-04-25T16:32:10.205194+0000","last_peered":"2024-04-25T16:32:10.205194+0000","last_clean":"2024-04-25T16:32:10.205194+0000","last_became_active":"2024-04-25T16:30:09.999319+0000","last_became_peered":"2024-04-25T16:30:09.999319+0000","last_unstale":"2024-04-25T16:32:10.205194+0000","last_undegraded":"2024-04-25T16:32:10.205194+0000","last_fullsized":"2024-04-25T16:32:10.205194+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:14.043201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017299+0000","last_change":"2024-04-25T16:30:13.999609+0000","last_active":"2024-04-25T16:32:10.017299+0000","last_peered":"2024-04-25T16:32:10.017299+0000","last_clean":"2024-04-25T16:32:10.017299+0000","last_became_active":"2024-04-25T16:30:13.999172+0000","last_became_peered":"2024-04-25T16:30:13.999172+0000","last_unstale":"2024-04-25T16:32:10.017299+0000","last_undegraded":"2024-04-25T16:32:10.017299+0000","last_fullsized":"2024-04-25T16:32:10.017299+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:49.878153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'8","reported_seq":158,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.240175+0000","last_change":"2024-04-25T16:30:08.988919+0000","last_active":"2024-04-25T16:32:31.240175+0000","last_peered":"2024-04-25T16:32:31.240175+0000","last_clean":"2024-04-25T16:32:31.240175+0000","last_became_active":"2024-04-25T16:30:08.988218+0000","last_became_peered":"2024-04-25T16:30:08.988218+0000","last_unstale":"2024-04-25T16:32:31.240175+0000","last_undegraded":"2024-04-25T16:32:31.240175+0000","last_fullsized":"2024-04-25T16:32:31.240175+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:57:57.046055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1647,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":119,"num_read_kb":4,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"83'7","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020493+0000","last_change":"2024-04-25T16:30:09.992498+0000","last_active":"2024-04-25T16:32:10.020493+0000","last_peered":"2024-04-25T16:32:10.020493+0000","last_clean":"2024-04-25T16:32:10.020493+0000","last_became_active":"2024-04-25T16:30:09.992190+0000","last_became_peered":"2024-04-25T16:30:09.992190+0000","last_unstale":"2024-04-25T16:32:10.020493+0000","last_undegraded":"2024-04-25T16:32:10.020493+0000","last_fullsized":"2024-04-25T16:32:10.020493+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:37:34.407963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017060+0000","last_change":"2024-04-25T16:30:14.007371+0000","last_active":"2024-04-25T16:32:10.017060+0000","last_peered":"2024-04-25T16:32:10.017060+0000","last_clean":"2024-04-25T16:32:10.017060+0000","last_became_active":"2024-04-25T16:30:14.006811+0000","last_became_peered":"2024-04-25T16:30:14.006811+0000","last_unstale":"2024-04-25T16:32:10.017060+0000","last_undegraded":"2024-04-25T16:32:10.017060+0000","last_fullsized":"2024-04-25T16:32:10.017060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:01.473142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014900+0000","last_change":"2024-04-25T16:30:08.984080+0000","last_active":"2024-04-25T16:32:10.014900+0000","last_peered":"2024-04-25T16:32:10.014900+0000","last_clean":"2024-04-25T16:32:10.014900+0000","last_became_active":"2024-04-25T16:30:08.983702+0000","last_became_peered":"2024-04-25T16:30:08.983702+0000","last_unstale":"2024-04-25T16:32:10.014900+0000","last_undegraded":"2024-04-25T16:32:10.014900+0000","last_fullsized":"2024-04-25T16:32:10.014900+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:22.311986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"83'8","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205200+0000","last_change":"2024-04-25T16:31:09.187653+0000","last_active":"2024-04-25T16:32:10.205200+0000","last_peered":"2024-04-25T16:32:10.205200+0000","last_clean":"2024-04-25T16:32:10.205200+0000","last_became_active":"2024-04-25T16:31:09.187183+0000","last_became_peered":"2024-04-25T16:31:09.187183+0000","last_unstale":"2024-04-25T16:32:10.205200+0000","last_undegraded":"2024-04-25T16:32:10.205200+0000","last_fullsized":"2024-04-25T16:32:10.205200+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:44:35.157126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3],"acting":[5,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021131+0000","last_change":"2024-04-25T16:30:14.003218+0000","last_active":"2024-04-25T16:32:10.021131+0000","last_peered":"2024-04-25T16:32:10.021131+0000","last_clean":"2024-04-25T16:32:10.021131+0000","last_became_active":"2024-04-25T16:30:14.002782+0000","last_became_peered":"2024-04-25T16:30:14.002782+0000","last_unstale":"2024-04-25T16:32:10.021131+0000","last_undegraded":"2024-04-25T16:32:10.021131+0000","last_fullsized":"2024-04-25T16:32:10.021131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:33:27.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013630+0000","last_change":"2024-04-25T16:30:08.993447+0000","last_active":"2024-04-25T16:32:10.013630+0000","last_peered":"2024-04-25T16:32:10.013630+0000","last_clean":"2024-04-25T16:32:10.013630+0000","last_became_active":"2024-04-25T16:30:08.993108+0000","last_became_peered":"2024-04-25T16:30:08.993108+0000","last_unstale":"2024-04-25T16:32:10.013630+0000","last_undegraded":"2024-04-25T16:32:10.013630+0000","last_fullsized":"2024-04-25T16:32:10.013630+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:48:00.511392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016351+0000","last_change":"2024-04-25T16:30:09.989863+0000","last_active":"2024-04-25T16:32:10.016351+0000","last_peered":"2024-04-25T16:32:10.016351+0000","last_clean":"2024-04-25T16:32:10.016351+0000","last_became_active":"2024-04-25T16:30:09.989520+0000","last_became_peered":"2024-04-25T16:30:09.989520+0000","last_unstale":"2024-04-25T16:32:10.016351+0000","last_undegraded":"2024-04-25T16:32:10.016351+0000","last_fullsized":"2024-04-25T16:32:10.016351+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:28:54.178298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781912+0000","last_change":"2024-04-25T16:30:13.999904+0000","last_active":"2024-04-25T16:32:10.781912+0000","last_peered":"2024-04-25T16:32:10.781912+0000","last_clean":"2024-04-25T16:32:10.781912+0000","last_became_active":"2024-04-25T16:30:13.999468+0000","last_became_peered":"2024-04-25T16:30:13.999468+0000","last_unstale":"2024-04-25T16:32:10.781912+0000","last_undegraded":"2024-04-25T16:32:10.781912+0000","last_fullsized":"2024-04-25T16:32:10.781912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:09.653365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204676+0000","last_change":"2024-04-25T16:30:08.987550+0000","last_active":"2024-04-25T16:32:10.204676+0000","last_peered":"2024-04-25T16:32:10.204676+0000","last_clean":"2024-04-25T16:32:10.204676+0000","last_became_active":"2024-04-25T16:30:08.987231+0000","last_became_peered":"2024-04-25T16:30:08.987231+0000","last_unstale":"2024-04-25T16:32:10.204676+0000","last_undegraded":"2024-04-25T16:32:10.204676+0000","last_fullsized":"2024-04-25T16:32:10.204676+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:20:01.110960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017634+0000","last_change":"2024-04-25T16:30:09.991985+0000","last_active":"2024-04-25T16:32:10.017634+0000","last_peered":"2024-04-25T16:32:10.017634+0000","last_clean":"2024-04-25T16:32:10.017634+0000","last_became_active":"2024-04-25T16:30:09.991645+0000","last_became_peered":"2024-04-25T16:30:09.991645+0000","last_unstale":"2024-04-25T16:32:10.017634+0000","last_undegraded":"2024-04-25T16:32:10.017634+0000","last_fullsized":"2024-04-25T16:32:10.017634+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:30.419416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016667+0000","last_change":"2024-04-25T16:30:13.999931+0000","last_active":"2024-04-25T16:32:10.016667+0000","last_peered":"2024-04-25T16:32:10.016667+0000","last_clean":"2024-04-25T16:32:10.016667+0000","last_became_active":"2024-04-25T16:30:13.997093+0000","last_became_peered":"2024-04-25T16:30:13.997093+0000","last_unstale":"2024-04-25T16:32:10.016667+0000","last_undegraded":"2024-04-25T16:32:10.016667+0000","last_fullsized":"2024-04-25T16:32:10.016667+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:58:31.973370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'7","reported_seq":264,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:30.837890+0000","last_change":"2024-04-25T16:30:08.985721+0000","last_active":"2024-04-25T16:32:30.837890+0000","last_peered":"2024-04-25T16:32:30.837890+0000","last_clean":"2024-04-25T16:32:30.837890+0000","last_became_active":"2024-04-25T16:30:08.985097+0000","last_became_peered":"2024-04-25T16:30:08.985097+0000","last_unstale":"2024-04-25T16:32:30.837890+0000","last_undegraded":"2024-04-25T16:32:30.837890+0000","last_fullsized":"2024-04-25T16:32:30.837890+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:00.713027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":226,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"83'10","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.531050+0000","last_change":"2024-04-25T16:30:09.992659+0000","last_active":"2024-04-25T16:32:10.531050+0000","last_peered":"2024-04-25T16:32:10.531050+0000","last_clean":"2024-04-25T16:32:10.531050+0000","last_became_active":"2024-04-25T16:30:09.985936+0000","last_became_peered":"2024-04-25T16:30:09.985936+0000","last_unstale":"2024-04-25T16:32:10.531050+0000","last_undegraded":"2024-04-25T16:32:10.531050+0000","last_fullsized":"2024-04-25T16:32:10.531050+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:21:10.808082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017564+0000","last_change":"2024-04-25T16:30:14.009352+0000","last_active":"2024-04-25T16:32:10.017564+0000","last_peered":"2024-04-25T16:32:10.017564+0000","last_clean":"2024-04-25T16:32:10.017564+0000","last_became_active":"2024-04-25T16:30:14.005250+0000","last_became_peered":"2024-04-25T16:30:14.005250+0000","last_unstale":"2024-04-25T16:32:10.017564+0000","last_undegraded":"2024-04-25T16:32:10.017564+0000","last_fullsized":"2024-04-25T16:32:10.017564+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:23.133435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022369+0000","last_change":"2024-04-25T16:30:08.990547+0000","last_active":"2024-04-25T16:32:10.022369+0000","last_peered":"2024-04-25T16:32:10.022369+0000","last_clean":"2024-04-25T16:32:10.022369+0000","last_became_active":"2024-04-25T16:30:08.989927+0000","last_became_peered":"2024-04-25T16:30:08.989927+0000","last_unstale":"2024-04-25T16:32:10.022369+0000","last_undegraded":"2024-04-25T16:32:10.022369+0000","last_fullsized":"2024-04-25T16:32:10.022369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:13:05.648102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781967+0000","last_change":"2024-04-25T16:30:10.002459+0000","last_active":"2024-04-25T16:32:10.781967+0000","last_peered":"2024-04-25T16:32:10.781967+0000","last_clean":"2024-04-25T16:32:10.781967+0000","last_became_active":"2024-04-25T16:30:10.002005+0000","last_became_peered":"2024-04-25T16:30:10.002005+0000","last_unstale":"2024-04-25T16:32:10.781967+0000","last_undegraded":"2024-04-25T16:32:10.781967+0000","last_fullsized":"2024-04-25T16:32:10.781967+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:54:01.220371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212172+0000","last_change":"2024-04-25T16:31:09.186538+0000","last_active":"2024-04-25T16:32:10.212172+0000","last_peered":"2024-04-25T16:32:10.212172+0000","last_clean":"2024-04-25T16:32:10.212172+0000","last_became_active":"2024-04-25T16:31:09.185930+0000","last_became_peered":"2024-04-25T16:31:09.185930+0000","last_unstale":"2024-04-25T16:32:10.212172+0000","last_undegraded":"2024-04-25T16:32:10.212172+0000","last_fullsized":"2024-04-25T16:32:10.212172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:14:38.930206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205179+0000","last_change":"2024-04-25T16:30:08.985264+0000","last_active":"2024-04-25T16:32:10.205179+0000","last_peered":"2024-04-25T16:32:10.205179+0000","last_clean":"2024-04-25T16:32:10.205179+0000","last_became_active":"2024-04-25T16:30:08.984901+0000","last_became_peered":"2024-04-25T16:30:08.984901+0000","last_unstale":"2024-04-25T16:32:10.205179+0000","last_undegraded":"2024-04-25T16:32:10.205179+0000","last_fullsized":"2024-04-25T16:32:10.205179+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:37:15.271981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530865+0000","last_change":"2024-04-25T16:30:09.997002+0000","last_active":"2024-04-25T16:32:10.530865+0000","last_peered":"2024-04-25T16:32:10.530865+0000","last_clean":"2024-04-25T16:32:10.530865+0000","last_became_active":"2024-04-25T16:30:09.996636+0000","last_became_peered":"2024-04-25T16:30:09.996636+0000","last_unstale":"2024-04-25T16:32:10.530865+0000","last_undegraded":"2024-04-25T16:32:10.530865+0000","last_fullsized":"2024-04-25T16:32:10.530865+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:32.522050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017760+0000","last_change":"2024-04-25T16:30:14.006403+0000","last_active":"2024-04-25T16:32:10.017760+0000","last_peered":"2024-04-25T16:32:10.017760+0000","last_clean":"2024-04-25T16:32:10.017760+0000","last_became_active":"2024-04-25T16:30:14.004816+0000","last_became_peered":"2024-04-25T16:30:14.004816+0000","last_unstale":"2024-04-25T16:32:10.017760+0000","last_undegraded":"2024-04-25T16:32:10.017760+0000","last_fullsized":"2024-04-25T16:32:10.017760+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:51:10.010002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":305,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:30.846875+0000","last_change":"2024-04-25T16:30:08.986119+0000","last_active":"2024-04-25T16:32:30.846875+0000","last_peered":"2024-04-25T16:32:30.846875+0000","last_clean":"2024-04-25T16:32:30.846875+0000","last_became_active":"2024-04-25T16:30:08.985806+0000","last_became_peered":"2024-04-25T16:30:08.985806+0000","last_unstale":"2024-04-25T16:32:30.846875+0000","last_undegraded":"2024-04-25T16:32:30.846875+0000","last_fullsized":"2024-04-25T16:32:30.846875+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:26:17.125886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64098,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":233,"num_read_kb":179,"num_write":42,"num_write_kb":82,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021027+0000","last_change":"2024-04-25T16:30:09.996891+0000","last_active":"2024-04-25T16:32:10.021027+0000","last_peered":"2024-04-25T16:32:10.021027+0000","last_clean":"2024-04-25T16:32:10.021027+0000","last_became_active":"2024-04-25T16:30:09.995851+0000","last_became_peered":"2024-04-25T16:30:09.995851+0000","last_unstale":"2024-04-25T16:32:10.021027+0000","last_undegraded":"2024-04-25T16:32:10.021027+0000","last_fullsized":"2024-04-25T16:32:10.021027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:19:42.602854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205467+0000","last_change":"2024-04-25T16:30:13.998277+0000","last_active":"2024-04-25T16:32:10.205467+0000","last_peered":"2024-04-25T16:32:10.205467+0000","last_clean":"2024-04-25T16:32:10.205467+0000","last_became_active":"2024-04-25T16:30:13.997313+0000","last_became_peered":"2024-04-25T16:30:13.997313+0000","last_unstale":"2024-04-25T16:32:10.205467+0000","last_undegraded":"2024-04-25T16:32:10.205467+0000","last_fullsized":"2024-04-25T16:32:10.205467+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:33:56.379806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017045+0000","last_change":"2024-04-25T16:30:08.986377+0000","last_active":"2024-04-25T16:32:10.017045+0000","last_peered":"2024-04-25T16:32:10.017045+0000","last_clean":"2024-04-25T16:32:10.017045+0000","last_became_active":"2024-04-25T16:30:08.985892+0000","last_became_peered":"2024-04-25T16:30:08.985892+0000","last_unstale":"2024-04-25T16:32:10.017045+0000","last_undegraded":"2024-04-25T16:32:10.017045+0000","last_fullsized":"2024-04-25T16:32:10.017045+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:53.651789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"83'9","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013775+0000","last_change":"2024-04-25T16:31:09.010551+0000","last_active":"2024-04-25T16:32:10.013775+0000","last_peered":"2024-04-25T16:32:10.013775+0000","last_clean":"2024-04-25T16:32:10.013775+0000","last_became_active":"2024-04-25T16:31:09.010141+0000","last_became_peered":"2024-04-25T16:31:09.010141+0000","last_unstale":"2024-04-25T16:32:10.013775+0000","last_undegraded":"2024-04-25T16:32:10.013775+0000","last_fullsized":"2024-04-25T16:32:10.013775+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:13:54.650080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,9],"acting":[6,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022080+0000","last_change":"2024-04-25T16:30:14.006064+0000","last_active":"2024-04-25T16:32:10.022080+0000","last_peered":"2024-04-25T16:32:10.022080+0000","last_clean":"2024-04-25T16:32:10.022080+0000","last_became_active":"2024-04-25T16:30:14.005545+0000","last_became_peered":"2024-04-25T16:30:14.005545+0000","last_unstale":"2024-04-25T16:32:10.022080+0000","last_undegraded":"2024-04-25T16:32:10.022080+0000","last_fullsized":"2024-04-25T16:32:10.022080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:59:36.611008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205079+0000","last_change":"2024-04-25T16:30:08.988413+0000","last_active":"2024-04-25T16:32:10.205079+0000","last_peered":"2024-04-25T16:32:10.205079+0000","last_clean":"2024-04-25T16:32:10.205079+0000","last_became_active":"2024-04-25T16:30:08.988075+0000","last_became_peered":"2024-04-25T16:30:08.988075+0000","last_unstale":"2024-04-25T16:32:10.205079+0000","last_undegraded":"2024-04-25T16:32:10.205079+0000","last_fullsized":"2024-04-25T16:32:10.205079+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:27:16.635876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"83'15","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022110+0000","last_change":"2024-04-25T16:30:09.997697+0000","last_active":"2024-04-25T16:32:10.022110+0000","last_peered":"2024-04-25T16:32:10.022110+0000","last_clean":"2024-04-25T16:32:10.022110+0000","last_became_active":"2024-04-25T16:30:09.997086+0000","last_became_peered":"2024-04-25T16:30:09.997086+0000","last_unstale":"2024-04-25T16:32:10.022110+0000","last_undegraded":"2024-04-25T16:32:10.022110+0000","last_fullsized":"2024-04-25T16:32:10.022110+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:19:35.345902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017958+0000","last_change":"2024-04-25T16:30:14.010486+0000","last_active":"2024-04-25T16:32:10.017958+0000","last_peered":"2024-04-25T16:32:10.017958+0000","last_clean":"2024-04-25T16:32:10.017958+0000","last_became_active":"2024-04-25T16:30:14.009273+0000","last_became_peered":"2024-04-25T16:30:14.009273+0000","last_unstale":"2024-04-25T16:32:10.017958+0000","last_undegraded":"2024-04-25T16:32:10.017958+0000","last_fullsized":"2024-04-25T16:32:10.017958+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:11.863575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'8","reported_seq":159,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:32.245066+0000","last_change":"2024-04-25T16:30:08.986661+0000","last_active":"2024-04-25T16:32:32.245066+0000","last_peered":"2024-04-25T16:32:32.245066+0000","last_clean":"2024-04-25T16:32:32.245066+0000","last_became_active":"2024-04-25T16:30:08.986306+0000","last_became_peered":"2024-04-25T16:30:08.986306+0000","last_unstale":"2024-04-25T16:32:32.245066+0000","last_undegraded":"2024-04-25T16:32:32.245066+0000","last_fullsized":"2024-04-25T16:32:32.245066+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:52:44.363693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1613,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":5,"num_write":8,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021934+0000","last_change":"2024-04-25T16:30:09.993443+0000","last_active":"2024-04-25T16:32:10.021934+0000","last_peered":"2024-04-25T16:32:10.021934+0000","last_clean":"2024-04-25T16:32:10.021934+0000","last_became_active":"2024-04-25T16:30:09.993031+0000","last_became_peered":"2024-04-25T16:30:09.993031+0000","last_unstale":"2024-04-25T16:32:10.021934+0000","last_undegraded":"2024-04-25T16:32:10.021934+0000","last_fullsized":"2024-04-25T16:32:10.021934+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:52:22.908713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014172+0000","last_change":"2024-04-25T16:30:14.003183+0000","last_active":"2024-04-25T16:32:10.014172+0000","last_peered":"2024-04-25T16:32:10.014172+0000","last_clean":"2024-04-25T16:32:10.014172+0000","last_became_active":"2024-04-25T16:30:14.002182+0000","last_became_peered":"2024-04-25T16:30:14.002182+0000","last_unstale":"2024-04-25T16:32:10.014172+0000","last_undegraded":"2024-04-25T16:32:10.014172+0000","last_fullsized":"2024-04-25T16:32:10.014172+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:51:08.616556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020096+0000","last_change":"2024-04-25T16:30:08.995860+0000","last_active":"2024-04-25T16:32:10.020096+0000","last_peered":"2024-04-25T16:32:10.020096+0000","last_clean":"2024-04-25T16:32:10.020096+0000","last_became_active":"2024-04-25T16:30:08.995487+0000","last_became_peered":"2024-04-25T16:30:08.995487+0000","last_unstale":"2024-04-25T16:32:10.020096+0000","last_undegraded":"2024-04-25T16:32:10.020096+0000","last_fullsized":"2024-04-25T16:32:10.020096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:34:48.206502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204879+0000","last_change":"2024-04-25T16:30:09.998820+0000","last_active":"2024-04-25T16:32:10.204879+0000","last_peered":"2024-04-25T16:32:10.204879+0000","last_clean":"2024-04-25T16:32:10.204879+0000","last_became_active":"2024-04-25T16:30:09.998448+0000","last_became_peered":"2024-04-25T16:30:09.998448+0000","last_unstale":"2024-04-25T16:32:10.204879+0000","last_undegraded":"2024-04-25T16:32:10.204879+0000","last_fullsized":"2024-04-25T16:32:10.204879+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:17:54.361250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204753+0000","last_change":"2024-04-25T16:30:13.992379+0000","last_active":"2024-04-25T16:32:10.204753+0000","last_peered":"2024-04-25T16:32:10.204753+0000","last_clean":"2024-04-25T16:32:10.204753+0000","last_became_active":"2024-04-25T16:30:13.991940+0000","last_became_peered":"2024-04-25T16:30:13.991940+0000","last_unstale":"2024-04-25T16:32:10.204753+0000","last_undegraded":"2024-04-25T16:32:10.204753+0000","last_fullsized":"2024-04-25T16:32:10.204753+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:48:59.152384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:27.942459+0000","last_change":"2024-04-25T16:30:08.987701+0000","last_active":"2024-04-25T16:32:27.942459+0000","last_peered":"2024-04-25T16:32:27.942459+0000","last_clean":"2024-04-25T16:32:27.942459+0000","last_became_active":"2024-04-25T16:30:08.987376+0000","last_became_peered":"2024-04-25T16:30:08.987376+0000","last_unstale":"2024-04-25T16:32:27.942459+0000","last_undegraded":"2024-04-25T16:32:27.942459+0000","last_fullsized":"2024-04-25T16:32:27.942459+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:18:21.810100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"83'5","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021777+0000","last_change":"2024-04-25T16:30:09.993005+0000","last_active":"2024-04-25T16:32:10.021777+0000","last_peered":"2024-04-25T16:32:10.021777+0000","last_clean":"2024-04-25T16:32:10.021777+0000","last_became_active":"2024-04-25T16:30:09.992619+0000","last_became_peered":"2024-04-25T16:30:09.992619+0000","last_unstale":"2024-04-25T16:32:10.021777+0000","last_undegraded":"2024-04-25T16:32:10.021777+0000","last_fullsized":"2024-04-25T16:32:10.021777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:39:40.998241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020416+0000","last_change":"2024-04-25T16:30:14.147895+0000","last_active":"2024-04-25T16:32:10.020416+0000","last_peered":"2024-04-25T16:32:10.020416+0000","last_clean":"2024-04-25T16:32:10.020416+0000","last_became_active":"2024-04-25T16:30:14.147298+0000","last_became_peered":"2024-04-25T16:30:14.147298+0000","last_unstale":"2024-04-25T16:32:10.020416+0000","last_undegraded":"2024-04-25T16:32:10.020416+0000","last_fullsized":"2024-04-25T16:32:10.020416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:06.269587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018035+0000","last_change":"2024-04-25T16:30:08.982683+0000","last_active":"2024-04-25T16:32:10.018035+0000","last_peered":"2024-04-25T16:32:10.018035+0000","last_clean":"2024-04-25T16:32:10.018035+0000","last_became_active":"2024-04-25T16:30:08.982236+0000","last_became_peered":"2024-04-25T16:30:08.982236+0000","last_unstale":"2024-04-25T16:32:10.018035+0000","last_undegraded":"2024-04-25T16:32:10.018035+0000","last_fullsized":"2024-04-25T16:32:10.018035+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:49:48.781791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.015356+0000","last_change":"2024-04-25T16:30:09.986734+0000","last_active":"2024-04-25T16:32:10.015356+0000","last_peered":"2024-04-25T16:32:10.015356+0000","last_clean":"2024-04-25T16:32:10.015356+0000","last_became_active":"2024-04-25T16:30:09.986337+0000","last_became_peered":"2024-04-25T16:30:09.986337+0000","last_unstale":"2024-04-25T16:32:10.015356+0000","last_undegraded":"2024-04-25T16:32:10.015356+0000","last_fullsized":"2024-04-25T16:32:10.015356+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.678009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013878+0000","last_change":"2024-04-25T16:30:14.003324+0000","last_active":"2024-04-25T16:32:10.013878+0000","last_peered":"2024-04-25T16:32:10.013878+0000","last_clean":"2024-04-25T16:32:10.013878+0000","last_became_active":"2024-04-25T16:30:14.002904+0000","last_became_peered":"2024-04-25T16:30:14.002904+0000","last_unstale":"2024-04-25T16:32:10.013878+0000","last_undegraded":"2024-04-25T16:32:10.013878+0000","last_fullsized":"2024-04-25T16:32:10.013878+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:28:28.750102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'6","reported_seq":155,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.232485+0000","last_change":"2024-04-25T16:30:08.993469+0000","last_active":"2024-04-25T16:32:31.232485+0000","last_peered":"2024-04-25T16:32:31.232485+0000","last_clean":"2024-04-25T16:32:31.232485+0000","last_became_active":"2024-04-25T16:30:08.993126+0000","last_became_peered":"2024-04-25T16:30:08.993126+0000","last_unstale":"2024-04-25T16:32:31.232485+0000","last_undegraded":"2024-04-25T16:32:31.232485+0000","last_fullsized":"2024-04-25T16:32:31.232485+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:10.274645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014980+0000","last_change":"2024-04-25T16:30:10.000597+0000","last_active":"2024-04-25T16:32:10.014980+0000","last_peered":"2024-04-25T16:32:10.014980+0000","last_clean":"2024-04-25T16:32:10.014980+0000","last_became_active":"2024-04-25T16:30:09.998892+0000","last_became_peered":"2024-04-25T16:30:09.998892+0000","last_unstale":"2024-04-25T16:32:10.014980+0000","last_undegraded":"2024-04-25T16:32:10.014980+0000","last_fullsized":"2024-04-25T16:32:10.014980+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:53:09.257682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013930+0000","last_change":"2024-04-25T16:30:14.147240+0000","last_active":"2024-04-25T16:32:10.013930+0000","last_peered":"2024-04-25T16:32:10.013930+0000","last_clean":"2024-04-25T16:32:10.013930+0000","last_became_active":"2024-04-25T16:30:14.146793+0000","last_became_peered":"2024-04-25T16:30:14.146793+0000","last_unstale":"2024-04-25T16:32:10.013930+0000","last_undegraded":"2024-04-25T16:32:10.013930+0000","last_fullsized":"2024-04-25T16:32:10.013930+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:42:55.713386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204864+0000","last_change":"2024-04-25T16:30:09.994595+0000","last_active":"2024-04-25T16:32:10.204864+0000","last_peered":"2024-04-25T16:32:10.204864+0000","last_clean":"2024-04-25T16:32:10.204864+0000","last_became_active":"2024-04-25T16:30:09.994305+0000","last_became_peered":"2024-04-25T16:30:09.994305+0000","last_unstale":"2024-04-25T16:32:10.204864+0000","last_undegraded":"2024-04-25T16:32:10.204864+0000","last_fullsized":"2024-04-25T16:32:10.204864+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:09:27.839500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'7","reported_seq":156,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:32.244380+0000","last_change":"2024-04-25T16:30:08.984203+0000","last_active":"2024-04-25T16:32:32.244380+0000","last_peered":"2024-04-25T16:32:32.244380+0000","last_clean":"2024-04-25T16:32:32.244380+0000","last_became_active":"2024-04-25T16:30:08.982883+0000","last_became_peered":"2024-04-25T16:30:08.982883+0000","last_unstale":"2024-04-25T16:32:32.244380+0000","last_undegraded":"2024-04-25T16:32:32.244380+0000","last_fullsized":"2024-04-25T16:32:32.244380+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:28:18.447624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":117,"num_read_kb":117,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022086+0000","last_change":"2024-04-25T16:31:09.017505+0000","last_active":"2024-04-25T16:32:10.022086+0000","last_peered":"2024-04-25T16:32:10.022086+0000","last_clean":"2024-04-25T16:32:10.022086+0000","last_became_active":"2024-04-25T16:31:09.011812+0000","last_became_peered":"2024-04-25T16:31:09.011812+0000","last_unstale":"2024-04-25T16:32:10.022086+0000","last_undegraded":"2024-04-25T16:32:10.022086+0000","last_fullsized":"2024-04-25T16:32:10.022086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:40:02.404908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,11,9],"acting":[7,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016285+0000","last_change":"2024-04-25T16:30:09.990998+0000","last_active":"2024-04-25T16:32:10.016285+0000","last_peered":"2024-04-25T16:32:10.016285+0000","last_clean":"2024-04-25T16:32:10.016285+0000","last_became_active":"2024-04-25T16:30:09.990644+0000","last_became_peered":"2024-04-25T16:30:09.990644+0000","last_unstale":"2024-04-25T16:32:10.016285+0000","last_undegraded":"2024-04-25T16:32:10.016285+0000","last_fullsized":"2024-04-25T16:32:10.016285+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:17:42.175520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212092+0000","last_change":"2024-04-25T16:30:08.988315+0000","last_active":"2024-04-25T16:32:10.212092+0000","last_peered":"2024-04-25T16:32:10.212092+0000","last_clean":"2024-04-25T16:32:10.212092+0000","last_became_active":"2024-04-25T16:30:08.987930+0000","last_became_peered":"2024-04-25T16:30:08.987930+0000","last_unstale":"2024-04-25T16:32:10.212092+0000","last_undegraded":"2024-04-25T16:32:10.212092+0000","last_fullsized":"2024-04-25T16:32:10.212092+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:17:44.884602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":106,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021552+0000","last_change":"2024-04-25T16:28:58.219510+0000","last_active":"2024-04-25T16:32:10.021552+0000","last_peered":"2024-04-25T16:32:10.021552+0000","last_clean":"2024-04-25T16:32:10.021552+0000","last_became_active":"2024-04-25T16:28:58.213888+0000","last_became_peered":"2024-04-25T16:28:58.213888+0000","last_unstale":"2024-04-25T16:32:10.021552+0000","last_undegraded":"2024-04-25T16:32:10.021552+0000","last_fullsized":"2024-04-25T16:32:10.021552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:25:33.939186+0000","last_clean_scrub_stamp":"2024-04-25T16:25:33.939186+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:15.378406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205344+0000","last_change":"2024-04-25T16:31:09.012831+0000","last_active":"2024-04-25T16:32:10.205344+0000","last_peered":"2024-04-25T16:32:10.205344+0000","last_clean":"2024-04-25T16:32:10.205344+0000","last_became_active":"2024-04-25T16:31:09.012375+0000","last_became_peered":"2024-04-25T16:31:09.012375+0000","last_unstale":"2024-04-25T16:32:10.205344+0000","last_undegraded":"2024-04-25T16:32:10.205344+0000","last_fullsized":"2024-04-25T16:32:10.205344+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:58:02.214734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,6,10],"acting":[5,8,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013551+0000","last_change":"2024-04-25T16:30:10.005210+0000","last_active":"2024-04-25T16:32:10.013551+0000","last_peered":"2024-04-25T16:32:10.013551+0000","last_clean":"2024-04-25T16:32:10.013551+0000","last_became_active":"2024-04-25T16:30:10.004936+0000","last_became_peered":"2024-04-25T16:30:10.004936+0000","last_unstale":"2024-04-25T16:32:10.013551+0000","last_undegraded":"2024-04-25T16:32:10.013551+0000","last_fullsized":"2024-04-25T16:32:10.013551+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:53.292675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021680+0000","last_change":"2024-04-25T16:30:08.990162+0000","last_active":"2024-04-25T16:32:10.021680+0000","last_peered":"2024-04-25T16:32:10.021680+0000","last_clean":"2024-04-25T16:32:10.021680+0000","last_became_active":"2024-04-25T16:30:08.989613+0000","last_became_peered":"2024-04-25T16:30:08.989613+0000","last_unstale":"2024-04-25T16:32:10.021680+0000","last_undegraded":"2024-04-25T16:32:10.021680+0000","last_fullsized":"2024-04-25T16:32:10.021680+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:37:41.595381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021251+0000","last_change":"2024-04-25T16:30:14.001850+0000","last_active":"2024-04-25T16:32:10.021251+0000","last_peered":"2024-04-25T16:32:10.021251+0000","last_clean":"2024-04-25T16:32:10.021251+0000","last_became_active":"2024-04-25T16:30:14.001348+0000","last_became_peered":"2024-04-25T16:30:14.001348+0000","last_unstale":"2024-04-25T16:32:10.021251+0000","last_undegraded":"2024-04-25T16:32:10.021251+0000","last_fullsized":"2024-04-25T16:32:10.021251+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:56:41.149783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021095+0000","last_change":"2024-04-25T16:30:08.995833+0000","last_active":"2024-04-25T16:32:10.021095+0000","last_peered":"2024-04-25T16:32:10.021095+0000","last_clean":"2024-04-25T16:32:10.021095+0000","last_became_active":"2024-04-25T16:30:08.995545+0000","last_became_peered":"2024-04-25T16:30:08.995545+0000","last_unstale":"2024-04-25T16:32:10.021095+0000","last_undegraded":"2024-04-25T16:32:10.021095+0000","last_fullsized":"2024-04-25T16:32:10.021095+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:31.452219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014096+0000","last_change":"2024-04-25T16:30:09.990225+0000","last_active":"2024-04-25T16:32:10.014096+0000","last_peered":"2024-04-25T16:32:10.014096+0000","last_clean":"2024-04-25T16:32:10.014096+0000","last_became_active":"2024-04-25T16:30:09.989859+0000","last_became_peered":"2024-04-25T16:30:09.989859+0000","last_unstale":"2024-04-25T16:32:10.014096+0000","last_undegraded":"2024-04-25T16:32:10.014096+0000","last_fullsized":"2024-04-25T16:32:10.014096+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:48:40.589395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016177+0000","last_change":"2024-04-25T16:30:14.147223+0000","last_active":"2024-04-25T16:32:10.016177+0000","last_peered":"2024-04-25T16:32:10.016177+0000","last_clean":"2024-04-25T16:32:10.016177+0000","last_became_active":"2024-04-25T16:30:14.146821+0000","last_became_peered":"2024-04-25T16:30:14.146821+0000","last_unstale":"2024-04-25T16:32:10.016177+0000","last_undegraded":"2024-04-25T16:32:10.016177+0000","last_fullsized":"2024-04-25T16:32:10.016177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:44:01.289165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'16","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:33.364360+0000","last_change":"2024-04-25T16:32:10.029459+0000","last_active":"2024-04-25T16:32:33.364360+0000","last_peered":"2024-04-25T16:32:33.364360+0000","last_clean":"2024-04-25T16:32:33.364360+0000","last_became_active":"2024-04-25T16:32:10.022574+0000","last_became_peered":"2024-04-25T16:32:10.022574+0000","last_unstale":"2024-04-25T16:32:33.364360+0000","last_undegraded":"2024-04-25T16:32:33.364360+0000","last_fullsized":"2024-04-25T16:32:33.364360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:42:41.675443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016438+0000","last_change":"2024-04-25T16:30:09.986088+0000","last_active":"2024-04-25T16:32:10.016438+0000","last_peered":"2024-04-25T16:32:10.016438+0000","last_clean":"2024-04-25T16:32:10.016438+0000","last_became_active":"2024-04-25T16:30:09.985256+0000","last_became_peered":"2024-04-25T16:30:09.985256+0000","last_unstale":"2024-04-25T16:32:10.016438+0000","last_undegraded":"2024-04-25T16:32:10.016438+0000","last_fullsized":"2024-04-25T16:32:10.016438+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:18:29.197558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016928+0000","last_change":"2024-04-25T16:30:14.000699+0000","last_active":"2024-04-25T16:32:10.016928+0000","last_peered":"2024-04-25T16:32:10.016928+0000","last_clean":"2024-04-25T16:32:10.016928+0000","last_became_active":"2024-04-25T16:30:14.000255+0000","last_became_peered":"2024-04-25T16:30:14.000255+0000","last_unstale":"2024-04-25T16:32:10.016928+0000","last_undegraded":"2024-04-25T16:32:10.016928+0000","last_fullsized":"2024-04-25T16:32:10.016928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:33.436693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212550+0000","last_change":"2024-04-25T16:30:08.981926+0000","last_active":"2024-04-25T16:32:10.212550+0000","last_peered":"2024-04-25T16:32:10.212550+0000","last_clean":"2024-04-25T16:32:10.212550+0000","last_became_active":"2024-04-25T16:30:08.981590+0000","last_became_peered":"2024-04-25T16:30:08.981590+0000","last_unstale":"2024-04-25T16:32:10.212550+0000","last_undegraded":"2024-04-25T16:32:10.212550+0000","last_fullsized":"2024-04-25T16:32:10.212550+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:40:57.818774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781964+0000","last_change":"2024-04-25T16:30:10.003998+0000","last_active":"2024-04-25T16:32:10.781964+0000","last_peered":"2024-04-25T16:32:10.781964+0000","last_clean":"2024-04-25T16:32:10.781964+0000","last_became_active":"2024-04-25T16:30:10.003626+0000","last_became_peered":"2024-04-25T16:30:10.003626+0000","last_unstale":"2024-04-25T16:32:10.781964+0000","last_undegraded":"2024-04-25T16:32:10.781964+0000","last_fullsized":"2024-04-25T16:32:10.781964+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:31:10.686092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212401+0000","last_change":"2024-04-25T16:30:14.149278+0000","last_active":"2024-04-25T16:32:10.212401+0000","last_peered":"2024-04-25T16:32:10.212401+0000","last_clean":"2024-04-25T16:32:10.212401+0000","last_became_active":"2024-04-25T16:30:14.148654+0000","last_became_peered":"2024-04-25T16:30:14.148654+0000","last_unstale":"2024-04-25T16:32:10.212401+0000","last_undegraded":"2024-04-25T16:32:10.212401+0000","last_fullsized":"2024-04-25T16:32:10.212401+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:11:24.823222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530250+0000","last_change":"2024-04-25T16:30:08.981612+0000","last_active":"2024-04-25T16:32:10.530250+0000","last_peered":"2024-04-25T16:32:10.530250+0000","last_clean":"2024-04-25T16:32:10.530250+0000","last_became_active":"2024-04-25T16:30:08.981222+0000","last_became_peered":"2024-04-25T16:30:08.981222+0000","last_unstale":"2024-04-25T16:32:10.530250+0000","last_undegraded":"2024-04-25T16:32:10.530250+0000","last_fullsized":"2024-04-25T16:32:10.530250+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:14.914658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"83'2","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.019805+0000","last_change":"2024-04-25T16:30:09.989592+0000","last_active":"2024-04-25T16:32:10.019805+0000","last_peered":"2024-04-25T16:32:10.019805+0000","last_clean":"2024-04-25T16:32:10.019805+0000","last_became_active":"2024-04-25T16:30:09.989222+0000","last_became_peered":"2024-04-25T16:30:09.989222+0000","last_unstale":"2024-04-25T16:32:10.019805+0000","last_undegraded":"2024-04-25T16:32:10.019805+0000","last_fullsized":"2024-04-25T16:32:10.019805+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:26:52.403239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.018128+0000","last_change":"2024-04-25T16:30:14.010480+0000","last_active":"2024-04-25T16:32:10.018128+0000","last_peered":"2024-04-25T16:32:10.018128+0000","last_clean":"2024-04-25T16:32:10.018128+0000","last_became_active":"2024-04-25T16:30:14.009023+0000","last_became_peered":"2024-04-25T16:30:14.009023+0000","last_unstale":"2024-04-25T16:32:10.018128+0000","last_undegraded":"2024-04-25T16:32:10.018128+0000","last_fullsized":"2024-04-25T16:32:10.018128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:05:10.311916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017158+0000","last_change":"2024-04-25T16:30:08.986849+0000","last_active":"2024-04-25T16:32:10.017158+0000","last_peered":"2024-04-25T16:32:10.017158+0000","last_clean":"2024-04-25T16:32:10.017158+0000","last_became_active":"2024-04-25T16:30:08.986479+0000","last_became_peered":"2024-04-25T16:30:08.986479+0000","last_unstale":"2024-04-25T16:32:10.017158+0000","last_undegraded":"2024-04-25T16:32:10.017158+0000","last_fullsized":"2024-04-25T16:32:10.017158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:42:12.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022094+0000","last_change":"2024-04-25T16:30:09.990927+0000","last_active":"2024-04-25T16:32:10.022094+0000","last_peered":"2024-04-25T16:32:10.022094+0000","last_clean":"2024-04-25T16:32:10.022094+0000","last_became_active":"2024-04-25T16:30:09.990482+0000","last_became_peered":"2024-04-25T16:30:09.990482+0000","last_unstale":"2024-04-25T16:32:10.022094+0000","last_undegraded":"2024-04-25T16:32:10.022094+0000","last_fullsized":"2024-04-25T16:32:10.022094+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:18.649721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"81'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212337+0000","last_change":"2024-04-25T16:30:14.006159+0000","last_active":"2024-04-25T16:32:10.212337+0000","last_peered":"2024-04-25T16:32:10.212337+0000","last_clean":"2024-04-25T16:32:10.212337+0000","last_became_active":"2024-04-25T16:30:14.005413+0000","last_became_peered":"2024-04-25T16:30:14.005413+0000","last_unstale":"2024-04-25T16:32:10.212337+0000","last_undegraded":"2024-04-25T16:32:10.212337+0000","last_fullsized":"2024-04-25T16:32:10.212337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:29:52.934385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.026829+0000","last_change":"2024-04-25T16:30:08.986207+0000","last_active":"2024-04-25T16:32:31.026829+0000","last_peered":"2024-04-25T16:32:31.026829+0000","last_clean":"2024-04-25T16:32:31.026829+0000","last_became_active":"2024-04-25T16:30:08.985857+0000","last_became_peered":"2024-04-25T16:30:08.985857+0000","last_unstale":"2024-04-25T16:32:31.026829+0000","last_undegraded":"2024-04-25T16:32:31.026829+0000","last_fullsized":"2024-04-25T16:32:31.026829+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:37:45.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016835+0000","last_change":"2024-04-25T16:30:09.993566+0000","last_active":"2024-04-25T16:32:10.016835+0000","last_peered":"2024-04-25T16:32:10.016835+0000","last_clean":"2024-04-25T16:32:10.016835+0000","last_became_active":"2024-04-25T16:30:09.992250+0000","last_became_peered":"2024-04-25T16:30:09.992250+0000","last_unstale":"2024-04-25T16:32:10.016835+0000","last_undegraded":"2024-04-25T16:32:10.016835+0000","last_fullsized":"2024-04-25T16:32:10.016835+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:33.279300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.212478+0000","last_change":"2024-04-25T16:30:14.147443+0000","last_active":"2024-04-25T16:32:10.212478+0000","last_peered":"2024-04-25T16:32:10.212478+0000","last_clean":"2024-04-25T16:32:10.212478+0000","last_became_active":"2024-04-25T16:30:14.146838+0000","last_became_peered":"2024-04-25T16:30:14.146838+0000","last_unstale":"2024-04-25T16:32:10.212478+0000","last_undegraded":"2024-04-25T16:32:10.212478+0000","last_fullsized":"2024-04-25T16:32:10.212478+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:51.532685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.020627+0000","last_change":"2024-04-25T16:31:09.015526+0000","last_active":"2024-04-25T16:32:10.020627+0000","last_peered":"2024-04-25T16:32:10.020627+0000","last_clean":"2024-04-25T16:32:10.020627+0000","last_became_active":"2024-04-25T16:31:09.007972+0000","last_became_peered":"2024-04-25T16:31:09.007972+0000","last_unstale":"2024-04-25T16:32:10.020627+0000","last_undegraded":"2024-04-25T16:32:10.020627+0000","last_fullsized":"2024-04-25T16:32:10.020627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:56.464426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"83'5","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022010+0000","last_change":"2024-04-25T16:30:09.998486+0000","last_active":"2024-04-25T16:32:10.022010+0000","last_peered":"2024-04-25T16:32:10.022010+0000","last_clean":"2024-04-25T16:32:10.022010+0000","last_became_active":"2024-04-25T16:30:09.998154+0000","last_became_peered":"2024-04-25T16:30:09.998154+0000","last_unstale":"2024-04-25T16:32:10.022010+0000","last_undegraded":"2024-04-25T16:32:10.022010+0000","last_fullsized":"2024-04-25T16:32:10.022010+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:55.467244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530492+0000","last_change":"2024-04-25T16:30:13.995864+0000","last_active":"2024-04-25T16:32:10.530492+0000","last_peered":"2024-04-25T16:32:10.530492+0000","last_clean":"2024-04-25T16:32:10.530492+0000","last_became_active":"2024-04-25T16:30:13.995326+0000","last_became_peered":"2024-04-25T16:30:13.995326+0000","last_unstale":"2024-04-25T16:32:10.530492+0000","last_undegraded":"2024-04-25T16:32:10.530492+0000","last_fullsized":"2024-04-25T16:32:10.530492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:11.718826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":156,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:31.265799+0000","last_change":"2024-04-25T16:30:08.984189+0000","last_active":"2024-04-25T16:32:31.265799+0000","last_peered":"2024-04-25T16:32:31.265799+0000","last_clean":"2024-04-25T16:32:31.265799+0000","last_became_active":"2024-04-25T16:30:08.983821+0000","last_became_peered":"2024-04-25T16:30:08.983821+0000","last_unstale":"2024-04-25T16:32:31.265799+0000","last_undegraded":"2024-04-25T16:32:31.265799+0000","last_fullsized":"2024-04-25T16:32:31.265799+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:37:51.315397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1663,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022285+0000","last_change":"2024-04-25T16:30:09.998763+0000","last_active":"2024-04-25T16:32:10.022285+0000","last_peered":"2024-04-25T16:32:10.022285+0000","last_clean":"2024-04-25T16:32:10.022285+0000","last_became_active":"2024-04-25T16:30:09.998439+0000","last_became_peered":"2024-04-25T16:30:09.998439+0000","last_unstale":"2024-04-25T16:32:10.022285+0000","last_undegraded":"2024-04-25T16:32:10.022285+0000","last_fullsized":"2024-04-25T16:32:10.022285+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:14.279883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781635+0000","last_change":"2024-04-25T16:30:14.005427+0000","last_active":"2024-04-25T16:32:10.781635+0000","last_peered":"2024-04-25T16:32:10.781635+0000","last_clean":"2024-04-25T16:32:10.781635+0000","last_became_active":"2024-04-25T16:30:14.005006+0000","last_became_peered":"2024-04-25T16:30:14.005006+0000","last_unstale":"2024-04-25T16:32:10.781635+0000","last_undegraded":"2024-04-25T16:32:10.781635+0000","last_fullsized":"2024-04-25T16:32:10.781635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:45:49.563365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.205387+0000","last_change":"2024-04-25T16:30:08.985557+0000","last_active":"2024-04-25T16:32:10.205387+0000","last_peered":"2024-04-25T16:32:10.205387+0000","last_clean":"2024-04-25T16:32:10.205387+0000","last_became_active":"2024-04-25T16:30:08.985215+0000","last_became_peered":"2024-04-25T16:30:08.985215+0000","last_unstale":"2024-04-25T16:32:10.205387+0000","last_undegraded":"2024-04-25T16:32:10.205387+0000","last_fullsized":"2024-04-25T16:32:10.205387+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:51.504089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022076+0000","last_change":"2024-04-25T16:30:09.991336+0000","last_active":"2024-04-25T16:32:10.022076+0000","last_peered":"2024-04-25T16:32:10.022076+0000","last_clean":"2024-04-25T16:32:10.022076+0000","last_became_active":"2024-04-25T16:30:09.990819+0000","last_became_peered":"2024-04-25T16:30:09.990819+0000","last_unstale":"2024-04-25T16:32:10.022076+0000","last_undegraded":"2024-04-25T16:32:10.022076+0000","last_fullsized":"2024-04-25T16:32:10.022076+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:45.988061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204595+0000","last_change":"2024-04-25T16:30:13.998463+0000","last_active":"2024-04-25T16:32:10.204595+0000","last_peered":"2024-04-25T16:32:10.204595+0000","last_clean":"2024-04-25T16:32:10.204595+0000","last_became_active":"2024-04-25T16:30:13.997952+0000","last_became_peered":"2024-04-25T16:30:13.997952+0000","last_unstale":"2024-04-25T16:32:10.204595+0000","last_undegraded":"2024-04-25T16:32:10.204595+0000","last_fullsized":"2024-04-25T16:32:10.204595+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:12.977068+0000","last_clean_scrub_stamp":"2024-04-25T16:30:12.977068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:18:58.507942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":49,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.536390+0000","last_change":"2024-04-25T16:32:10.535950+0000","last_active":"2024-04-25T16:32:10.536390+0000","last_peered":"2024-04-25T16:32:10.536390+0000","last_clean":"2024-04-25T16:32:10.536390+0000","last_became_active":"2024-04-25T16:32:10.531823+0000","last_became_peered":"2024-04-25T16:32:10.531823+0000","last_unstale":"2024-04-25T16:32:10.536390+0000","last_undegraded":"2024-04-25T16:32:10.536390+0000","last_fullsized":"2024-04-25T16:32:10.536390+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:07.954913+0000","last_clean_scrub_stamp":"2024-04-25T16:30:07.954913+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:36.471975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"83'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022474+0000","last_change":"2024-04-25T16:30:09.988566+0000","last_active":"2024-04-25T16:32:10.022474+0000","last_peered":"2024-04-25T16:32:10.022474+0000","last_clean":"2024-04-25T16:32:10.022474+0000","last_became_active":"2024-04-25T16:30:09.988164+0000","last_became_peered":"2024-04-25T16:30:09.988164+0000","last_unstale":"2024-04-25T16:32:10.022474+0000","last_undegraded":"2024-04-25T16:32:10.022474+0000","last_fullsized":"2024-04-25T16:32:10.022474+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:40.476766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"83'9","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022056+0000","last_change":"2024-04-25T16:30:09.996349+0000","last_active":"2024-04-25T16:32:10.022056+0000","last_peered":"2024-04-25T16:32:10.022056+0000","last_clean":"2024-04-25T16:32:10.022056+0000","last_became_active":"2024-04-25T16:30:09.994974+0000","last_became_peered":"2024-04-25T16:30:09.994974+0000","last_unstale":"2024-04-25T16:32:10.022056+0000","last_undegraded":"2024-04-25T16:32:10.022056+0000","last_fullsized":"2024-04-25T16:32:10.022056+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:46.929057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"83'12","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017091+0000","last_change":"2024-04-25T16:30:09.997837+0000","last_active":"2024-04-25T16:32:10.017091+0000","last_peered":"2024-04-25T16:32:10.017091+0000","last_clean":"2024-04-25T16:32:10.017091+0000","last_became_active":"2024-04-25T16:30:09.997361+0000","last_became_peered":"2024-04-25T16:30:09.997361+0000","last_unstale":"2024-04-25T16:32:10.017091+0000","last_undegraded":"2024-04-25T16:32:10.017091+0000","last_fullsized":"2024-04-25T16:32:10.017091+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:55:35.698528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016888+0000","last_change":"2024-04-25T16:30:09.991794+0000","last_active":"2024-04-25T16:32:10.016888+0000","last_peered":"2024-04-25T16:32:10.016888+0000","last_clean":"2024-04-25T16:32:10.016888+0000","last_became_active":"2024-04-25T16:30:09.991440+0000","last_became_peered":"2024-04-25T16:30:09.991440+0000","last_unstale":"2024-04-25T16:32:10.016888+0000","last_undegraded":"2024-04-25T16:32:10.016888+0000","last_fullsized":"2024-04-25T16:32:10.016888+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:11.322254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"83'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.781869+0000","last_change":"2024-04-25T16:30:10.002364+0000","last_active":"2024-04-25T16:32:10.781869+0000","last_peered":"2024-04-25T16:32:10.781869+0000","last_clean":"2024-04-25T16:32:10.781869+0000","last_became_active":"2024-04-25T16:30:10.001957+0000","last_became_peered":"2024-04-25T16:30:10.001957+0000","last_unstale":"2024-04-25T16:32:10.781869+0000","last_undegraded":"2024-04-25T16:32:10.781869+0000","last_fullsized":"2024-04-25T16:32:10.781869+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:36:19.150175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"83'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.014045+0000","last_change":"2024-04-25T16:30:10.004150+0000","last_active":"2024-04-25T16:32:10.014045+0000","last_peered":"2024-04-25T16:32:10.014045+0000","last_clean":"2024-04-25T16:32:10.014045+0000","last_became_active":"2024-04-25T16:30:10.003733+0000","last_became_peered":"2024-04-25T16:30:10.003733+0000","last_unstale":"2024-04-25T16:32:10.014045+0000","last_undegraded":"2024-04-25T16:32:10.014045+0000","last_fullsized":"2024-04-25T16:32:10.014045+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:20:11.850366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"83'6","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.021171+0000","last_change":"2024-04-25T16:30:09.991056+0000","last_active":"2024-04-25T16:32:10.021171+0000","last_peered":"2024-04-25T16:32:10.021171+0000","last_clean":"2024-04-25T16:32:10.021171+0000","last_became_active":"2024-04-25T16:30:09.988708+0000","last_became_peered":"2024-04-25T16:30:09.988708+0000","last_unstale":"2024-04-25T16:32:10.021171+0000","last_undegraded":"2024-04-25T16:32:10.021171+0000","last_fullsized":"2024-04-25T16:32:10.021171+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:01:13.062269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"83'15","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017612+0000","last_change":"2024-04-25T16:30:09.996571+0000","last_active":"2024-04-25T16:32:10.017612+0000","last_peered":"2024-04-25T16:32:10.017612+0000","last_clean":"2024-04-25T16:32:10.017612+0000","last_became_active":"2024-04-25T16:30:09.996206+0000","last_became_peered":"2024-04-25T16:30:09.996206+0000","last_unstale":"2024-04-25T16:32:10.017612+0000","last_undegraded":"2024-04-25T16:32:10.017612+0000","last_fullsized":"2024-04-25T16:32:10.017612+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:52:07.107949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"83'2","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.016603+0000","last_change":"2024-04-25T16:30:09.995067+0000","last_active":"2024-04-25T16:32:10.016603+0000","last_peered":"2024-04-25T16:32:10.016603+0000","last_clean":"2024-04-25T16:32:10.016603+0000","last_became_active":"2024-04-25T16:30:09.994678+0000","last_became_peered":"2024-04-25T16:30:09.994678+0000","last_unstale":"2024-04-25T16:32:10.016603+0000","last_undegraded":"2024-04-25T16:32:10.016603+0000","last_fullsized":"2024-04-25T16:32:10.016603+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:09.000869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"83'6","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.204798+0000","last_change":"2024-04-25T16:30:09.996821+0000","last_active":"2024-04-25T16:32:10.204798+0000","last_peered":"2024-04-25T16:32:10.204798+0000","last_clean":"2024-04-25T16:32:10.204798+0000","last_became_active":"2024-04-25T16:30:09.996464+0000","last_became_peered":"2024-04-25T16:30:09.996464+0000","last_unstale":"2024-04-25T16:32:10.204798+0000","last_undegraded":"2024-04-25T16:32:10.204798+0000","last_fullsized":"2024-04-25T16:32:10.204798+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:15:54.961544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"83'8","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.017023+0000","last_change":"2024-04-25T16:30:09.991733+0000","last_active":"2024-04-25T16:32:10.017023+0000","last_peered":"2024-04-25T16:32:10.017023+0000","last_clean":"2024-04-25T16:32:10.017023+0000","last_became_active":"2024-04-25T16:30:09.991401+0000","last_became_peered":"2024-04-25T16:30:09.991401+0000","last_unstale":"2024-04-25T16:32:10.017023+0000","last_undegraded":"2024-04-25T16:32:10.017023+0000","last_fullsized":"2024-04-25T16:32:10.017023+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:52:14.752597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"83'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.530708+0000","last_change":"2024-04-25T16:30:09.999087+0000","last_active":"2024-04-25T16:32:10.530708+0000","last_peered":"2024-04-25T16:32:10.530708+0000","last_clean":"2024-04-25T16:32:10.530708+0000","last_became_active":"2024-04-25T16:30:09.998725+0000","last_became_peered":"2024-04-25T16:30:09.998725+0000","last_unstale":"2024-04-25T16:32:10.530708+0000","last_undegraded":"2024-04-25T16:32:10.530708+0000","last_fullsized":"2024-04-25T16:32:10.530708+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:19.155171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"83'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.013940+0000","last_change":"2024-04-25T16:30:09.995880+0000","last_active":"2024-04-25T16:32:10.013940+0000","last_peered":"2024-04-25T16:32:10.013940+0000","last_clean":"2024-04-25T16:32:10.013940+0000","last_became_active":"2024-04-25T16:30:09.995514+0000","last_became_peered":"2024-04-25T16:30:09.995514+0000","last_unstale":"2024-04-25T16:32:10.013940+0000","last_undegraded":"2024-04-25T16:32:10.013940+0000","last_fullsized":"2024-04-25T16:32:10.013940+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:04:50.804020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"83'8","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-25T16:32:10.022174+0000","last_change":"2024-04-25T16:30:09.998326+0000","last_active":"2024-04-25T16:32:10.022174+0000","last_peered":"2024-04-25T16:32:10.022174+0000","last_clean":"2024-04-25T16:32:10.022174+0000","last_became_active":"2024-04-25T16:30:09.995803+0000","last_became_peered":"2024-04-25T16:30:09.995803+0000","last_unstale":"2024-04-25T16:32:10.022174+0000","last_undegraded":"2024-04-25T16:32:10.022174+0000","last_fullsized":"2024-04-25T16:32:10.022174+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T16:30:08.960513+0000","last_clean_scrub_stamp":"2024-04-25T16:30:08.960513+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:07:46.983947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73556,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1186,"num_read_kb":672,"num_write":273,"num_write_kb":164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":225764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":235,"ondisk_log_size":235,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":72,"seq":309237645354,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":475742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":66,"seq":283467841583,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36972,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708180,"statfs":{"total":95995035648,"available":95957176320,"internally_reserved":0,"allocated":1671168,"data_stored":1130298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037812,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":479275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234041,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":991232,"data_stored":475557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1074206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":471176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36352,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708800,"statfs":{"total":95995035648,"available":95957811200,"internally_reserved":0,"allocated":1040384,"data_stored":536304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":473023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":537549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":983040,"data_stored":470391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36292,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708860,"statfs":{"total":95995035648,"available":95957872640,"internally_reserved":0,"allocated":978944,"data_stored":475782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790025,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36892,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708260,"statfs":{"total":95995035648,"available":95957258240,"internally_reserved":0,"allocated":1593344,"data_stored":1064174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":71048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":67588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":66859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:32:37.985 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/3303702968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34104470 msgr2=0x7f3d3410abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/3303702968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34104470 0x7f3d3410abb0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009ca0 tx=0x7f3d2c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/3303702968 shutdown_connections 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/3303702968 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34104470 0x7f3d3410abb0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/3303702968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d34103930 0x7f3d34103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/3303702968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d34102f50 0x7f3d34103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/3303702968 >> 172.21.15.98:0/3303702968 conn(0x7f3d340fd370 msgr2=0x7f3d340ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/3303702968 shutdown_connections 2024-04-25T16:32:38.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.131+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/3303702968 wait complete. 2024-04-25T16:32:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d3b4c6640 1 Processor -- start 2024-04-25T16:32:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d3b4c6640 1 -- start start 2024-04-25T16:32:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d3b4c6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d34102f50 0x7f3d34102210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d3b4c6640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34103930 0x7f3d3419ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d3b4c6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d34104470 0x7f3d341a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d3b4c6640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410e4a0 con 0x7f3d34103930 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d3b4c6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410e320 con 0x7f3d34102f50 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d3b4c6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410e620 con 0x7f3d34104470 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d38a61640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34103930 0x7f3d3419ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d38a61640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34103930 0x7f3d3419ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37160/0 (socket says 172.21.15.98:37160) 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d38a61640 1 -- 172.21.15.98:0/1036466656 learned_addr learned my addr 172.21.15.98:0/1036466656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d39262640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d34102f50 0x7f3d34102210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.132+0000 7f3d39a63640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d34104470 0x7f3d341a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d38a61640 1 -- 172.21.15.98:0/1036466656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d34104470 msgr2=0x7f3d341a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d38a61640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d34104470 0x7f3d341a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d38a61640 1 -- 172.21.15.98:0/1036466656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d34102f50 msgr2=0x7f3d34102210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d38a61640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d34102f50 0x7f3d34102210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d38a61640 1 -- 172.21.15.98:0/1036466656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d341a1c00 con 0x7f3d34103930 2024-04-25T16:32:38.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d38a61640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34103930 0x7f3d3419ee70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000b4a0 tx=0x7f3d3000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d267fc640 1 -- 172.21.15.98:0/1036466656 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30014070 con 0x7f3d34103930 2024-04-25T16:32:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d267fc640 1 -- 172.21.15.98:0/1036466656 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d30010aa0 con 0x7f3d34103930 2024-04-25T16:32:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d341a1ef0 con 0x7f3d34103930 2024-04-25T16:32:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d267fc640 1 -- 172.21.15.98:0/1036466656 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30017b70 con 0x7f3d34103930 2024-04-25T16:32:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.133+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d341a2300 con 0x7f3d34103930 2024-04-25T16:32:38.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.134+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d34109900 con 0x7f3d34103930 2024-04-25T16:32:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.135+0000 7f3d267fc640 1 -- 172.21.15.98:0/1036466656 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f3d30017070 con 0x7f3d34103930 2024-04-25T16:32:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.136+0000 7f3d267fc640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3d200afb80 0x7f3d200b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.136+0000 7f3d39262640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3d200afb80 0x7f3d200b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.136+0000 7f3d267fc640 1 -- 172.21.15.98:0/1036466656 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f3d300de740 con 0x7f3d34103930 2024-04-25T16:32:38.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.137+0000 7f3d39262640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3d200afb80 0x7f3d200b2090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3d3406b530 tx=0x7f3d2800b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:38.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.140+0000 7f3d267fc640 1 -- 172.21.15.98:0/1036466656 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d300a68d0 con 0x7f3d34103930 2024-04-25T16:32:38.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:38 smithi098 ceph-mon[29162]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:38 smithi098 ceph-mon[29162]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:38 smithi098 ceph-mon[29162]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:38 smithi098 ceph-mon[29162]: pgmap v360: 129 pgs: 129 active+clean; 655 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-25T16:32:38.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.322+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d3402dd20 con 0x7f3d34103930 2024-04-25T16:32:38.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.324+0000 7f3d267fc640 1 -- 172.21.15.98:0/1036466656 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24806 (secure 0 0 0) 0x7f3d300aabd0 con 0x7f3d34103930 2024-04-25T16:32:38.324 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:38.324 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:32:10.010488+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3d200afb80 msgr2=0x7f3d200b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3d200afb80 0x7f3d200b2090 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3d3406b530 tx=0x7f3d2800b390 comp rx=0 tx=0).stop 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34103930 msgr2=0x7f3d3419ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34103930 0x7f3d3419ee70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000b4a0 tx=0x7f3d3000b960 comp rx=0 tx=0).stop 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 shutdown_connections 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f3d200afb80 0x7f3d200b2090 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3d34104470 0x7f3d341a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3d34103930 0x7f3d3419ee70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 --2- 172.21.15.98:0/1036466656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3d34102f50 0x7f3d34102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 >> 172.21.15.98:0/1036466656 conn(0x7f3d340fd370 msgr2=0x7f3d34106ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:38.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 shutdown_connections 2024-04-25T16:32:38.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.326+0000 7f3d3b4c6640 1 -- 172.21.15.98:0/1036466656 wait complete. 2024-04-25T16:32:38.338 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:32:38.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:32:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:38 smithi203 ceph-mon[33115]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:38 smithi203 ceph-mon[33115]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:38 smithi203 ceph-mon[33115]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:38 smithi203 ceph-mon[33115]: pgmap v360: 129 pgs: 129 active+clean; 655 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-25T16:32:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 -- 172.21.15.98:0/727274429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620104d20 msgr2=0x7f3620105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 --2- 172.21.15.98:0/727274429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620104d20 0x7f3620105130 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3610009900 tx=0x7f361002a3f0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 -- 172.21.15.98:0/727274429 shutdown_connections 2024-04-25T16:32:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 --2- 172.21.15.98:0/727274429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3620109a20 0x7f362010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 --2- 172.21.15.98:0/727274429 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620104d20 0x7f3620105130 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 --2- 172.21.15.98:0/727274429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3620103ae0 0x7f36201047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 -- 172.21.15.98:0/727274429 >> 172.21.15.98:0/727274429 conn(0x7f36200fecd0 msgr2=0x7f3620101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 -- 172.21.15.98:0/727274429 shutdown_connections 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.450+0000 7f3628661640 1 -- 172.21.15.98:0/727274429 wait complete. 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3628661640 1 Processor -- start 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3628661640 1 -- start start 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3628661640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3620103ae0 0x7f362019a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3628661640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3620104d20 0x7f362019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3628661640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620109a20 0x7f36201a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3628661640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f362010f4c0 con 0x7f3620109a20 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3628661640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f362010f340 con 0x7f3620103ae0 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3628661640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f362010f640 con 0x7f3620104d20 2024-04-25T16:32:38.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3626bfe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620109a20 0x7f36201a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3626bfe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620109a20 0x7f36201a1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37182/0 (socket says 172.21.15.98:37182) 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3626bfe640 1 -- 172.21.15.98:0/1010878978 learned_addr learned my addr 172.21.15.98:0/1010878978 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3625bfc640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3620104d20 0x7f362019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f36263fd640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3620103ae0 0x7f362019a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3626bfe640 1 -- 172.21.15.98:0/1010878978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3620104d20 msgr2=0x7f362019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3626bfe640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3620104d20 0x7f362019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3626bfe640 1 -- 172.21.15.98:0/1010878978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3620103ae0 msgr2=0x7f362019a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3626bfe640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3620103ae0 0x7f362019a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f3626bfe640 1 -- 172.21.15.98:0/1010878978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36201a1c70 con 0x7f3620109a20 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.451+0000 7f36263fd640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3620103ae0 0x7f362019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.452+0000 7f3626bfe640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620109a20 0x7f36201a1600 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f361800eaa0 tx=0x7f361800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:38.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.452+0000 7f36177fe640 1 -- 172.21.15.98:0/1010878978 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361800cdb0 con 0x7f3620109a20 2024-04-25T16:32:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.452+0000 7f36177fe640 1 -- 172.21.15.98:0/1010878978 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3618005020 con 0x7f3620109a20 2024-04-25T16:32:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.452+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36201a1f60 con 0x7f3620109a20 2024-04-25T16:32:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.452+0000 7f36177fe640 1 -- 172.21.15.98:0/1010878978 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36180116d0 con 0x7f3620109a20 2024-04-25T16:32:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.452+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36201a23c0 con 0x7f3620109a20 2024-04-25T16:32:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.453+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36200692e0 con 0x7f3620109a20 2024-04-25T16:32:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.454+0000 7f36177fe640 1 -- 172.21.15.98:0/1010878978 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f36180051d0 con 0x7f3620109a20 2024-04-25T16:32:38.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.455+0000 7f36177fe640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f360c0afb80 0x7f360c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.455+0000 7f36177fe640 1 -- 172.21.15.98:0/1010878978 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f36180df7b0 con 0x7f3620109a20 2024-04-25T16:32:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.455+0000 7f36263fd640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f360c0afb80 0x7f360c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.455+0000 7f36263fd640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f360c0afb80 0x7f360c0b2090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f36201046e0 tx=0x7f361c005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:38.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.458+0000 7f36177fe640 1 -- 172.21.15.98:0/1010878978 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3618016030 con 0x7f3620109a20 2024-04-25T16:32:38.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:38 smithi173 ceph-mon[32556]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:38 smithi173 ceph-mon[32556]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:38 smithi173 ceph-mon[32556]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:32:38.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:38 smithi173 ceph-mon[32556]: pgmap v360: 129 pgs: 129 active+clean; 655 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-25T16:32:38.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.635+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f362002dbd0 con 0x7f3620109a20 2024-04-25T16:32:38.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.635+0000 7f36177fe640 1 -- 172.21.15.98:0/1010878978 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f36180a7e80 con 0x7f3620109a20 2024-04-25T16:32:38.635 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:32:38.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f360c0afb80 msgr2=0x7f360c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f360c0afb80 0x7f360c0b2090 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f36201046e0 tx=0x7f361c005f90 comp rx=0 tx=0).stop 2024-04-25T16:32:38.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620109a20 msgr2=0x7f36201a1600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620109a20 0x7f36201a1600 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f361800eaa0 tx=0x7f361800ef60 comp rx=0 tx=0).stop 2024-04-25T16:32:38.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 shutdown_connections 2024-04-25T16:32:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f360c0afb80 0x7f360c0b2090 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3620109a20 0x7f36201a1600 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3620104d20 0x7f362019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 --2- 172.21.15.98:0/1010878978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3620103ae0 0x7f362019a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 >> 172.21.15.98:0/1010878978 conn(0x7f36200fecd0 msgr2=0x7f3620105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 shutdown_connections 2024-04-25T16:32:38.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.637+0000 7f3628661640 1 -- 172.21.15.98:0/1010878978 wait complete. 2024-04-25T16:32:38.648 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:32:38.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 -- 172.21.15.98:0/1275843376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294109890 msgr2=0x7fc29410bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 --2- 172.21.15.98:0/1275843376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294109890 0x7fc29410bcb0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc28c009900 tx=0x7fc28c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 -- 172.21.15.98:0/1275843376 shutdown_connections 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 --2- 172.21.15.98:0/1275843376 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294109890 0x7fc29410bcb0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 --2- 172.21.15.98:0/1275843376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc294106f30 0x7fc294109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 --2- 172.21.15.98:0/1275843376 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc2941045d0 0x7fc2941069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 -- 172.21.15.98:0/1275843376 >> 172.21.15.98:0/1275843376 conn(0x7fc2940fdbd0 msgr2=0x7fc294100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 -- 172.21.15.98:0/1275843376 shutdown_connections 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 -- 172.21.15.98:0/1275843376 wait complete. 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 Processor -- start 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.759+0000 7fc298d96640 1 -- start start 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc298d96640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2941045d0 0x7fc29419a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc298d96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294106f30 0x7fc29419a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc298d96640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc294109890 0x7fc2941a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc298d96640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc29410f4c0 con 0x7fc294106f30 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc298d96640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc29410f340 con 0x7fc2941045d0 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc298d96640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc29410f640 con 0x7fc294109890 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294106f30 0x7fc29419a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294106f30 0x7fc29419a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37200/0 (socket says 172.21.15.98:37200) 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 -- 172.21.15.98:0/3563205780 learned_addr learned my addr 172.21.15.98:0/3563205780 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc29259c640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2941045d0 0x7fc29419a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc292d9d640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc294109890 0x7fc2941a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 -- 172.21.15.98:0/3563205780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc294109890 msgr2=0x7fc2941a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc294109890 0x7fc2941a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 -- 172.21.15.98:0/3563205780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2941045d0 msgr2=0x7fc29419a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2941045d0 0x7fc29419a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 -- 172.21.15.98:0/3563205780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2941a1f00 con 0x7fc294106f30 2024-04-25T16:32:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc292d9d640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc294109890 0x7fc2941a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc291d9b640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294106f30 0x7fc29419a9b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc28800c920 tx=0x7fc28800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc2837fe640 1 -- 172.21.15.98:0/3563205780 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288007c70 con 0x7fc294106f30 2024-04-25T16:32:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc2837fe640 1 -- 172.21.15.98:0/3563205780 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc288007e10 con 0x7fc294106f30 2024-04-25T16:32:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.760+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2941a21f0 con 0x7fc294106f30 2024-04-25T16:32:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.761+0000 7fc2837fe640 1 -- 172.21.15.98:0/3563205780 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc288018710 con 0x7fc294106f30 2024-04-25T16:32:38.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.761+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2941a2740 con 0x7fc294106f30 2024-04-25T16:32:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.762+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc29406c2a0 con 0x7fc294106f30 2024-04-25T16:32:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.763+0000 7fc2837fe640 1 -- 172.21.15.98:0/3563205780 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fc28800f450 con 0x7fc294106f30 2024-04-25T16:32:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.763+0000 7fc2837fe640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc27c0afb80 0x7fc27c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.763+0000 7fc29259c640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc27c0afb80 0x7fc27c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.764+0000 7fc2837fe640 1 -- 172.21.15.98:0/3563205780 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fc2880df350 con 0x7fc294106f30 2024-04-25T16:32:38.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.764+0000 7fc29259c640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc27c0afb80 0x7fc27c0b2090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc284005c40 tx=0x7fc28400a570 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:38.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.768+0000 7fc2837fe640 1 -- 172.21.15.98:0/3563205780 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc288014030 con 0x7fc294106f30 2024-04-25T16:32:38.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.945+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc2941a2a00 con 0x7fc294106f30 2024-04-25T16:32:38.944 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.945+0000 7fc2837fe640 1 -- 172.21.15.98:0/3563205780 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7fc2880a7a20 con 0x7fc294106f30 2024-04-25T16:32:38.945 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc27c0afb80 msgr2=0x7fc27c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc27c0afb80 0x7fc27c0b2090 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc284005c40 tx=0x7fc28400a570 comp rx=0 tx=0).stop 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294106f30 msgr2=0x7fc29419a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294106f30 0x7fc29419a9b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc28800c920 tx=0x7fc28800cde0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 shutdown_connections 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc27c0afb80 0x7fc27c0b2090 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc294109890 0x7fc2941a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc294106f30 0x7fc29419a9b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 --2- 172.21.15.98:0/3563205780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2941045d0 0x7fc29419a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:38.947 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 >> 172.21.15.98:0/3563205780 conn(0x7fc2940fdbd0 msgr2=0x7fc294105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:38.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 shutdown_connections 2024-04-25T16:32:38.948 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:38.947+0000 7fc298d96640 1 -- 172.21.15.98:0/3563205780 wait complete. 2024-04-25T16:32:38.958 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:32:39.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 -- 172.21.15.98:0/3287878654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981033d0 msgr2=0x7fa6981037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/3287878654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981033d0 0x7fa6981037e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa688009900 tx=0x7fa68802a410 comp rx=0 tx=0).stop 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 -- 172.21.15.98:0/3287878654 shutdown_connections 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/3287878654 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6981080d0 0x7fa69810a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/3287878654 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981033d0 0x7fa6981037e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/3287878654 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa698102190 0x7fa698102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 -- 172.21.15.98:0/3287878654 >> 172.21.15.98:0/3287878654 conn(0x7fa6980fd380 msgr2=0x7fa6980ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 -- 172.21.15.98:0/3287878654 shutdown_connections 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.070+0000 7fa69f4b8640 1 -- 172.21.15.98:0/3287878654 wait complete. 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69f4b8640 1 Processor -- start 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69f4b8640 1 -- start start 2024-04-25T16:32:39.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69f4b8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa698102190 0x7fa69819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69f4b8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6981033d0 0x7fa69819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69f4b8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981080d0 0x7fa6981a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69f4b8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa69810e3e0 con 0x7fa6981080d0 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69f4b8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa69810e260 con 0x7fa698102190 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69f4b8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa69810e560 con 0x7fa6981033d0 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981080d0 0x7fa6981a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981080d0 0x7fa6981a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37218/0 (socket says 172.21.15.98:37218) 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 -- 172.21.15.98:0/2539681870 learned_addr learned my addr 172.21.15.98:0/2539681870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69d254640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa698102190 0x7fa69819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 -- 172.21.15.98:0/2539681870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6981033d0 msgr2=0x7fa69819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6981033d0 0x7fa69819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 -- 172.21.15.98:0/2539681870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa698102190 msgr2=0x7fa69819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa698102190 0x7fa69819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 -- 172.21.15.98:0/2539681870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6981a1cd0 con 0x7fa6981080d0 2024-04-25T16:32:39.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.071+0000 7fa69da55640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981080d0 0x7fa6981a1660 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa69000cad0 tx=0x7fa69000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:39.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.072+0000 7fa68e7fc640 1 -- 172.21.15.98:0/2539681870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69000ae20 con 0x7fa6981080d0 2024-04-25T16:32:39.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.072+0000 7fa68e7fc640 1 -- 172.21.15.98:0/2539681870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa690005020 con 0x7fa6981080d0 2024-04-25T16:32:39.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.072+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6981a1fc0 con 0x7fa6981080d0 2024-04-25T16:32:39.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.072+0000 7fa68e7fc640 1 -- 172.21.15.98:0/2539681870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69000f6b0 con 0x7fa6981080d0 2024-04-25T16:32:39.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.072+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6981a2430 con 0x7fa6981080d0 2024-04-25T16:32:39.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.073+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6981a26f0 con 0x7fa6981080d0 2024-04-25T16:32:39.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.075+0000 7fa68e7fc640 1 -- 172.21.15.98:0/2539681870 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fa6900051d0 con 0x7fa6981080d0 2024-04-25T16:32:39.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.075+0000 7fa68e7fc640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6840afb80 0x7fa6840b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.075+0000 7fa69d254640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6840afb80 0x7fa6840b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.075+0000 7fa68e7fc640 1 -- 172.21.15.98:0/2539681870 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa6900df5d0 con 0x7fa6981080d0 2024-04-25T16:32:39.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.076+0000 7fa69d254640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6840afb80 0x7fa6840b2090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa694005fd0 tx=0x7fa694005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:39.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.079+0000 7fa68e7fc640 1 -- 172.21.15.98:0/2539681870 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa690014030 con 0x7fa6981080d0 2024-04-25T16:32:39.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:39 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1036466656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:39.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:39 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1010878978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:39.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:39 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3563205780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:39.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.256+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa6981a28c0 con 0x7fa6981080d0 2024-04-25T16:32:39.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.257+0000 7fa68e7fc640 1 -- 172.21.15.98:0/2539681870 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7fa6900a7ca0 con 0x7fa6981080d0 2024-04-25T16:32:39.256 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:32:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6840afb80 msgr2=0x7fa6840b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6840afb80 0x7fa6840b2090 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa694005fd0 tx=0x7fa694005850 comp rx=0 tx=0).stop 2024-04-25T16:32:39.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981080d0 msgr2=0x7fa6981a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981080d0 0x7fa6981a1660 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa69000cad0 tx=0x7fa69000cf90 comp rx=0 tx=0).stop 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 shutdown_connections 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6840afb80 0x7fa6840b2090 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6981080d0 0x7fa6981a1660 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6981033d0 0x7fa69819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 --2- 172.21.15.98:0/2539681870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa698102190 0x7fa69819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 >> 172.21.15.98:0/2539681870 conn(0x7fa6980fd380 msgr2=0x7fa698109600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 shutdown_connections 2024-04-25T16:32:39.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.259+0000 7fa69f4b8640 1 -- 172.21.15.98:0/2539681870 wait complete. 2024-04-25T16:32:39.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:32:39.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 -- 172.21.15.98:0/1310187145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 msgr2=0x7efc58105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1310187145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 0x7efc58105150 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7efc48009900 tx=0x7efc4802a410 comp rx=0 tx=0).stop 2024-04-25T16:32:39.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 -- 172.21.15.98:0/1310187145 shutdown_connections 2024-04-25T16:32:39.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1310187145 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc58109a40 0x7efc5810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1310187145 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 0x7efc58105150 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1310187145 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58103b00 0x7efc58104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 -- 172.21.15.98:0/1310187145 >> 172.21.15.98:0/1310187145 conn(0x7efc580fed10 msgr2=0x7efc581011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 -- 172.21.15.98:0/1310187145 shutdown_connections 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.380+0000 7efc5fdce640 1 -- 172.21.15.98:0/1310187145 wait complete. 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5fdce640 1 Processor -- start 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5fdce640 1 -- start start 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5fdce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58103b00 0x7efc5819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5fdce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 0x7efc5819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5fdce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc58109a40 0x7efc581a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5fdce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7efc5810f4e0 con 0x7efc58104d40 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5fdce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7efc5810f360 con 0x7efc58103b00 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5fdce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7efc5810f660 con 0x7efc58109a40 2024-04-25T16:32:39.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5d369640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 0x7efc5819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5d369640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 0x7efc5819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37226/0 (socket says 172.21.15.98:37226) 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5d369640 1 -- 172.21.15.98:0/1120679758 learned_addr learned my addr 172.21.15.98:0/1120679758 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.381+0000 7efc5db6a640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58103b00 0x7efc5819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5e36b640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc58109a40 0x7efc581a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5d369640 1 -- 172.21.15.98:0/1120679758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc58109a40 msgr2=0x7efc581a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5d369640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc58109a40 0x7efc581a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5d369640 1 -- 172.21.15.98:0/1120679758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58103b00 msgr2=0x7efc5819a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5d369640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58103b00 0x7efc5819a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5d369640 1 -- 172.21.15.98:0/1120679758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc581a1c30 con 0x7efc58104d40 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5e36b640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc58109a40 0x7efc581a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5d369640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 0x7efc5819a6e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efc48009230 tx=0x7efc480042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc4effd640 1 -- 172.21.15.98:0/1120679758 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4803f030 con 0x7efc58104d40 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc4effd640 1 -- 172.21.15.98:0/1120679758 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efc48003c90 con 0x7efc58104d40 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc4effd640 1 -- 172.21.15.98:0/1120679758 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48035070 con 0x7efc58104d40 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc581a1ec0 con 0x7efc58104d40 2024-04-25T16:32:39.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.382+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc581a22d0 con 0x7efc58104d40 2024-04-25T16:32:39.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.383+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc58108850 con 0x7efc58104d40 2024-04-25T16:32:39.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.385+0000 7efc4effd640 1 -- 172.21.15.98:0/1120679758 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7efc48042030 con 0x7efc58104d40 2024-04-25T16:32:39.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.386+0000 7efc4effd640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc440afb80 0x7efc440b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.386+0000 7efc5db6a640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc440afb80 0x7efc440b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.386+0000 7efc4effd640 1 -- 172.21.15.98:0/1120679758 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7efc480ffc70 con 0x7efc58104d40 2024-04-25T16:32:39.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.386+0000 7efc5db6a640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc440afb80 0x7efc440b2090 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efc58104700 tx=0x7efc54005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:39.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.390+0000 7efc4effd640 1 -- 172.21.15.98:0/1120679758 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc480c8340 con 0x7efc58104d40 2024-04-25T16:32:39.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:39 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1036466656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:39 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1010878978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:39.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:39 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3563205780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:39.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:39 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1036466656' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:39.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:39 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1010878978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:32:39.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:39 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3563205780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:32:39.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.567+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efc5802dce0 con 0x7efc58104d40 2024-04-25T16:32:39.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.568+0000 7efc4effd640 1 -- 172.21.15.98:0/1120679758 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7efc480c7c40 con 0x7efc58104d40 2024-04-25T16:32:39.568 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:32:39.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc440afb80 msgr2=0x7efc440b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc440afb80 0x7efc440b2090 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7efc58104700 tx=0x7efc54005f90 comp rx=0 tx=0).stop 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 msgr2=0x7efc5819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 0x7efc5819a6e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7efc48009230 tx=0x7efc480042f0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 shutdown_connections 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7efc440afb80 0x7efc440b2090 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efc58109a40 0x7efc581a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7efc58104d40 0x7efc5819a6e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 --2- 172.21.15.98:0/1120679758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc58103b00 0x7efc5819a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 >> 172.21.15.98:0/1120679758 conn(0x7efc580fed10 msgr2=0x7efc58105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 shutdown_connections 2024-04-25T16:32:39.570 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.570+0000 7efc5fdce640 1 -- 172.21.15.98:0/1120679758 wait complete. 2024-04-25T16:32:39.581 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:32:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.688+0000 7f85a6fa8640 1 -- 172.21.15.98:0/2965879530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 msgr2=0x7f85a010bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.688+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/2965879530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 0x7f85a010bcb0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8598009ca0 tx=0x7f859802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.688+0000 7f85a6fa8640 1 -- 172.21.15.98:0/2965879530 shutdown_connections 2024-04-25T16:32:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.688+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/2965879530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 0x7f85a010bcb0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.688+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/2965879530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85a0101750 0x7f85a0101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.690 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.688+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/2965879530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85a01050a0 0x7f85a01054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 -- 172.21.15.98:0/2965879530 >> 172.21.15.98:0/2965879530 conn(0x7f85a00fd370 msgr2=0x7f85a00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 -- 172.21.15.98:0/2965879530 shutdown_connections 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 -- 172.21.15.98:0/2965879530 wait complete. 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 Processor -- start 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 -- start start 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85a0101750 0x7f85a019a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 0x7f85a019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85a01050a0 0x7f85a01a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f85a010f530 con 0x7f85a0102250 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f85a010f3b0 con 0x7f85a0101750 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f85a6fa8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f85a010f6b0 con 0x7f85a01050a0 2024-04-25T16:32:39.691 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f859ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 0x7f85a019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f859ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 0x7f85a019a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37248/0 (socket says 172.21.15.98:37248) 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.689+0000 7f859ffff640 1 -- 172.21.15.98:0/1268062799 learned_addr learned my addr 172.21.15.98:0/1268062799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f85a5545640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85a01050a0 0x7f85a01a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f85a4d44640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85a0101750 0x7f85a019a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859ffff640 1 -- 172.21.15.98:0/1268062799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85a01050a0 msgr2=0x7f85a01a1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859ffff640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85a01050a0 0x7f85a01a1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859ffff640 1 -- 172.21.15.98:0/1268062799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85a0101750 msgr2=0x7f85a019a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859ffff640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85a0101750 0x7f85a019a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859ffff640 1 -- 172.21.15.98:0/1268062799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a01a1b30 con 0x7f85a0102250 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859ffff640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 0x7f85a019a5e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f859400c920 tx=0x7f859400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859dffb640 1 -- 172.21.15.98:0/1268062799 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8594007c70 con 0x7f85a0102250 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859dffb640 1 -- 172.21.15.98:0/1268062799 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8594007e10 con 0x7f85a0102250 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a01a1e20 con 0x7f85a0102250 2024-04-25T16:32:39.692 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f859dffb640 1 -- 172.21.15.98:0/1268062799 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8594018710 con 0x7f85a0102250 2024-04-25T16:32:39.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.690+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a01a2290 con 0x7f85a0102250 2024-04-25T16:32:39.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.691+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a010aa40 con 0x7f85a0102250 2024-04-25T16:32:39.693 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.692+0000 7f859dffb640 1 -- 172.21.15.98:0/1268062799 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f859400f450 con 0x7f85a0102250 2024-04-25T16:32:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.693+0000 7f859dffb640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85880afb80 0x7f85880b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.693+0000 7f859dffb640 1 -- 172.21.15.98:0/1268062799 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f85940df350 con 0x7f85a0102250 2024-04-25T16:32:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.693+0000 7f85a4d44640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85880afb80 0x7f85880b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:39.694 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.693+0000 7f85a4d44640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85880afb80 0x7f85880b2090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8590005fd0 tx=0x7f8590005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:39.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.698+0000 7f859dffb640 1 -- 172.21.15.98:0/1268062799 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8594014030 con 0x7f85a0102250 2024-04-25T16:32:39.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.913+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85a002db50 con 0x7f85a0102250 2024-04-25T16:32:39.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.914+0000 7f859dffb640 1 -- 172.21.15.98:0/1268062799 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11935 (secure 0 0 0) 0x7f85940a7ac0 con 0x7f85a0102250 2024-04-25T16:32:39.913 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:39.913 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:32:32.844737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24494,24473,24457,24484,24539]},"id":1}]} 2024-04-25T16:32:39.914 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T16:32:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85880afb80 msgr2=0x7f85880b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85880afb80 0x7f85880b2090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8590005fd0 tx=0x7f8590005850 comp rx=0 tx=0).stop 2024-04-25T16:32:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 msgr2=0x7f85a019a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:39.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 0x7f85a019a5e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f859400c920 tx=0x7f859400cde0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 shutdown_connections 2024-04-25T16:32:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85880afb80 0x7f85880b2090 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85a01050a0 0x7f85a01a1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f85a0102250 0x7f85a019a5e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 --2- 172.21.15.98:0/1268062799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85a0101750 0x7f85a019a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 >> 172.21.15.98:0/1268062799 conn(0x7f85a00fd370 msgr2=0x7f85a0107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 shutdown_connections 2024-04-25T16:32:39.916 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:39.916+0000 7f85a6fa8640 1 -- 172.21.15.98:0/1268062799 wait complete. 2024-04-25T16:32:39.927 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:32:40.038 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.036+0000 7f1675ee7640 1 -- 172.21.15.98:0/369929197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 msgr2=0x7f1670104000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.036+0000 7f1675ee7640 1 --2- 172.21.15.98:0/369929197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 0x7f1670104000 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f1664009900 tx=0x7f166402a410 comp rx=0 tx=0).stop 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 -- 172.21.15.98:0/369929197 shutdown_connections 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 --2- 172.21.15.98:0/369929197 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1670104690 0x7f167010ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 --2- 172.21.15.98:0/369929197 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 0x7f1670104000 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 --2- 172.21.15.98:0/369929197 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670102fd0 0x7f16701033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 -- 172.21.15.98:0/369929197 >> 172.21.15.98:0/369929197 conn(0x7f16700fd370 msgr2=0x7f16700ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 -- 172.21.15.98:0/369929197 shutdown_connections 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 -- 172.21.15.98:0/369929197 wait complete. 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 Processor -- start 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.037+0000 7f1675ee7640 1 -- start start 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f1675ee7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670102fd0 0x7f1670199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f1675ee7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 0x7f167019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f1675ee7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f167019f4a0 0x7f16701a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f1675ee7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f167010f790 con 0x7f1670103b30 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f1675ee7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f167010f610 con 0x7f1670102fd0 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f1675ee7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f167010f910 con 0x7f167019f4a0 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 0x7f167019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 0x7f167019a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37268/0 (socket says 172.21.15.98:37268) 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 -- 172.21.15.98:0/1764628019 learned_addr learned my addr 172.21.15.98:0/1764628019 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166ffff640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f167019f4a0 0x7f16701a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166f7fe640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670102fd0 0x7f1670199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 -- 172.21.15.98:0/1764628019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f167019f4a0 msgr2=0x7f16701a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f167019f4a0 0x7f16701a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 -- 172.21.15.98:0/1764628019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670102fd0 msgr2=0x7f1670199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670102fd0 0x7f1670199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 -- 172.21.15.98:0/1764628019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16701a1f30 con 0x7f1670103b30 2024-04-25T16:32:40.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166f7fe640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670102fd0 0x7f1670199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166effd640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 0x7f167019a520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1664009230 tx=0x7f16640040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166cff9640 1 -- 172.21.15.98:0/1764628019 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166403f030 con 0x7f1670103b30 2024-04-25T16:32:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166cff9640 1 -- 172.21.15.98:0/1764628019 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16640044c0 con 0x7f1670103b30 2024-04-25T16:32:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16701a21c0 con 0x7f1670103b30 2024-04-25T16:32:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f166cff9640 1 -- 172.21.15.98:0/1764628019 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1664035070 con 0x7f1670103b30 2024-04-25T16:32:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.038+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16701a2730 con 0x7f1670103b30 2024-04-25T16:32:40.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.039+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167006b3a0 con 0x7f1670103b30 2024-04-25T16:32:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.041+0000 7f166cff9640 1 -- 172.21.15.98:0/1764628019 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f1664042030 con 0x7f1670103b30 2024-04-25T16:32:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.041+0000 7f166cff9640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f16580afb80 0x7f16580b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.041+0000 7f166f7fe640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f16580afb80 0x7f16580b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.041+0000 7f166cff9640 1 -- 172.21.15.98:0/1764628019 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f16640ffb10 con 0x7f1670103b30 2024-04-25T16:32:40.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.042+0000 7f166f7fe640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f16580afb80 0x7f16580b2090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1660006fd0 tx=0x7f1660008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.045+0000 7f166cff9640 1 -- 172.21.15.98:0/1764628019 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16640c81e0 con 0x7f1670103b30 2024-04-25T16:32:40.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.228+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f167002ddd0 con 0x7f1670103b30 2024-04-25T16:32:40.229 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.230+0000 7f166cff9640 1 -- 172.21.15.98:0/1764628019 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24806 (secure 0 0 0) 0x7f16640d0260 con 0x7f1670103b30 2024-04-25T16:32:40.229 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:32:40.229 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":85,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:32:10.010488+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:32:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f16580afb80 msgr2=0x7f16580b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f16580afb80 0x7f16580b2090 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1660006fd0 tx=0x7f1660008040 comp rx=0 tx=0).stop 2024-04-25T16:32:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 msgr2=0x7f167019a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 0x7f167019a520 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f1664009230 tx=0x7f16640040c0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 shutdown_connections 2024-04-25T16:32:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f16580afb80 0x7f16580b2090 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.231 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f167019f4a0 0x7f16701a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1670103b30 0x7f167019a520 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 --2- 172.21.15.98:0/1764628019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1670102fd0 0x7f1670199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 >> 172.21.15.98:0/1764628019 conn(0x7f16700fd370 msgr2=0x7f16700ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 shutdown_connections 2024-04-25T16:32:40.232 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.232+0000 7f1675ee7640 1 -- 172.21.15.98:0/1764628019 wait complete. 2024-04-25T16:32:40.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:40 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2539681870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:40.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:40 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1120679758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:40.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:40 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1268062799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:40.242 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:40 smithi098 ceph-mon[29162]: pgmap v361: 129 pgs: 129 active+clean; 655 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-25T16:32:40.243 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T16:32:40.244 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T16:32:40.261 INFO:tasks.workunit:Pulling workunits from ref 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:32:40.261 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T16:32:40.261 INFO:tasks.workunit:timeout=3h 2024-04-25T16:32:40.261 INFO:tasks.workunit:cleanup=True 2024-04-25T16:32:40.262 DEBUG:teuthology.orchestra.run.smithi098:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:40.262 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T16:32:40.262 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:32:40.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:32:40.263 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:32:40.277 INFO:teuthology.orchestra.run.smithi098.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:40.278 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-25T16:32:40.278 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 73h/115d Inode: 1 Links: 2 2024-04-25T16:32:40.278 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:32:40.278 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:fusefs_t:s0 2024-04-25T16:32:40.278 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-04-25 16:30:36.738627550 +0000 2024-04-25T16:32:40.278 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-04-25 16:30:36.738627550 +0000 2024-04-25T16:32:40.278 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-04-25 16:32:32.450223047 +0000 2024-04-25T16:32:40.278 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: - 2024-04-25T16:32:40.278 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T16:32:40.278 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T16:32:40.350 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:32:40.378 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.377+0000 7f0a16557640 1 -- 172.21.15.98:0/900648457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 msgr2=0x7f0a1010abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.377+0000 7f0a16557640 1 --2- 172.21.15.98:0/900648457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 0x7f0a1010abb0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f0a08009ca0 tx=0x7f0a0802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 -- 172.21.15.98:0/900648457 shutdown_connections 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 --2- 172.21.15.98:0/900648457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 0x7f0a1010abb0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 --2- 172.21.15.98:0/900648457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a10103940 0x7f0a10103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 --2- 172.21.15.98:0/900648457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a10102f60 0x7f0a10103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 -- 172.21.15.98:0/900648457 >> 172.21.15.98:0/900648457 conn(0x7f0a100fd340 msgr2=0x7f0a100ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 -- 172.21.15.98:0/900648457 shutdown_connections 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 -- 172.21.15.98:0/900648457 wait complete. 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 Processor -- start 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.378+0000 7f0a16557640 1 -- start start 2024-04-25T16:32:40.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a16557640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a10102f60 0x7f0a1019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a16557640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a10103940 0x7f0a1019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a16557640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 0x7f0a101a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a16557640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1010f600 con 0x7f0a10104460 2024-04-25T16:32:40.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a16557640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1010f480 con 0x7f0a10102f60 2024-04-25T16:32:40.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a16557640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1010f780 con 0x7f0a10103940 2024-04-25T16:32:40.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 0x7f0a101a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 0x7f0a101a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37290/0 (socket says 172.21.15.98:37290) 2024-04-25T16:32:40.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 -- 172.21.15.98:0/2104300561 learned_addr learned my addr 172.21.15.98:0/2104300561 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a10102f60 0x7f0a1019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a10103940 0x7f0a1019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 -- 172.21.15.98:0/2104300561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a10103940 msgr2=0x7f0a1019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a10103940 0x7f0a1019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 -- 172.21.15.98:0/2104300561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a10102f60 msgr2=0x7f0a1019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a10102f60 0x7f0a1019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 -- 172.21.15.98:0/2104300561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a101a1d20 con 0x7f0a10104460 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a14af4640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 0x7f0a101a16b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0a08031040 tx=0x7f0a08003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a0d7fa640 1 -- 172.21.15.98:0/2104300561 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08004240 con 0x7f0a10104460 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.379+0000 7f0a0d7fa640 1 -- 172.21.15.98:0/2104300561 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a08003e50 con 0x7f0a10104460 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.380+0000 7f0a16557640 1 -- 172.21.15.98:0/2104300561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a101a1fb0 con 0x7f0a10104460 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.380+0000 7f0a0d7fa640 1 -- 172.21.15.98:0/2104300561 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08030dd0 con 0x7f0a10104460 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.380+0000 7f0a16557640 1 -- 172.21.15.98:0/2104300561 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a101a24a0 con 0x7f0a10104460 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a1d407740 1 Processor -- start 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a0d7fa640 1 -- 172.21.15.98:0/2104300561 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f0a080305c0 con 0x7f0a10104460 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a1d407740 1 -- start start 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d6b410 0x55721610c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721610ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 0x557216113920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a1d407740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557216071b00 con 0x557215d42550 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a1d407740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55721600d920 con 0x557215d6b410 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.382+0000 7f0a1d407740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55721606dac0 con 0x557215d4a730 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721610ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721610ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37298/0 (socket says 172.21.15.98:37298) 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 -- 172.21.15.98:0/49702502 learned_addr learned my addr 172.21.15.98:0/49702502 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 0x557216113920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d6b410 0x55721610c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 -- 172.21.15.98:0/49702502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 msgr2=0x557216113920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 0x557216113920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 -- 172.21.15.98:0/49702502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d6b410 msgr2=0x55721610c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d6b410 0x55721610c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 -- 172.21.15.98:0/49702502 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557215e4b7e0 con 0x557215d42550 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d6b410 0x55721610c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0d7fa640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09f80afda0 0x7f09f80b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721610ca20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009a40 tx=0x7f0a00009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c001cc846c460d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09f80afda0 0x7f09f80b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f0a0d7fa640 1 -- 172.21.15.98:0/2104300561 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f0a08100790 con 0x7f0a10104460 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f09f67fc640 1 -- 172.21.15.98:0/49702502 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00017070 con 0x557215d42550 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f09f67fc640 1 -- 172.21.15.98:0/49702502 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a00013470 con 0x557215d42550 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.383+0000 7f09f67fc640 1 -- 172.21.15.98:0/49702502 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00012630 con 0x557215d42550 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.384+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09f80afda0 0x7f09f80b22b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55721610d670 tx=0x7f0a04009110 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.384+0000 7f0a1d407740 1 -- 172.21.15.98:0/49702502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 msgr2=0x55721610ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.384+0000 7f0a1d407740 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721610ca20 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009a40 tx=0x7f0a00009f00 comp rx=0 tx=0).stop 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 -- 172.21.15.98:0/49702502 shutdown_connections 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 0x557216113920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721610ca20 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 --2- 172.21.15.98:0/49702502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d6b410 0x55721610c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 -- 172.21.15.98:0/49702502 >> 172.21.15.98:0/49702502 conn(0x557215d5f560 msgr2=0x557216107e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 -- 172.21.15.98:0/49702502 shutdown_connections 2024-04-25T16:32:40.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 -- 172.21.15.98:0/49702502 wait complete. 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 -- start start 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.385+0000 7f0a1d407740 1 Processor -- start 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x5572161162a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d4a730 0x5572161167e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a1d407740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x557216114370 con 0x557215d4a730 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a1d407740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55721600b5d0 con 0x557215d42550 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a1d407740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5572161144f0 con 0x557215d6b410 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a1d407740 10 client.0 _ll_register_callbacks cb 0x557215ff9080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d4a730 0x5572161167e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d4a730 0x5572161167e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37306/0 (socket says 172.21.15.98:37306) 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 -- 172.21.15.98:0/3668091457 learned_addr learned my addr 172.21.15.98:0/3668091457 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x5572161162a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a14af4640 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 -- 172.21.15.98:0/3668091457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 msgr2=0x55721611d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 -- 172.21.15.98:0/3668091457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 msgr2=0x5572161162a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x5572161162a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 -- 172.21.15.98:0/3668091457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557215eabf70 con 0x557215d4a730 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d4a730 0x5572161167e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0a04015df0 tx=0x7f0a040047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa7754cb7706c82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f09ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3668091457 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04025030 con 0x557215d4a730 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3668091457 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f0a0400f040 con 0x557215d4a730 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.386+0000 7f0a1d407740 1 -- 172.21.15.98:0/3668091457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a00013470 con 0x557215d4a730 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3668091457 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a04032050 con 0x557215d4a730 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a1d407740 10 client.15690 fetch_fsmap learned FSMap version 29 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a1d407740 1 -- 172.21.15.98:0/3668091457 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557215d43170 con 0x557215d4a730 2024-04-25T16:32:40.392 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3668091457 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a00013470 con 0x557215d4a730 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3668091457 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0a04021ad0 con 0x557215d4a730 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a1d407740 10 client.15690 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a1d407740 10 client.15690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a1d407740 20 client.15690 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x557215d2a1f0 0x557216123d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a1d407740 4 client.15690 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a1d407740 1 -- 172.21.15.98:0/3668091457 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55721607f710 con 0x557215d2a1f0 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.387+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x557215d2a1f0 0x557216123d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.388+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x557215d2a1f0 0x557216123d10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a958363f174e64c server_cookie=4c7a792347654229 in_seq=0 out_seq=0 2024-04-25T16:32:40.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.388+0000 7f09ef7fe640 10 client.15690 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:40.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3668091457 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55721607f710 con 0x557215d2a1f0 2024-04-25T16:32:40.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f09ef7fe640 10 client.15690 handle_command_reply: tid=0 2024-04-25T16:32:40.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 client.15690 shutdown 2024-04-25T16:32:40.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 -- 172.21.15.98:0/3668091457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d4a730 msgr2=0x5572161167e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d4a730 0x5572161167e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f0a04015df0 tx=0x7f0a040047d0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 -- 172.21.15.98:0/3668091457 shutdown_connections 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x557215d2a1f0 0x557216123d10 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d4a730 0x5572161167e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3668091457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x5572161162a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 -- 172.21.15.98:0/3668091457 >> 172.21.15.98:0/3668091457 conn(0x557215d5f560 msgr2=0x557216106ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f09ef7fe640 0 client.15690 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 -- 172.21.15.98:0/3668091457 shutdown_connections 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 1 -- 172.21.15.98:0/3668091457 wait complete. 2024-04-25T16:32:40.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.389+0000 7f0a1d407740 20 client.15690 trim_cache size 0 max 16384 2024-04-25T16:32:40.396 INFO:tasks.workunit.client.0.smithi098.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T16:32:40.418 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.407+0000 7f0a1d407740 1 Processor -- start 2024-04-25T16:32:40.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.407+0000 7f0a1d407740 1 -- start start 2024-04-25T16:32:40.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.407+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x557216110d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.407+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 0x55721610b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.407+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d6b410 0x55721610b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.407+0000 7f0a1d407740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a101ac980 con 0x557215d6b410 2024-04-25T16:32:40.419 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.407+0000 7f0a1d407740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a101ae4c0 con 0x557215d42550 2024-04-25T16:32:40.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a1d407740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a101ad050 con 0x557215d4a730 2024-04-25T16:32:40.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d6b410 0x55721610b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d6b410 0x55721610b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37308/0 (socket says 172.21.15.98:37308) 2024-04-25T16:32:40.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 -- 172.21.15.98:0/2400132341 learned_addr learned my addr 172.21.15.98:0/2400132341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x557216110d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 0x55721610b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.420 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 -- 172.21.15.98:0/2400132341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 msgr2=0x55721610b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 0x55721610b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 -- 172.21.15.98:0/2400132341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 msgr2=0x557216110d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x557216110d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 -- 172.21.15.98:0/2400132341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55721610bfb0 con 0x557215d6b410 2024-04-25T16:32:40.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x557216110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:32:40.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f0a14af4640 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d6b410 0x55721610b880 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0a080fde70 tx=0x7f0a080c4060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461d12f0a519d989 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f09f5ffb640 1 -- 172.21.15.98:0/2400132341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a080c8810 con 0x557215d6b410 2024-04-25T16:32:40.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f09f5ffb640 1 -- 172.21.15.98:0/2400132341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a08036040 con 0x557215d6b410 2024-04-25T16:32:40.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.408+0000 7f09f5ffb640 1 -- 172.21.15.98:0/2400132341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a080fe080 con 0x557215d6b410 2024-04-25T16:32:40.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 -- 172.21.15.98:0/2400132341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d6b410 msgr2=0x55721610b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d6b410 0x55721610b880 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0a080fde70 tx=0x7f0a080c4060 comp rx=0 tx=0).stop 2024-04-25T16:32:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 -- 172.21.15.98:0/2400132341 shutdown_connections 2024-04-25T16:32:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d6b410 0x55721610b880 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d4a730 0x55721610b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 --2- 172.21.15.98:0/2400132341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d42550 0x557216110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 -- 172.21.15.98:0/2400132341 >> 172.21.15.98:0/2400132341 conn(0x557215d5f560 msgr2=0x55721610ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 -- 172.21.15.98:0/2400132341 shutdown_connections 2024-04-25T16:32:40.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.409+0000 7f0a1d407740 1 -- 172.21.15.98:0/2400132341 wait complete. 2024-04-25T16:32:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 1 -- start start 2024-04-25T16:32:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 1 Processor -- start 2024-04-25T16:32:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721611b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d4a730 0x55721611b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0a101addf0 con 0x557215d42550 2024-04-25T16:32:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0a101ad720 con 0x557215d4a730 2024-04-25T16:32:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0a101aa960 con 0x557215d6b410 2024-04-25T16:32:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a1d407740 10 client.0 _ll_register_callbacks cb 0x7f0a04015aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a0ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721611b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a0ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721611b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37320/0 (socket says 172.21.15.98:37320) 2024-04-25T16:32:40.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a0ffff640 1 -- 172.21.15.98:0/3310494381 learned_addr learned my addr 172.21.15.98:0/3310494381 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d4a730 0x55721611b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.410+0000 7f0a14af4640 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a0ffff640 1 -- 172.21.15.98:0/3310494381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 msgr2=0x55721611c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.426 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a0ffff640 1 -- 172.21.15.98:0/3310494381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d4a730 msgr2=0x55721611b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d4a730 0x55721611b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a0ffff640 1 -- 172.21.15.98:0/3310494381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55721611cde0 con 0x557215d42550 2024-04-25T16:32:40.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a14af4640 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a0ffff640 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721611b3b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0a00008010 tx=0x7f0a000047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef8fce03d64243 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f09ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3310494381 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a00017040 con 0x557215d42550 2024-04-25T16:32:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a1d407740 1 -- 172.21.15.98:0/3310494381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0a08036040 con 0x557215d42550 2024-04-25T16:32:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3310494381 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f0a00005bc0 con 0x557215d42550 2024-04-25T16:32:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3310494381 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0a00036050 con 0x557215d42550 2024-04-25T16:32:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a1d407740 10 client.15696 fetch_fsmap learned FSMap version 29 2024-04-25T16:32:40.428 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f0a1d407740 1 -- 172.21.15.98:0/3310494381 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55721611e110 con 0x557215d42550 2024-04-25T16:32:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.411+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3310494381 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0a08036040 con 0x557215d42550 2024-04-25T16:32:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3310494381 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f0a000136a0 con 0x557215d42550 2024-04-25T16:32:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f0a1d407740 10 client.15696 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:32:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f0a1d407740 10 client.15696 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f0a1d407740 20 client.15696 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x557215cff630 0x55721618e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f0a1d407740 4 client.15696 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:32:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f0a1d407740 1 -- 172.21.15.98:0/3310494381 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557216079580 con 0x557215cff630 2024-04-25T16:32:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x557215cff630 0x55721618e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f09ef7fe640 10 client.15696 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.412+0000 7f0a0f7fe640 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x557215cff630 0x55721618e650 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3e2d3b091e7483a server_cookie=907225942a3bf87d in_seq=0 out_seq=0 2024-04-25T16:32:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.413+0000 7f09ef7fe640 1 -- 172.21.15.98:0/3310494381 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557216079580 con 0x557215cff630 2024-04-25T16:32:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.413+0000 7f09ef7fe640 10 client.15696 handle_command_reply: tid=0 2024-04-25T16:32:40.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.413+0000 7f0a1d407740 1 client.15696 shutdown 2024-04-25T16:32:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 -- 172.21.15.98:0/3310494381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 msgr2=0x55721611b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721611b3b0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0a00008010 tx=0x7f0a000047d0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 -- 172.21.15.98:0/3310494381 shutdown_connections 2024-04-25T16:32:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x557215cff630 0x55721618e650 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557215d6b410 0x55721611c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557215d4a730 0x55721611b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f09ef7fe640 0 client.15696 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:40.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 --2- 172.21.15.98:0/3310494381 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x557215d42550 0x55721611b3b0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 -- 172.21.15.98:0/3310494381 >> 172.21.15.98:0/3310494381 conn(0x557215d5f560 msgr2=0x55721610b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 -- 172.21.15.98:0/3310494381 shutdown_connections 2024-04-25T16:32:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 1 -- 172.21.15.98:0/3310494381 wait complete. 2024-04-25T16:32:40.432 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a1d407740 20 client.15696 trim_cache size 0 max 16384 2024-04-25T16:32:40.433 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:40.433 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-04-25T16:32:40.433 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "da17da13-7ac2-4d29-8397-4f618c0a242c", 2024-04-25T16:32:40.433 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-04-25T16:32:40.433 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a16557640 1 -- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09f80afda0 msgr2=0x7f09f80b22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a16557640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09f80afda0 0x7f09f80b22b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55721610d670 tx=0x7f0a04009110 comp rx=0 tx=0).stop 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a16557640 1 -- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 msgr2=0x7f0a101a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.414+0000 7f0a16557640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 0x7f0a101a16b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0a08031040 tx=0x7f0a08003ce0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.415+0000 7f0a16557640 1 -- 172.21.15.98:0/2104300561 shutdown_connections 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.415+0000 7f0a16557640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f09f80afda0 0x7f09f80b22b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.415+0000 7f0a16557640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0a10104460 0x7f0a101a16b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.415+0000 7f0a16557640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a10103940 0x7f0a1019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.415+0000 7f0a16557640 1 --2- 172.21.15.98:0/2104300561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0a10102f60 0x7f0a1019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.415+0000 7f0a16557640 1 -- 172.21.15.98:0/2104300561 >> 172.21.15.98:0/2104300561 conn(0x7f0a100fd340 msgr2=0x7f0a10106ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.415+0000 7f0a16557640 1 -- 172.21.15.98:0/2104300561 shutdown_connections 2024-04-25T16:32:40.435 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.415+0000 7f0a16557640 1 -- 172.21.15.98:0/2104300561 wait complete. 2024-04-25T16:32:40.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:32:40.436 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:32:40.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:40 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2539681870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:40.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:40 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1120679758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:40.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:40 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1268062799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:40.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:40 smithi203 ceph-mon[33115]: pgmap v361: 129 pgs: 129 active+clean; 655 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-25T16:32:40.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:40 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2539681870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:32:40.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:40 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1120679758' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:32:40.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:40 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1268062799' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:32:40.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:40 smithi173 ceph-mon[32556]: pgmap v361: 129 pgs: 129 active+clean; 655 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-25T16:32:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 -- 172.21.15.98:0/683859325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681045d0 msgr2=0x7f8f681069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 --2- 172.21.15.98:0/683859325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681045d0 0x7f8f681069f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009900 tx=0x7f8f5c02a410 comp rx=0 tx=0).stop 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 -- 172.21.15.98:0/683859325 shutdown_connections 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 --2- 172.21.15.98:0/683859325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68109890 0x7f8f6810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 --2- 172.21.15.98:0/683859325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f68106f30 0x7f8f68109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 --2- 172.21.15.98:0/683859325 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f681045d0 0x7f8f681069f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 -- 172.21.15.98:0/683859325 >> 172.21.15.98:0/683859325 conn(0x7f8f680fdbd0 msgr2=0x7f8f68100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 -- 172.21.15.98:0/683859325 shutdown_connections 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 -- 172.21.15.98:0/683859325 wait complete. 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.587+0000 7f8f6f146640 1 Processor -- start 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f6f146640 1 -- start start 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f6f146640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f681045d0 0x7f8f6819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f6f146640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68106f30 0x7f8f6819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f6f146640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68109890 0x7f8f681a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f6f146640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810f490 con 0x7f8f68106f30 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f6f146640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810f310 con 0x7f8f681045d0 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f6f146640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6810f610 con 0x7f8f68109890 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68106f30 0x7f8f6819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68106f30 0x7f8f6819aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37332/0 (socket says 172.21.15.98:37332) 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 -- 172.21.15.98:0/196478221 learned_addr learned my addr 172.21.15.98:0/196478221 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f681045d0 0x7f8f6819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 -- 172.21.15.98:0/196478221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68109890 msgr2=0x7f8f681a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68109890 0x7f8f681a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 -- 172.21.15.98:0/196478221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f681045d0 msgr2=0x7f8f6819a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f681045d0 0x7f8f6819a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 -- 172.21.15.98:0/196478221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f681a2010 con 0x7f8f68106f30 2024-04-25T16:32:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f67fff640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68106f30 0x7f8f6819aa10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400d950 tx=0x7f8f5400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.588+0000 7f8f65ffb640 1 -- 172.21.15.98:0/196478221 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f54017070 con 0x7f8f68106f30 2024-04-25T16:32:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.589+0000 7f8f65ffb640 1 -- 172.21.15.98:0/196478221 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8f54009c30 con 0x7f8f68106f30 2024-04-25T16:32:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.589+0000 7f8f6f146640 1 -- 172.21.15.98:0/196478221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f681a2300 con 0x7f8f68106f30 2024-04-25T16:32:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.589+0000 7f8f65ffb640 1 -- 172.21.15.98:0/196478221 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5400f620 con 0x7f8f68106f30 2024-04-25T16:32:40.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.589+0000 7f8f6f146640 1 -- 172.21.15.98:0/196478221 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f681a2850 con 0x7f8f68106f30 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.591+0000 7f8f65ffb640 1 -- 172.21.15.98:0/196478221 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f8f54010460 con 0x7f8f68106f30 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.591+0000 7f8f75ffc740 1 Processor -- start 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.591+0000 7f8f75ffc740 1 -- start start 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e9129a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9129fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e912ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5571e908ed80 con 0x5571e8d88390 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5571e90488b0 con 0x5571e8d5f4d0 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5571e908ad40 con 0x5571e8d676b0 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e9129a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e9129a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37348/0 (socket says 172.21.15.98:37348) 2024-04-25T16:32:40.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/3364850318 learned_addr learned my addr 172.21.15.98:0/3364850318 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f67fff640 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e912ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f65ffb640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f580afb80 0x7f8f580b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9129fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/3364850318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 msgr2=0x5571e912ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e912ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/3364850318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 msgr2=0x5571e9129fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9129fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/3364850318 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571e9019a40 con 0x5571e8d88390 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f67fff640 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e912ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f580afb80 0x7f8f580b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f65ffb640 1 -- 172.21.15.98:0/196478221 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f8f540dedd0 con 0x7f8f68106f30 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e9129a60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009e90 tx=0x7f8f6000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646c2c16e1d5f141 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.592+0000 7f8f52ffd640 1 -- 172.21.15.98:0/3364850318 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6001b550 con 0x5571e8d88390 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f580afb80 0x7f8f580b2090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8f6001f000 tx=0x7f8f60020000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f52ffd640 1 -- 172.21.15.98:0/3364850318 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8f6001bb80 con 0x5571e8d88390 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f52ffd640 1 -- 172.21.15.98:0/3364850318 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f600118a0 con 0x5571e8d88390 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f75ffc740 1 -- 172.21.15.98:0/3364850318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 msgr2=0x5571e9129a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e9129a60 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009e90 tx=0x7f8f6000e6c0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f75ffc740 1 -- 172.21.15.98:0/3364850318 shutdown_connections 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e912ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9129fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/3364850318 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e9129a60 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.593+0000 7f8f75ffc740 1 -- 172.21.15.98:0/3364850318 >> 172.21.15.98:0/3364850318 conn(0x5571e8d7c4e0 msgr2=0x5571e9125270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 -- 172.21.15.98:0/3364850318 shutdown_connections 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 -- 172.21.15.98:0/3364850318 wait complete. 2024-04-25T16:32:40.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 -- start start 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 Processor -- start 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9133900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e9133e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e913ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5571e912b780 con 0x5571e8d88390 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5571e9015d10 con 0x5571e8d5f4d0 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5571e912b900 con 0x5571e8d676b0 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.594+0000 7f8f75ffc740 10 client.0 _ll_register_callbacks cb 0x5571e9042210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e913ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e913ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37358/0 (socket says 172.21.15.98:37358) 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/2917982460 learned_addr learned my addr 172.21.15.98:0/2917982460 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9133900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f67fff640 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e9133e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/2917982460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 msgr2=0x5571e9133e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e9133e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/2917982460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 msgr2=0x5571e9133900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9133900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/2917982460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571e8e6a7d0 con 0x5571e8d88390 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e913ada0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8f60005b90 tx=0x7f8f6001bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca2c7dfb357e360 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f47fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f47fff640 1 -- 172.21.15.98:0/2917982460 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60034060 con 0x5571e8d88390 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f47fff640 1 -- 172.21.15.98:0/2917982460 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f8f60021050 con 0x5571e8d88390 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2917982460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f6001bb80 con 0x5571e8d88390 2024-04-25T16:32:40.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f47fff640 1 -- 172.21.15.98:0/2917982460 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8f60011830 con 0x5571e8d88390 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f75ffc740 10 client.15708 fetch_fsmap learned FSMap version 29 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.595+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2917982460 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5571e8da03f0 con 0x5571e8d88390 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f47fff640 1 -- 172.21.15.98:0/2917982460 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8f6001bb80 con 0x5571e8d88390 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f47fff640 1 -- 172.21.15.98:0/2917982460 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8f6000f040 con 0x5571e8d88390 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f75ffc740 10 client.15708 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f75ffc740 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f75ffc740 20 client.15708 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5571e8d46f60 0x5571e9140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f75ffc740 4 client.15708 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2917982460 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571e909c990 con 0x5571e8d46f60 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5571e8d46f60 0x5571e9140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f47fff640 10 client.15708 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:40.604 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.596+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5571e8d46f60 0x5571e9140fd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8584ff81273332ca server_cookie=7e7509da71e1729e in_seq=0 out_seq=0 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f47fff640 1 -- 172.21.15.98:0/2917982460 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5571e909c990 con 0x5571e8d46f60 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f47fff640 10 client.15708 handle_command_reply: tid=0 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 client.15708 shutdown 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2917982460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 msgr2=0x5571e913ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e913ada0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8f60005b90 tx=0x7f8f6001bf40 comp rx=0 tx=0).stop 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2917982460 shutdown_connections 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5571e8d46f60 0x5571e9140fd0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d88390 0x5571e913ada0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e9133e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2917982460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9133900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f47fff640 0 client.15708 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2917982460 >> 172.21.15.98:0/2917982460 conn(0x5571e8d7c4e0 msgr2=0x5571e9128180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2917982460 shutdown_connections 2024-04-25T16:32:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.597+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2917982460 wait complete. 2024-04-25T16:32:40.606 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.598+0000 7f8f75ffc740 20 client.15708 trim_cache size 0 max 16384 2024-04-25T16:32:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.615+0000 7f8f75ffc740 1 Processor -- start 2024-04-25T16:32:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.615+0000 7f8f75ffc740 1 -- start start 2024-04-25T16:32:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d46f60 0x5571e9124ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e91251e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e9131f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681acac0 con 0x5571e8d46f60 2024-04-25T16:32:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681ae600 con 0x5571e8d5f4d0 2024-04-25T16:32:40.621 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681ad190 con 0x5571e8d676b0 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d46f60 0x5571e9124ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d46f60 0x5571e9124ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37368/0 (socket says 172.21.15.98:37368) 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 -- 172.21.15.98:0/2870208564 learned_addr learned my addr 172.21.15.98:0/2870208564 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f67fff640 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e91251e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e9131f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 -- 172.21.15.98:0/2870208564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 msgr2=0x5571e9131f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e9131f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 -- 172.21.15.98:0/2870208564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 msgr2=0x5571e91251e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e91251e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 -- 172.21.15.98:0/2870208564 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571e912e150 con 0x5571e8d46f60 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d46f60 0x5571e9124ca0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0057c0 tx=0x7f8f5c0042d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d5b2daf24c7284 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f527fc640 1 -- 172.21.15.98:0/2870208564 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c035040 con 0x5571e8d46f60 2024-04-25T16:32:40.622 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f527fc640 1 -- 172.21.15.98:0/2870208564 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8f5c04a070 con 0x5571e8d46f60 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.616+0000 7f8f527fc640 1 -- 172.21.15.98:0/2870208564 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c031c40 con 0x5571e8d46f60 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2870208564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d46f60 msgr2=0x5571e9124ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d46f60 0x5571e9124ca0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0057c0 tx=0x7f8f5c0042d0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2870208564 shutdown_connections 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d676b0 0x5571e9131f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e91251e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2870208564 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d46f60 0x5571e9124ca0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2870208564 >> 172.21.15.98:0/2870208564 conn(0x5571e8d7c4e0 msgr2=0x5571e912e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2870208564 shutdown_connections 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.617+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2870208564 wait complete. 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 1 -- start start 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 1 Processor -- start 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d46f60 0x5571e9137440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.623 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9137980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d676b0 0x5571e91a7290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681adf30 con 0x5571e8d676b0 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681ad860 con 0x5571e8d5f4d0 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f8f681aaaa0 con 0x5571e8d46f60 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f75ffc740 10 client.0 _ll_register_callbacks cb 0x7f8f6001e020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6d6e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d676b0 0x5571e91a7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6d6e3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d676b0 0x5571e91a7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37376/0 (socket says 172.21.15.98:37376) 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/2527355037 learned_addr learned my addr 172.21.15.98:0/2527355037 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f67fff640 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d46f60 0x5571e9137440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/2527355037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d46f60 msgr2=0x5571e9137440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d46f60 0x5571e9137440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/2527355037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 msgr2=0x5571e9137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.618+0000 7f8f6d6e3640 1 -- 172.21.15.98:0/2527355037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571e8d600f0 con 0x5571e8d676b0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f6d6e3640 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d676b0 0x5571e91a7290 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000e700 tx=0x7f8f6001b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7485bf271f7bf60a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f47fff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f47fff640 1 -- 172.21.15.98:0/2527355037 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f60034070 con 0x5571e8d676b0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f47fff640 1 -- 172.21.15.98:0/2527355037 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f8f60021040 con 0x5571e8d676b0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2527355037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f5c04a070 con 0x5571e8d676b0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f47fff640 1 -- 172.21.15.98:0/2527355037 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8f6001bb80 con 0x5571e8d676b0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 10 client.15714 fetch_fsmap learned FSMap version 29 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2527355037 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5571e904f4b0 con 0x5571e8d676b0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f47fff640 1 -- 172.21.15.98:0/2527355037 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8f5c04a070 con 0x5571e8d676b0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f47fff640 1 -- 172.21.15.98:0/2527355037 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f8f6003d030 con 0x5571e8d676b0 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 10 client.15714 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 10 client.15714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 20 client.15714 populate_metadata read hostname 'smithi098' 2024-04-25T16:32:40.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5571e8d88390 0x5571e91ab080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 4 client.15714 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.619+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2527355037 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5571e9096800 con 0x5571e8d88390 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5571e8d88390 0x5571e91ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f6cee2640 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5571e8d88390 0x5571e91ab080 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4530baa26f98680b server_cookie=e05e15cc005d6940 in_seq=0 out_seq=0 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f47fff640 10 client.15714 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f47fff640 1 -- 172.21.15.98:0/2527355037 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5571e9096800 con 0x5571e8d88390 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f47fff640 10 client.15714 handle_command_reply: tid=0 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 client.15714 shutdown 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2527355037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d676b0 msgr2=0x5571e91a7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d676b0 0x5571e91a7290 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000e700 tx=0x7f8f6001b780 comp rx=0 tx=0).stop 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2527355037 shutdown_connections 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5571e8d88390 0x5571e91ab080 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5571e8d676b0 0x5571e91a7290 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5571e8d5f4d0 0x5571e9137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f47fff640 0 client.15714 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:32:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 --2- 172.21.15.98:0/2527355037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5571e8d46f60 0x5571e9137440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.620+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2527355037 >> 172.21.15.98:0/2527355037 conn(0x5571e8d7c4e0 msgr2=0x5571e9124c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2527355037 shutdown_connections 2024-04-25T16:32:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f75ffc740 1 -- 172.21.15.98:0/2527355037 wait complete. 2024-04-25T16:32:40.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f75ffc740 20 client.15714 trim_cache size 0 max 16384 2024-04-25T16:32:40.627 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout: "da17da13-7ac2-4d29-8397-4f618c0a242c": { 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "da17da13-7ac2-4d29-8397-4f618c0a242c", 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:32:40.628 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 -- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f580afb80 msgr2=0x7f8f580b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f580afb80 0x7f8f580b2090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8f6001f000 tx=0x7f8f60020000 comp rx=0 tx=0).stop 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 -- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68106f30 msgr2=0x7f8f6819aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68106f30 0x7f8f6819aa10 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400d950 tx=0x7f8f5400de10 comp rx=0 tx=0).stop 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 -- 172.21.15.98:0/196478221 shutdown_connections 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f8f580afb80 0x7f8f580b2090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f68109890 0x7f8f681a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8f68106f30 0x7f8f6819aa10 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 --2- 172.21.15.98:0/196478221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f681045d0 0x7f8f6819a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 -- 172.21.15.98:0/196478221 >> 172.21.15.98:0/196478221 conn(0x7f8f680fdbd0 msgr2=0x7f8f68104cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:32:40.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 -- 172.21.15.98:0/196478221 shutdown_connections 2024-04-25T16:32:40.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:32:40.621+0000 7f8f6f146640 1 -- 172.21.15.98:0/196478221 wait complete. 2024-04-25T16:32:40.632 INFO:tasks.cephfs.filesystem:scrub status for tag:da17da13-7ac2-4d29-8397-4f618c0a242c - {'path': '/', 'tag': 'da17da13-7ac2-4d29-8397-4f618c0a242c', 'options': 'recursive,force'} 2024-04-25T16:32:41.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:41 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1764628019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:41.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:41 smithi203 ceph-mon[33115]: scrub queued for path: / 2024-04-25T16:32:41.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:41 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:32:41.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:41 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:32:41.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:41 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:32:41.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:41 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1764628019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:41.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:41 smithi098 ceph-mon[29162]: scrub queued for path: / 2024-04-25T16:32:41.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:41 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:32:41.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:41 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:32:41.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:41 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:32:41.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:41 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1764628019' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:32:41.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:41 smithi173 ceph-mon[32556]: scrub queued for path: / 2024-04-25T16:32:41.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:41 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:32:41.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:41 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:32:41.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:41 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:32:42.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:42 smithi203 ceph-mon[33115]: pgmap v362: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-04-25T16:32:42.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:42 smithi098 ceph-mon[29162]: pgmap v362: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-04-25T16:32:42.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:42 smithi173 ceph-mon[32556]: pgmap v362: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 13 op/s 2024-04-25T16:32:44.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:44 smithi203 ceph-mon[33115]: pgmap v363: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-25T16:32:44.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:44 smithi098 ceph-mon[29162]: pgmap v363: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-25T16:32:44.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:44 smithi173 ceph-mon[32556]: pgmap v363: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2024-04-25T16:32:46.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:46 smithi203 ceph-mon[33115]: pgmap v364: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T16:32:46.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:46 smithi098 ceph-mon[29162]: pgmap v364: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T16:32:46.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:46 smithi173 ceph-mon[32556]: pgmap v364: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T16:32:47.329 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:47 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:32:47.347 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:47 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:32:47.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:47 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:32:49.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: pgmap v365: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T16:32:49.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:32:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:32:49.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: pgmap v365: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:32:49.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: pgmap v365: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T16:32:49.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:32:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:32:49.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:32:51.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:51 smithi173 ceph-mon[32556]: pgmap v366: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T16:32:51.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:51 smithi203 ceph-mon[33115]: pgmap v366: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T16:32:51.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:51 smithi098 ceph-mon[29162]: pgmap v366: 129 pgs: 129 active+clean; 662 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T16:32:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:53 smithi173 ceph-mon[32556]: pgmap v367: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:32:53.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:53 smithi203 ceph-mon[33115]: pgmap v367: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:32:53.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:53 smithi098 ceph-mon[29162]: pgmap v367: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:32:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:54 smithi173 ceph-mon[32556]: pgmap v368: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-25T16:32:54.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:54 smithi203 ceph-mon[33115]: pgmap v368: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-25T16:32:54.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:54 smithi098 ceph-mon[29162]: pgmap v368: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-25T16:32:55.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:55 smithi203 ceph-mon[33115]: scrub summary: idle 2024-04-25T16:32:55.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:55 smithi098 ceph-mon[29162]: scrub summary: idle 2024-04-25T16:32:55.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:55 smithi173 ceph-mon[32556]: scrub summary: idle 2024-04-25T16:32:56.352 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:56 smithi203 ceph-mon[33115]: pgmap v369: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-25T16:32:56.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:56 smithi098 ceph-mon[29162]: pgmap v369: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-25T16:32:56.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:56 smithi173 ceph-mon[32556]: pgmap v369: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-25T16:32:58.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:32:58 smithi203 ceph-mon[33115]: pgmap v370: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-25T16:32:58.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:32:58 smithi098 ceph-mon[29162]: pgmap v370: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-25T16:32:58.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:32:58 smithi173 ceph-mon[32556]: pgmap v370: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 682 B/s wr, 11 op/s 2024-04-25T16:33:00.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:00 smithi098 ceph-mon[29162]: pgmap v371: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-25T16:33:00.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:00 smithi203 ceph-mon[33115]: pgmap v371: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-25T16:33:00.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:00 smithi173 ceph-mon[32556]: pgmap v371: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 682 B/s wr, 10 op/s 2024-04-25T16:33:02.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:02 smithi203 ceph-mon[33115]: pgmap v372: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:33:02.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:02 smithi098 ceph-mon[29162]: pgmap v372: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:33:02.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:02 smithi173 ceph-mon[32556]: pgmap v372: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-25T16:33:04.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:04 smithi203 ceph-mon[33115]: pgmap v373: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:04.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:04 smithi098 ceph-mon[29162]: pgmap v373: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:04.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:04 smithi173 ceph-mon[32556]: pgmap v373: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:06.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:06 smithi203 ceph-mon[33115]: pgmap v374: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:33:06.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:06 smithi098 ceph-mon[29162]: pgmap v374: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:33:06.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:06 smithi173 ceph-mon[32556]: pgmap v374: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:33:08.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:08 smithi203 ceph-mon[33115]: pgmap v375: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:33:08.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:08 smithi098 ceph-mon[29162]: pgmap v375: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:33:08.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:08 smithi173 ceph-mon[32556]: pgmap v375: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:33:10.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:10 smithi203 ceph-mon[33115]: pgmap v376: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:33:10.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:10 smithi098 ceph-mon[29162]: pgmap v376: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:33:10.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:10 smithi173 ceph-mon[32556]: pgmap v376: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:33:10.633 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:33:10.633 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:33:10.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.760+0000 7f302a8aa640 1 -- 172.21.15.98:0/1549976088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024109790 msgr2=0x7f302410bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.760 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.760+0000 7f302a8aa640 1 --2- 172.21.15.98:0/1549976088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024109790 0x7f302410bca0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f301c009ca0 tx=0x7f301c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 -- 172.21.15.98:0/1549976088 shutdown_connections 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 --2- 172.21.15.98:0/1549976088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024109790 0x7f302410bca0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 --2- 172.21.15.98:0/1549976088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3024106c20 0x7f3024109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 --2- 172.21.15.98:0/1549976088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3024101e40 0x7f3024102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 -- 172.21.15.98:0/1549976088 >> 172.21.15.98:0/1549976088 conn(0x7f30240fd350 msgr2=0x7f30240ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 -- 172.21.15.98:0/1549976088 shutdown_connections 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 -- 172.21.15.98:0/1549976088 wait complete. 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 Processor -- start 2024-04-25T16:33:10.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.761+0000 7f302a8aa640 1 -- start start 2024-04-25T16:33:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f302a8aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024101e40 0x7f3024104310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f302a8aa640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3024106c20 0x7f30241029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f302a8aa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3024109790 0x7f3024102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f302a8aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f302410f520 con 0x7f3024101e40 2024-04-25T16:33:10.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f302a8aa640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f302410f3a0 con 0x7f3024106c20 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f302a8aa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f302410f6a0 con 0x7f3024109790 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024101e40 0x7f3024104310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024101e40 0x7f3024104310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44288/0 (socket says 172.21.15.98:44288) 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 -- 172.21.15.98:0/3842627067 learned_addr learned my addr 172.21.15.98:0/3842627067 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f30237fe640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3024106c20 0x7f30241029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3028e47640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3024109790 0x7f3024102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 -- 172.21.15.98:0/3842627067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3024109790 msgr2=0x7f3024102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3024109790 0x7f3024102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 -- 172.21.15.98:0/3842627067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3024106c20 msgr2=0x7f30241029b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3024106c20 0x7f30241029b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 -- 172.21.15.98:0/3842627067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3024103620 con 0x7f3024101e40 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f3023fff640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024101e40 0x7f3024104310 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f301400d740 tx=0x7f301400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:10.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f30217fa640 1 -- 172.21.15.98:0/3842627067 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014017070 con 0x7f3024101e40 2024-04-25T16:33:10.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f30217fa640 1 -- 172.21.15.98:0/3842627067 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f301400de50 con 0x7f3024101e40 2024-04-25T16:33:10.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f302a8aa640 1 -- 172.21.15.98:0/3842627067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30241a41c0 con 0x7f3024101e40 2024-04-25T16:33:10.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f30217fa640 1 -- 172.21.15.98:0/3842627067 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3014004c60 con 0x7f3024101e40 2024-04-25T16:33:10.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.762+0000 7f302a8aa640 1 -- 172.21.15.98:0/3842627067 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30241a4630 con 0x7f3024101e40 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3031834740 1 Processor -- start 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3031834740 1 -- start start 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f30217fa640 1 -- 172.21.15.98:0/3842627067 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f3014004e00 con 0x7f3024101e40 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3031834740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf446a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3031834740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf07c4d0 0x5645bf446fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3031834740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 0x5645bf447c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3031834740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5645bf3abd80 con 0x5645bf07c4d0 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3031834740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5645bf3658b0 con 0x5645bf0a5390 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3031834740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5645bf3a7d40 con 0x5645bf0846b0 2024-04-25T16:33:10.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3023fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf07c4d0 0x5645bf446fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3023fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf07c4d0 0x5645bf446fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44298/0 (socket says 172.21.15.98:44298) 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3023fff640 1 -- 172.21.15.98:0/1817382883 learned_addr learned my addr 172.21.15.98:0/1817382883 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f30237fe640 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 0x5645bf447c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.765+0000 7f3028e47640 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf446a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f3023fff640 1 -- 172.21.15.98:0/1817382883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 msgr2=0x5645bf447c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f3023fff640 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 0x5645bf447c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f3023fff640 1 -- 172.21.15.98:0/1817382883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 msgr2=0x5645bf446a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f3023fff640 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf446a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f3023fff640 1 -- 172.21.15.98:0/1817382883 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645bf336a40 con 0x5645bf07c4d0 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f30237fe640 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 0x5645bf447c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f30217fa640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f300c0afb80 0x7f300c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f3023fff640 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf07c4d0 0x5645bf446fe0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f30140dc110 tx=0x7f30140e7730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acde17f59e82cf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f3028e47640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f300c0afb80 0x7f300c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f30217fa640 1 -- 172.21.15.98:0/3842627067 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f30140debd0 con 0x7f3024101e40 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f300a7fc640 1 -- 172.21.15.98:0/1817382883 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30140a2a60 con 0x5645bf07c4d0 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f300a7fc640 1 -- 172.21.15.98:0/1817382883 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30140c7e50 con 0x5645bf07c4d0 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.766+0000 7f300a7fc640 1 -- 172.21.15.98:0/1817382883 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30140a6310 con 0x5645bf07c4d0 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 -- 172.21.15.98:0/1817382883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf07c4d0 msgr2=0x5645bf446fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3028e47640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f300c0afb80 0x7f300c0b2090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f301c031920 tx=0x7f301c0319a0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf07c4d0 0x5645bf446fe0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f30140dc110 tx=0x7f30140e7730 comp rx=0 tx=0).stop 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 -- 172.21.15.98:0/1817382883 shutdown_connections 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 0x5645bf447c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf07c4d0 0x5645bf446fe0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 --2- 172.21.15.98:0/1817382883 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf446a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 -- 172.21.15.98:0/1817382883 >> 172.21.15.98:0/1817382883 conn(0x5645bf0994e0 msgr2=0x5645bf442270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 -- 172.21.15.98:0/1817382883 shutdown_connections 2024-04-25T16:33:10.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 -- 172.21.15.98:0/1817382883 wait complete. 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.767+0000 7f3031834740 1 -- start start 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3031834740 1 Processor -- start 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3031834740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf450a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3031834740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf450f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3031834740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf457eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3031834740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5645bf448780 con 0x5645bf0846b0 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3031834740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5645bf332d10 con 0x5645bf0a5390 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3031834740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5645bf448900 con 0x5645bf07c4d0 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3031834740 10 client.0 _ll_register_callbacks cb 0x5645bf35f210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf450f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf450f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44304/0 (socket says 172.21.15.98:44304) 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 -- 172.21.15.98:0/2898550447 learned_addr learned my addr 172.21.15.98:0/2898550447 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3023fff640 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf450a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f3028e47640 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf457eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 -- 172.21.15.98:0/2898550447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 msgr2=0x5645bf450a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf450a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 -- 172.21.15.98:0/2898550447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 msgr2=0x5645bf457eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf457eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 -- 172.21.15.98:0/2898550447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645bf1877d0 con 0x5645bf0846b0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30237fe640 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf450f50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f301800bee0 tx=0x7f30180117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16ef287a02d87a71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.768+0000 7f30037fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f30037fe640 1 -- 172.21.15.98:0/2898550447 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018012650 con 0x5645bf0846b0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f30037fe640 1 -- 172.21.15.98:0/2898550447 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f30180127f0 con 0x5645bf0846b0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 1 -- 172.21.15.98:0/2898550447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f30140c7e50 con 0x5645bf0846b0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f30037fe640 1 -- 172.21.15.98:0/2898550447 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f30180129f0 con 0x5645bf0846b0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 10 client.15726 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 1 -- 172.21.15.98:0/2898550447 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5645bf0bd3f0 con 0x5645bf0846b0 2024-04-25T16:33:10.776 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f30037fe640 1 -- 172.21.15.98:0/2898550447 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f30140c7e50 con 0x5645bf0846b0 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f30037fe640 1 -- 172.21.15.98:0/2898550447 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f301801bc90 con 0x5645bf0846b0 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 10 client.15726 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 20 client.15726 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5645bf063f60 0x5645bf45e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 4 client.15726 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.769+0000 7f3031834740 1 -- 172.21.15.98:0/2898550447 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5645bf3b9990 con 0x5645bf063f60 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.770+0000 7f3023fff640 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5645bf063f60 0x5645bf45e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.770+0000 7f30037fe640 10 client.15726 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.770+0000 7f3023fff640 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5645bf063f60 0x5645bf45e480 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b50c6ca7b14dcc5 server_cookie=3c0278281c343610 in_seq=0 out_seq=0 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f30037fe640 1 -- 172.21.15.98:0/2898550447 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5645bf3b9990 con 0x5645bf063f60 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f30037fe640 10 client.15726 handle_command_reply: tid=0 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 client.15726 shutdown 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 -- 172.21.15.98:0/2898550447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 msgr2=0x5645bf450f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf450f50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f301800bee0 tx=0x7f30180117b0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 -- 172.21.15.98:0/2898550447 shutdown_connections 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5645bf063f60 0x5645bf45e480 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf457eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf450f50 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f30037fe640 0 client.15726 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 --2- 172.21.15.98:0/2898550447 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf450a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 -- 172.21.15.98:0/2898550447 >> 172.21.15.98:0/2898550447 conn(0x5645bf0994e0 msgr2=0x5645bf445180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 -- 172.21.15.98:0/2898550447 shutdown_connections 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 1 -- 172.21.15.98:0/2898550447 wait complete. 2024-04-25T16:33:10.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.771+0000 7f3031834740 20 client.15726 trim_cache size 0 max 16384 2024-04-25T16:33:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.789+0000 7f3031834740 1 Processor -- start 2024-04-25T16:33:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.789+0000 7f3031834740 1 -- start start 2024-04-25T16:33:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.789+0000 7f3031834740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf07c4d0 0x5645bf44bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.789+0000 7f3031834740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 0x5645bf44f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.799 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.789+0000 7f3031834740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0a5390 0x5645bf457dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.789+0000 7f3031834740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30241aeb70 con 0x5645bf0a5390 2024-04-25T16:33:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.789+0000 7f3031834740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30241b06b0 con 0x5645bf07c4d0 2024-04-25T16:33:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.789+0000 7f3031834740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f30241af240 con 0x5645bf0846b0 2024-04-25T16:33:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0a5390 0x5645bf457dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0a5390 0x5645bf457dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44310/0 (socket says 172.21.15.98:44310) 2024-04-25T16:33:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 -- 172.21.15.98:0/1869085267 learned_addr learned my addr 172.21.15.98:0/1869085267 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:10.800 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3023fff640 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf07c4d0 0x5645bf44bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 -- 172.21.15.98:0/1869085267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 msgr2=0x5645bf44f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 0x5645bf44f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 -- 172.21.15.98:0/1869085267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf07c4d0 msgr2=0x5645bf44bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf07c4d0 0x5645bf44bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 -- 172.21.15.98:0/1869085267 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645bf07d0f0 con 0x5645bf0a5390 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3023fff640 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf07c4d0 0x5645bf44bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3028e47640 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0a5390 0x5645bf457dd0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f301c009ca0 tx=0x7f301c040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d407d8d3ce3c9343 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3009ffb640 1 -- 172.21.15.98:0/1869085267 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c0305c0 con 0x5645bf0a5390 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3009ffb640 1 -- 172.21.15.98:0/1869085267 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f301c036040 con 0x5645bf0a5390 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.790+0000 7f3009ffb640 1 -- 172.21.15.98:0/1869085267 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301c046a50 con 0x5645bf0a5390 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 -- 172.21.15.98:0/1869085267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0a5390 msgr2=0x5645bf457dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0a5390 0x5645bf457dd0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f301c009ca0 tx=0x7f301c040000 comp rx=0 tx=0).stop 2024-04-25T16:33:10.801 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 -- 172.21.15.98:0/1869085267 shutdown_connections 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0a5390 0x5645bf457dd0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf0846b0 0x5645bf44f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 --2- 172.21.15.98:0/1869085267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf07c4d0 0x5645bf44bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 -- 172.21.15.98:0/1869085267 >> 172.21.15.98:0/1869085267 conn(0x5645bf0994e0 msgr2=0x5645bf44f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 -- 172.21.15.98:0/1869085267 shutdown_connections 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 -- 172.21.15.98:0/1869085267 wait complete. 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.791+0000 7f3031834740 1 -- start start 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3031834740 1 Processor -- start 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3031834740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf455860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3031834740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf455da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3031834740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf456cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3031834740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f30241affe0 con 0x5645bf0846b0 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3031834740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f30241af910 con 0x5645bf0a5390 2024-04-25T16:33:10.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3031834740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f30241acb50 con 0x5645bf07c4d0 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3031834740 10 client.0 _ll_register_callbacks cb 0x7f301800b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf455da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf455da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44314/0 (socket says 172.21.15.98:44314) 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 -- 172.21.15.98:0/198185791 learned_addr learned my addr 172.21.15.98:0/198185791 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3023fff640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf455860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3028e47640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf456cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 -- 172.21.15.98:0/198185791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 msgr2=0x5645bf455860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf455860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 -- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 msgr2=0x5645bf456cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf456cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 -- 172.21.15.98:0/198185791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645bf36c4b0 con 0x5645bf0846b0 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3023fff640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf455860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:10.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f3028e47640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf456cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30237fe640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf455da0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3018006010 tx=0x7f301801a9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1347bf0949bae92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.792+0000 7f30037fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f30037fe640 1 -- 172.21.15.98:0/198185791 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301802d020 con 0x5645bf0846b0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f30037fe640 1 -- 172.21.15.98:0/198185791 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f301801ab10 con 0x5645bf0846b0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 1 -- 172.21.15.98:0/198185791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f301c036040 con 0x5645bf0846b0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f30037fe640 1 -- 172.21.15.98:0/198185791 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3018038050 con 0x5645bf0846b0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 10 client.15732 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 1 -- 172.21.15.98:0/198185791 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5645bf458350 con 0x5645bf0846b0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f30037fe640 1 -- 172.21.15.98:0/198185791 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f301c036040 con 0x5645bf0846b0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f30037fe640 1 -- 172.21.15.98:0/198185791 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f3018005270 con 0x5645bf0846b0 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 10 client.15732 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 10 client.15732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:10.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 20 client.15732 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5645bf0393a0 0x5645bf4c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 4 client.15732 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.793+0000 7f3031834740 1 -- 172.21.15.98:0/198185791 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5645bf3b3800 con 0x5645bf0393a0 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3023fff640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5645bf0393a0 0x5645bf4c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f30037fe640 10 client.15732 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3023fff640 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5645bf0393a0 0x5645bf4c8a30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=503b970caf866802 server_cookie=a9d633d8b475817b in_seq=0 out_seq=0 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f30037fe640 1 -- 172.21.15.98:0/198185791 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5645bf3b3800 con 0x5645bf0393a0 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f30037fe640 10 client.15732 handle_command_reply: tid=0 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 client.15732 shutdown 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 -- 172.21.15.98:0/198185791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 msgr2=0x5645bf455da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf455da0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f3018006010 tx=0x7f301801a9f0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 -- 172.21.15.98:0/198185791 shutdown_connections 2024-04-25T16:33:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5645bf0393a0 0x5645bf4c8a30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645bf0a5390 0x5645bf456cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645bf0846b0 0x5645bf455da0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 --2- 172.21.15.98:0/198185791 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645bf07c4d0 0x5645bf455860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f30037fe640 0 client.15732 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 -- 172.21.15.98:0/198185791 >> 172.21.15.98:0/198185791 conn(0x5645bf0994e0 msgr2=0x5645bf4409c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:10.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 -- 172.21.15.98:0/198185791 shutdown_connections 2024-04-25T16:33:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.794+0000 7f3031834740 1 -- 172.21.15.98:0/198185791 wait complete. 2024-04-25T16:33:10.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f3031834740 20 client.15732 trim_cache size 0 max 16384 2024-04-25T16:33:10.807 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:33:10.807 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-04-25T16:33:10.807 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-04-25T16:33:10.807 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 -- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f300c0afb80 msgr2=0x7f300c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f300c0afb80 0x7f300c0b2090 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f301c031920 tx=0x7f301c0319a0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 -- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024101e40 msgr2=0x7f3024104310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024101e40 0x7f3024104310 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f301400d740 tx=0x7f301400dc00 comp rx=0 tx=0).stop 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 -- 172.21.15.98:0/3842627067 shutdown_connections 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f300c0afb80 0x7f300c0b2090 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3024109790 0x7f3024102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3024106c20 0x7f30241029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.809 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 --2- 172.21.15.98:0/3842627067 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3024101e40 0x7f3024104310 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 -- 172.21.15.98:0/3842627067 >> 172.21.15.98:0/3842627067 conn(0x7f30240fd350 msgr2=0x7f3024107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:10.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 -- 172.21.15.98:0/3842627067 shutdown_connections 2024-04-25T16:33:10.810 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.795+0000 7f302a8aa640 1 -- 172.21.15.98:0/3842627067 wait complete. 2024-04-25T16:33:10.810 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:33:10.810 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:33:10.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 -- 172.21.15.98:0/938972607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 msgr2=0x7f04e410bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 --2- 172.21.15.98:0/938972607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 0x7f04e410bcd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f04dc009ca0 tx=0x7f04dc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 -- 172.21.15.98:0/938972607 shutdown_connections 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 --2- 172.21.15.98:0/938972607 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 0x7f04e410bcd0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 --2- 172.21.15.98:0/938972607 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e4106f50 0x7f04e4109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 --2- 172.21.15.98:0/938972607 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e41045f0 0x7f04e4106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 -- 172.21.15.98:0/938972607 >> 172.21.15.98:0/938972607 conn(0x7f04e40fdbf0 msgr2=0x7f04e41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 -- 172.21.15.98:0/938972607 shutdown_connections 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.919+0000 7f04e9b38640 1 -- 172.21.15.98:0/938972607 wait complete. 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e9b38640 1 Processor -- start 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e9b38640 1 -- start start 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e9b38640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e41045f0 0x7f04e419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e9b38640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e4106f50 0x7f04e419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e9b38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 0x7f04e41a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.921 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e9b38640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f04e410f530 con 0x7f04e41098b0 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e9b38640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f04e410f3b0 con 0x7f04e41045f0 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e9b38640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f04e410f6b0 con 0x7f04e4106f50 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 0x7f04e41a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e3fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 0x7f04e41a15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44344/0 (socket says 172.21.15.98:44344) 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e3fff640 1 -- 172.21.15.98:0/1430376167 learned_addr learned my addr 172.21.15.98:0/1430376167 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e37fe640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e41045f0 0x7f04e419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04dbfff640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e4106f50 0x7f04e419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e3fff640 1 -- 172.21.15.98:0/1430376167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e4106f50 msgr2=0x7f04e419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e3fff640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e4106f50 0x7f04e419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e3fff640 1 -- 172.21.15.98:0/1430376167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e41045f0 msgr2=0x7f04e419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e3fff640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e41045f0 0x7f04e419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.920+0000 7f04e3fff640 1 -- 172.21.15.98:0/1430376167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04e41a1c40 con 0x7f04e41098b0 2024-04-25T16:33:10.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.921+0000 7f04e3fff640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 0x7f04e41a15d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f04dc02aeb0 tx=0x7f04dc0093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:10.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.921+0000 7f04e17fa640 1 -- 172.21.15.98:0/1430376167 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc003db0 con 0x7f04e41098b0 2024-04-25T16:33:10.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.921+0000 7f04e17fa640 1 -- 172.21.15.98:0/1430376167 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f04dc0044d0 con 0x7f04e41098b0 2024-04-25T16:33:10.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.921+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04e41a1ed0 con 0x7f04e41098b0 2024-04-25T16:33:10.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.921+0000 7f04e17fa640 1 -- 172.21.15.98:0/1430376167 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc030ca0 con 0x7f04e41098b0 2024-04-25T16:33:10.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.921+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04e41a2330 con 0x7f04e41098b0 2024-04-25T16:33:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.922+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04e406a190 con 0x7f04e41098b0 2024-04-25T16:33:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.923+0000 7f04e17fa640 1 -- 172.21.15.98:0/1430376167 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f04dc004040 con 0x7f04e41098b0 2024-04-25T16:33:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.924+0000 7f04e17fa640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f04d00afb80 0x7f04d00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.924+0000 7f04e37fe640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f04d00afb80 0x7f04d00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.924+0000 7f04e17fa640 1 -- 172.21.15.98:0/1430376167 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f04dc1004a0 con 0x7f04e41098b0 2024-04-25T16:33:10.924 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.924+0000 7f04e37fe640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f04d00afb80 0x7f04d00b2090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f04d4005fd0 tx=0x7f04d400b390 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:10.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:10.929+0000 7f04e17fa640 1 -- 172.21.15.98:0/1430376167 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04dc0c8c80 con 0x7f04e41098b0 2024-04-25T16:33:11.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.165+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04e402dc10 con 0x7f04e41098b0 2024-04-25T16:33:11.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.165+0000 7f04e17fa640 1 -- 172.21.15.98:0/1430376167 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11935 (secure 0 0 0) 0x7f04dc0c8580 con 0x7f04e41098b0 2024-04-25T16:33:11.165 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:33:11.165 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:32:32.844737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24494,24473,24457,24484,24539]},"id":1}]} 2024-04-25T16:33:11.165 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T16:33:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.167+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f04d00afb80 msgr2=0x7f04d00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.167+0000 7f04e9b38640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f04d00afb80 0x7f04d00b2090 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f04d4005fd0 tx=0x7f04d400b390 comp rx=0 tx=0).stop 2024-04-25T16:33:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 msgr2=0x7f04e41a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 0x7f04e41a15d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f04dc02aeb0 tx=0x7f04dc0093e0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 shutdown_connections 2024-04-25T16:33:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f04d00afb80 0x7f04d00b2090 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f04e41098b0 0x7f04e41a15d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f04e4106f50 0x7f04e419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 --2- 172.21.15.98:0/1430376167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04e41045f0 0x7f04e419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 >> 172.21.15.98:0/1430376167 conn(0x7f04e40fdbf0 msgr2=0x7f04e4105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 shutdown_connections 2024-04-25T16:33:11.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.168+0000 7f04e9b38640 1 -- 172.21.15.98:0/1430376167 wait complete. 2024-04-25T16:33:11.178 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:33:11.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 -- 172.21.15.98:0/1895689100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204109a30 msgr2=0x7f520410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.290 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 --2- 172.21.15.98:0/1895689100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204109a30 0x7f520410be50 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f51fc009ca0 tx=0x7f51fc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 -- 172.21.15.98:0/1895689100 shutdown_connections 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 --2- 172.21.15.98:0/1895689100 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204109a30 0x7f520410be50 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 --2- 172.21.15.98:0/1895689100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5204104d30 0x7f5204105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 --2- 172.21.15.98:0/1895689100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204103af0 0x7f52041047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 -- 172.21.15.98:0/1895689100 >> 172.21.15.98:0/1895689100 conn(0x7f52040fece0 msgr2=0x7f5204101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 -- 172.21.15.98:0/1895689100 shutdown_connections 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 -- 172.21.15.98:0/1895689100 wait complete. 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 Processor -- start 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.289+0000 7f520a267640 1 -- start start 2024-04-25T16:33:11.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f520a267640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204103af0 0x7f520419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f520a267640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204104d30 0x7f520419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f520a267640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5204109a30 0x7f52041a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f520a267640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f520410f520 con 0x7f5204103af0 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f520a267640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f520410f3a0 con 0x7f5204104d30 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f520a267640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f520410f6a0 con 0x7f5204109a30 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5203fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204103af0 0x7f520419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5203fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204103af0 0x7f520419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44372/0 (socket says 172.21.15.98:44372) 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5203fff640 1 -- 172.21.15.98:0/1101358435 learned_addr learned my addr 172.21.15.98:0/1101358435 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f52037fe640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204104d30 0x7f520419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5208804640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5204109a30 0x7f52041a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5203fff640 1 -- 172.21.15.98:0/1101358435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5204109a30 msgr2=0x7f52041a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5203fff640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5204109a30 0x7f52041a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5203fff640 1 -- 172.21.15.98:0/1101358435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204104d30 msgr2=0x7f520419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5203fff640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204104d30 0x7f520419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f5203fff640 1 -- 172.21.15.98:0/1101358435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52041a1c70 con 0x7f5204103af0 2024-04-25T16:33:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.290+0000 7f52037fe640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204104d30 0x7f520419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.291+0000 7f5203fff640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204103af0 0x7f520419a130 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f51f4005fd0 tx=0x7f51f400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.291+0000 7f52017fa640 1 -- 172.21.15.98:0/1101358435 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4017070 con 0x7f5204103af0 2024-04-25T16:33:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.291+0000 7f52017fa640 1 -- 172.21.15.98:0/1101358435 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51f4009c30 con 0x7f5204103af0 2024-04-25T16:33:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.291+0000 7f520a267640 1 -- 172.21.15.98:0/1101358435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52041a1f60 con 0x7f5204103af0 2024-04-25T16:33:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.291+0000 7f52017fa640 1 -- 172.21.15.98:0/1101358435 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f4003c90 con 0x7f5204103af0 2024-04-25T16:33:11.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.291+0000 7f520a267640 1 -- 172.21.15.98:0/1101358435 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52041a23c0 con 0x7f5204103af0 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52111f9740 1 Processor -- start 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52111f9740 1 -- start start 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52111f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e7e390 0x55e873220810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52111f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 0x55e873220d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52111f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e873221a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52111f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e873184d80 con 0x55e872e5d6b0 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52111f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e87313ea20 con 0x55e872e7e390 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52111f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e873180d40 con 0x55e872e554d0 2024-04-25T16:33:11.300 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.293+0000 7f52037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e873221a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e873221a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44382/0 (socket says 172.21.15.98:44382) 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52017fa640 1 -- 172.21.15.98:0/1101358435 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f51f4004020 con 0x7f5204103af0 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52037fe640 1 -- 172.21.15.98:0/1995512610 learned_addr learned my addr 172.21.15.98:0/1995512610 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f5208804640 1 --2- 172.21.15.98:0/1995512610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e7e390 0x55e873220810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52037fe640 1 -- 172.21.15.98:0/1995512610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 msgr2=0x55e873220d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52037fe640 1 --2- 172.21.15.98:0/1995512610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 0x55e873220d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52037fe640 1 -- 172.21.15.98:0/1995512610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e7e390 msgr2=0x55e873220810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52037fe640 1 --2- 172.21.15.98:0/1995512610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e7e390 0x55e873220810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52037fe640 1 -- 172.21.15.98:0/1995512610 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e873149c30 con 0x55e872e5d6b0 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52037fe640 1 --2- 172.21.15.98:0/1995512610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e873221a20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f51f8011470 tx=0x7f51f8011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae66ca9c836766d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f51ea7fc640 1 -- 172.21.15.98:0/1995512610 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8004020 con 0x55e872e5d6b0 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f51ea7fc640 1 -- 172.21.15.98:0/1995512610 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51f80041c0 con 0x55e872e5d6b0 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f51ea7fc640 1 -- 172.21.15.98:0/1995512610 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f800f820 con 0x55e872e5d6b0 2024-04-25T16:33:11.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52017fa640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51ec0afb80 0x7f51ec0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f5208804640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51ec0afb80 0x7f51ec0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.294+0000 7f52017fa640 1 -- 172.21.15.98:0/1101358435 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f51f40dedb0 con 0x7f5204103af0 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 -- 172.21.15.98:0/1995512610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 msgr2=0x55e873221a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f5208804640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51ec0afb80 0x7f51ec0b2090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55e873221860 tx=0x7f51fc031000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 --2- 172.21.15.98:0/1995512610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e873221a20 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f51f8011470 tx=0x7f51f8011930 comp rx=0 tx=0).stop 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 -- 172.21.15.98:0/1995512610 shutdown_connections 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 --2- 172.21.15.98:0/1995512610 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e873221a20 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 --2- 172.21.15.98:0/1995512610 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 0x55e873220d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 --2- 172.21.15.98:0/1995512610 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e7e390 0x55e873220810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 -- 172.21.15.98:0/1995512610 >> 172.21.15.98:0/1995512610 conn(0x55e872e724e0 msgr2=0x55e87321c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 -- 172.21.15.98:0/1995512610 shutdown_connections 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 -- 172.21.15.98:0/1995512610 wait complete. 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.295+0000 7f52111f9740 1 -- start start 2024-04-25T16:33:11.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52111f9740 1 Processor -- start 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52111f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52111f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e87322aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52111f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e7e390 0x55e873231a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52111f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e873222530 con 0x55e872e5d6b0 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52111f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e87310bdc0 con 0x55e872e554d0 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52111f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e8732226b0 con 0x55e872e7e390 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52111f9740 10 client.0 _ll_register_callbacks cb 0x55e873127600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e87322aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e87322aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44386/0 (socket says 172.21.15.98:44386) 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 -- 172.21.15.98:0/2388197333 learned_addr learned my addr 172.21.15.98:0/2388197333 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f5203fff640 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 -- 172.21.15.98:0/2388197333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e7e390 msgr2=0x55e873231a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e7e390 0x55e873231a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 -- 172.21.15.98:0/2388197333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 msgr2=0x55e87322a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 -- 172.21.15.98:0/2388197333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e872e963f0 con 0x55e872e5d6b0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f52037fe640 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e87322aae0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f51f8005e00 tx=0x7f51f8003670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca1451614f8788cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.296+0000 7f51e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f51e37fe640 1 -- 172.21.15.98:0/2388197333 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8028020 con 0x55e872e5d6b0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f51e37fe640 1 -- 172.21.15.98:0/2388197333 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f51f8016040 con 0x55e872e5d6b0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 1 -- 172.21.15.98:0/2388197333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51f80041c0 con 0x55e872e5d6b0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f51e37fe640 1 -- 172.21.15.98:0/2388197333 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51f8033050 con 0x55e872e5d6b0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 10 client.15750 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 1 -- 172.21.15.98:0/2388197333 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e872f607d0 con 0x55e872e5d6b0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f51e37fe640 1 -- 172.21.15.98:0/2388197333 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51f80041c0 con 0x55e872e5d6b0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f51e37fe640 1 -- 172.21.15.98:0/2388197333 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f51f8004b10 con 0x55e872e5d6b0 2024-04-25T16:33:11.304 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 10 client.15750 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 10 client.15750 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 20 client.15750 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55e872e3cf60 0x55e873238170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 4 client.15750 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.297+0000 7f52111f9740 1 -- 172.21.15.98:0/2388197333 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e873192990 con 0x55e872e3cf60 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.298+0000 7f5203fff640 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55e872e3cf60 0x55e873238170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.298+0000 7f5203fff640 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55e872e3cf60 0x55e873238170 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a06640e130c495c2 server_cookie=9018b1cbca212852 in_seq=0 out_seq=0 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.298+0000 7f51e37fe640 10 client.15750 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f51e37fe640 1 -- 172.21.15.98:0/2388197333 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e873192990 con 0x55e872e3cf60 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f51e37fe640 10 client.15750 handle_command_reply: tid=0 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 client.15750 shutdown 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 -- 172.21.15.98:0/2388197333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 msgr2=0x55e87322aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e87322aae0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f51f8005e00 tx=0x7f51f8003670 comp rx=0 tx=0).stop 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 -- 172.21.15.98:0/2388197333 shutdown_connections 2024-04-25T16:33:11.305 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55e872e3cf60 0x55e873238170 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e7e390 0x55e873231a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e5d6b0 0x55e87322aae0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f51e37fe640 0 client.15750 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:33:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 --2- 172.21.15.98:0/2388197333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 -- 172.21.15.98:0/2388197333 >> 172.21.15.98:0/2388197333 conn(0x55e872e724e0 msgr2=0x55e87321b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 -- 172.21.15.98:0/2388197333 shutdown_connections 2024-04-25T16:33:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 1 -- 172.21.15.98:0/2388197333 wait complete. 2024-04-25T16:33:11.306 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.299+0000 7f52111f9740 20 client.15750 trim_cache size 0 max 16384 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52111f9740 1 Processor -- start 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52111f9740 1 -- start start 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52111f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 0x55e873217190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52111f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e5d6b0 0x55e873231960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52111f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873228d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52111f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52041ac9b0 con 0x55e872e7e390 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52111f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f52041ae4f0 con 0x55e872e5d6b0 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52111f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52041ad080 con 0x55e872e554d0 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873228d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873228d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44398/0 (socket says 172.21.15.98:44398) 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 -- 172.21.15.98:0/3515604135 learned_addr learned my addr 172.21.15.98:0/3515604135 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 -- 172.21.15.98:0/3515604135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 msgr2=0x55e873217190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f52037fe640 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e5d6b0 0x55e873231960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5203fff640 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 0x55e873217190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 0x55e873217190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 -- 172.21.15.98:0/3515604135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e5d6b0 msgr2=0x55e873231960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e5d6b0 0x55e873231960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 -- 172.21.15.98:0/3515604135 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8732259c0 con 0x55e872e7e390 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.313+0000 7f5208804640 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873228d20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f51fc035000 tx=0x7f51fc0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dca85a856effedeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f51e9ffb640 1 -- 172.21.15.98:0/3515604135 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc003a30 con 0x55e872e7e390 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f51e9ffb640 1 -- 172.21.15.98:0/3515604135 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51fc036050 con 0x55e872e7e390 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f51e9ffb640 1 -- 172.21.15.98:0/3515604135 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc030820 con 0x55e872e7e390 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 -- 172.21.15.98:0/3515604135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 msgr2=0x55e873228d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873228d20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f51fc035000 tx=0x7f51fc0095c0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 -- 172.21.15.98:0/3515604135 shutdown_connections 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873228d20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e5d6b0 0x55e873231960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 --2- 172.21.15.98:0/3515604135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e554d0 0x55e873217190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 -- 172.21.15.98:0/3515604135 >> 172.21.15.98:0/3515604135 conn(0x55e872e724e0 msgr2=0x55e8732252b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 -- 172.21.15.98:0/3515604135 shutdown_connections 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.314+0000 7f52111f9740 1 -- 172.21.15.98:0/3515604135 wait complete. 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 1 -- start start 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 1 Processor -- start 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e5d6b0 0x55e87322f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873230130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f52041ade20 con 0x55e872e7e390 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f52041ad750 con 0x55e872e554d0 2024-04-25T16:33:11.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f52041aa990 con 0x55e872e5d6b0 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52111f9740 10 client.0 _ll_register_callbacks cb 0x7f51fc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f5208804640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873230130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f5208804640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873230130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44406/0 (socket says 172.21.15.98:44406) 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f5208804640 1 -- 172.21.15.98:0/3919603365 learned_addr learned my addr 172.21.15.98:0/3919603365 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.315+0000 7f52037fe640 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e5d6b0 0x55e87322f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f5208804640 1 -- 172.21.15.98:0/3919603365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e5d6b0 msgr2=0x55e87322f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f5208804640 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e5d6b0 0x55e87322f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f5203fff640 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f5208804640 1 -- 172.21.15.98:0/3919603365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 msgr2=0x55e87322ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f5208804640 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f5208804640 1 -- 172.21.15.98:0/3919603365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8732306f0 con 0x55e872e7e390 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f5203fff640 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:11.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f5208804640 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873230130 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f51fc031840 tx=0x7f51fc031450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e927279a714ead server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f51e37fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f51e37fe640 1 -- 172.21.15.98:0/3919603365 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51fc004250 con 0x55e872e7e390 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f51e37fe640 1 -- 172.21.15.98:0/3919603365 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f51fc0043f0 con 0x55e872e7e390 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f52111f9740 1 -- 172.21.15.98:0/3919603365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51fc036050 con 0x55e872e7e390 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f51e37fe640 1 -- 172.21.15.98:0/3919603365 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51fc0045f0 con 0x55e872e7e390 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f52111f9740 10 client.15756 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f52111f9740 1 -- 172.21.15.98:0/3919603365 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e873231970 con 0x55e872e7e390 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f51e37fe640 1 -- 172.21.15.98:0/3919603365 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f51fc036050 con 0x55e872e7e390 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f51e37fe640 1 -- 172.21.15.98:0/3919603365 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f51fc051070 con 0x55e872e7e390 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f52111f9740 10 client.15756 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f52111f9740 10 client.15756 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f52111f9740 20 client.15756 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.316+0000 7f52111f9740 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55e872e123a0 0x55e8732a2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.317+0000 7f52111f9740 4 client.15756 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.317+0000 7f52111f9740 1 -- 172.21.15.98:0/3919603365 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e87318c800 con 0x55e872e123a0 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.317+0000 7f5203fff640 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55e872e123a0 0x55e8732a2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.317+0000 7f51e37fe640 10 client.15756 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.317+0000 7f5203fff640 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55e872e123a0 0x55e8732a2a00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=672cb990b5e44758 server_cookie=d49bee1cbfef15f5 in_seq=0 out_seq=0 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.317+0000 7f51e37fe640 1 -- 172.21.15.98:0/3919603365 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e87318c800 con 0x55e872e123a0 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.317+0000 7f51e37fe640 10 client.15756 handle_command_reply: tid=0 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.317+0000 7f52111f9740 1 client.15756 shutdown 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 -- 172.21.15.98:0/3919603365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 msgr2=0x55e873230130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873230130 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f51fc031840 tx=0x7f51fc031450 comp rx=0 tx=0).stop 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 -- 172.21.15.98:0/3919603365 shutdown_connections 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x55e872e123a0 0x55e8732a2a00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e872e7e390 0x55e873230130 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e872e5d6b0 0x55e87322f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f51e37fe640 0 client.15756 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:33:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 --2- 172.21.15.98:0/3919603365 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e872e554d0 0x55e87322ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 -- 172.21.15.98:0/3919603365 >> 172.21.15.98:0/3919603365 conn(0x55e872e724e0 msgr2=0x55e8732252b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 -- 172.21.15.98:0/3919603365 shutdown_connections 2024-04-25T16:33:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 1 -- 172.21.15.98:0/3919603365 wait complete. 2024-04-25T16:33:11.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f52111f9740 20 client.15756 trim_cache size 0 max 16384 2024-04-25T16:33:11.324 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 -- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51ec0afb80 msgr2=0x7f51ec0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51ec0afb80 0x7f51ec0b2090 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55e873221860 tx=0x7f51fc031000 comp rx=0 tx=0).stop 2024-04-25T16:33:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 -- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204103af0 msgr2=0x7f520419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204103af0 0x7f520419a130 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f51f4005fd0 tx=0x7f51f400dde0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 -- 172.21.15.98:0/1101358435 shutdown_connections 2024-04-25T16:33:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f51ec0afb80 0x7f51ec0b2090 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5204109a30 0x7f52041a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5204104d30 0x7f520419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 --2- 172.21.15.98:0/1101358435 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5204103af0 0x7f520419a130 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 -- 172.21.15.98:0/1101358435 >> 172.21.15.98:0/1101358435 conn(0x7f52040fece0 msgr2=0x7f5204105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 -- 172.21.15.98:0/1101358435 shutdown_connections 2024-04-25T16:33:11.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.318+0000 7f520a267640 1 -- 172.21.15.98:0/1101358435 wait complete. 2024-04-25T16:33:11.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:11 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1430376167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:33:11.329 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:33:11.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 -- 172.21.15.98:0/3148964054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908108940 msgr2=0x7fd90810ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/3148964054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908108940 0x7fd90810ad60 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd904009900 tx=0x7fd90402a410 comp rx=0 tx=0).stop 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 -- 172.21.15.98:0/3148964054 shutdown_connections 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/3148964054 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908108940 0x7fd90810ad60 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/3148964054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd908103c40 0x7fd908104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/3148964054 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd908102a00 0x7fd908103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 -- 172.21.15.98:0/3148964054 >> 172.21.15.98:0/3148964054 conn(0x7fd9080fdbf0 msgr2=0x7fd9081000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 -- 172.21.15.98:0/3148964054 shutdown_connections 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.440+0000 7fd90f6e2640 1 -- 172.21.15.98:0/3148964054 wait complete. 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90f6e2640 1 Processor -- start 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90f6e2640 1 -- start start 2024-04-25T16:33:11.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90f6e2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd908102a00 0x7fd90819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90f6e2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908103c40 0x7fd90819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90f6e2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd908108940 0x7fd9081a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90f6e2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd90810f4e0 con 0x7fd908103c40 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90f6e2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd90810f360 con 0x7fd908108940 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90f6e2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd90810f660 con 0x7fd908102a00 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90cc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908103c40 0x7fd90819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90cc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908103c40 0x7fd90819a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44412/0 (socket says 172.21.15.98:44412) 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90cc7d640 1 -- 172.21.15.98:0/1102867837 learned_addr learned my addr 172.21.15.98:0/1102867837 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90dc7f640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd908108940 0x7fd9081a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90cc7d640 1 -- 172.21.15.98:0/1102867837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd908102a00 msgr2=0x7fd90819a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd908102a00 0x7fd90819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90cc7d640 1 -- 172.21.15.98:0/1102867837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd908108940 msgr2=0x7fd9081a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd908108940 0x7fd9081a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.441+0000 7fd90cc7d640 1 -- 172.21.15.98:0/1102867837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9081a1c90 con 0x7fd908103c40 2024-04-25T16:33:11.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.442+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908103c40 0x7fd90819a690 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800c970 tx=0x7fd8f800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.442+0000 7fd8fe7fc640 1 -- 172.21.15.98:0/1102867837 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8007bb0 con 0x7fd908103c40 2024-04-25T16:33:11.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.442+0000 7fd8fe7fc640 1 -- 172.21.15.98:0/1102867837 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8f8007d50 con 0x7fd908103c40 2024-04-25T16:33:11.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.442+0000 7fd8fe7fc640 1 -- 172.21.15.98:0/1102867837 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8004e40 con 0x7fd908103c40 2024-04-25T16:33:11.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.442+0000 7fd90f6e2640 1 -- 172.21.15.98:0/1102867837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9081a1f80 con 0x7fd908103c40 2024-04-25T16:33:11.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.442+0000 7fd90f6e2640 1 -- 172.21.15.98:0/1102867837 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9081a23f0 con 0x7fd908103c40 2024-04-25T16:33:11.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd91658d740 1 Processor -- start 2024-04-25T16:33:11.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd8fe7fc640 1 -- 172.21.15.98:0/1102867837 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fd8f8005020 con 0x7fd908103c40 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd91658d740 1 -- start start 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd91658d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456bb390 0x563e45a5d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd91658d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e456924d0 0x563e45a5dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd91658d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e4569a6b0 0x563e45a5ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd91658d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e459c1d80 con 0x563e456bb390 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd91658d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563e4597ba20 con 0x563e456924d0 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd91658d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563e459bdd40 con 0x563e4569a6b0 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90dc7f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456bb390 0x563e45a5d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90dc7f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456bb390 0x563e45a5d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44418/0 (socket says 172.21.15.98:44418) 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90dc7f640 1 -- 172.21.15.98:0/665641361 learned_addr learned my addr 172.21.15.98:0/665641361 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90d47e640 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e456924d0 0x563e45a5dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e4569a6b0 0x563e45a5ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd8fe7fc640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd8f40afb80 0x7fd8f40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90dc7f640 1 -- 172.21.15.98:0/665641361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e4569a6b0 msgr2=0x563e45a5ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90dc7f640 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e4569a6b0 0x563e45a5ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90dc7f640 1 -- 172.21.15.98:0/665641361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e456924d0 msgr2=0x563e45a5dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90dc7f640 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e456924d0 0x563e45a5dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.445+0000 7fd90dc7f640 1 -- 172.21.15.98:0/665641361 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e45986c30 con 0x563e456bb390 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.446+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e4569a6b0 0x563e45a5ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.446+0000 7fd90dc7f640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd8f40afb80 0x7fd8f40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.446+0000 7fd8fe7fc640 1 -- 172.21.15.98:0/1102867837 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fd8f80def40 con 0x7fd908103c40 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.446+0000 7fd90dc7f640 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456bb390 0x563e45a5d810 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd904031460 tx=0x7fd9040317b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed4eef35fd40e2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.446+0000 7fd8f37fe640 1 -- 172.21.15.98:0/665641361 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd90404c070 con 0x563e456bb390 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.446+0000 7fd8f37fe640 1 -- 172.21.15.98:0/665641361 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd9040046c0 con 0x563e456bb390 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.446+0000 7fd90dc7f640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd8f40afb80 0x7fd8f40b2090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd904049000 tx=0x7fd90404a000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.446+0000 7fd8f37fe640 1 -- 172.21.15.98:0/665641361 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd904035040 con 0x563e456bb390 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 -- 172.21.15.98:0/665641361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456bb390 msgr2=0x563e45a5d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456bb390 0x563e45a5d810 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd904031460 tx=0x7fd9040317b0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 -- 172.21.15.98:0/665641361 shutdown_connections 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e4569a6b0 0x563e45a5ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e456924d0 0x563e45a5dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 --2- 172.21.15.98:0/665641361 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456bb390 0x563e45a5d810 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 -- 172.21.15.98:0/665641361 >> 172.21.15.98:0/665641361 conn(0x563e456af4e0 msgr2=0x563e45a59020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 -- 172.21.15.98:0/665641361 shutdown_connections 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 -- 172.21.15.98:0/665641361 wait complete. 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 -- start start 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.447+0000 7fd91658d740 1 Processor -- start 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd91658d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a67600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd91658d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45a67b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd91658d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e456bb390 0x563e45a6ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd91658d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563e45a5f530 con 0x563e456924d0 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd91658d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563e45948dc0 con 0x563e4569a6b0 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd91658d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563e45a5f6b0 con 0x563e456bb390 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd91658d740 10 client.0 _ll_register_callbacks cb 0x563e45964600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a67600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a67600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44424/0 (socket says 172.21.15.98:44424) 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 -- 172.21.15.98:0/3293240870 learned_addr learned my addr 172.21.15.98:0/3293240870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45a67b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90dc7f640 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e456bb390 0x563e45a6ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 -- 172.21.15.98:0/3293240870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e456bb390 msgr2=0x563e45a6ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e456bb390 0x563e45a6ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 -- 172.21.15.98:0/3293240870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 msgr2=0x563e45a67b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45a67b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 -- 172.21.15.98:0/3293240870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e456d33f0 con 0x563e456924d0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd90d47e640 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a67600 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd900003bf0 tx=0x7fd900004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d9f48439228eff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.448+0000 7fd8f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/3293240870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd900011850 con 0x563e456924d0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/3293240870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fd9000119f0 con 0x563e456924d0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 1 -- 172.21.15.98:0/3293240870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9040046c0 con 0x563e456924d0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/3293240870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd900022050 con 0x563e456924d0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 10 client.15768 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 1 -- 172.21.15.98:0/3293240870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e4579d7d0 con 0x563e456924d0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/3293240870 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd9040046c0 con 0x563e456924d0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/3293240870 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd90001ac40 con 0x563e456924d0 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 10 client.15768 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 10 client.15768 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 20 client.15768 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x563e45679f60 0x563e45a753f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 4 client.15768 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:11.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd91658d740 1 -- 172.21.15.98:0/3293240870 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e459cf990 con 0x563e45679f60 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.449+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x563e45679f60 0x563e45a753f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.450+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x563e45679f60 0x563e45a753f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=39d16ae1603b084a server_cookie=b65ff9f454a92670 in_seq=0 out_seq=0 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.450+0000 7fd8f0ff9640 10 client.15768 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/3293240870 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563e459cf990 con 0x563e45679f60 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd8f0ff9640 10 client.15768 handle_command_reply: tid=0 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 client.15768 shutdown 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 -- 172.21.15.98:0/3293240870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 msgr2=0x563e45a67600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a67600 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fd900003bf0 tx=0x7fd900004410 comp rx=0 tx=0).stop 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 -- 172.21.15.98:0/3293240870 shutdown_connections 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x563e45679f60 0x563e45a753f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e456bb390 0x563e45a6ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd8f0ff9640 0 client.15768 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45a67b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 --2- 172.21.15.98:0/3293240870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a67600 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 -- 172.21.15.98:0/3293240870 >> 172.21.15.98:0/3293240870 conn(0x563e456af4e0 msgr2=0x563e45a5bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 -- 172.21.15.98:0/3293240870 shutdown_connections 2024-04-25T16:33:11.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 1 -- 172.21.15.98:0/3293240870 wait complete. 2024-04-25T16:33:11.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.451+0000 7fd91658d740 20 client.15768 trim_cache size 0 max 16384 2024-04-25T16:33:11.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd91658d740 1 Processor -- start 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd91658d740 1 -- start start 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd91658d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 0x563e45a58a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd91658d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd91658d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45a58f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd91658d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9081ac9a0 con 0x563e456924d0 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd91658d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd9081ae4e0 con 0x563e4569a6b0 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd91658d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd9081ad070 con 0x563e45679f60 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44430/0 (socket says 172.21.15.98:44430) 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 -- 172.21.15.98:0/757985024 learned_addr learned my addr 172.21.15.98:0/757985024 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 -- 172.21.15.98:0/757985024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 msgr2=0x563e45a58a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/757985024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 0x563e45a58a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 -- 172.21.15.98:0/757985024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 msgr2=0x563e45a58f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/757985024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45a58f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 -- 172.21.15.98:0/757985024 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e45a6ee90 con 0x563e456924d0 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.465+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/757985024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6e950 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80a2f90 tx=0x7fd8f80e2600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549849f6e1e023c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd8f2ffd640 1 -- 172.21.15.98:0/757985024 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f801d040 con 0x563e456924d0 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd8f2ffd640 1 -- 172.21.15.98:0/757985024 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8f80a6f10 con 0x563e456924d0 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd8f2ffd640 1 -- 172.21.15.98:0/757985024 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f80dc860 con 0x563e456924d0 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 -- 172.21.15.98:0/757985024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 msgr2=0x563e45a6e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 --2- 172.21.15.98:0/757985024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6e950 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80a2f90 tx=0x7fd8f80e2600 comp rx=0 tx=0).stop 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 -- 172.21.15.98:0/757985024 shutdown_connections 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 --2- 172.21.15.98:0/757985024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45a58f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 --2- 172.21.15.98:0/757985024 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6e950 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 --2- 172.21.15.98:0/757985024 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 0x563e45a58a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 -- 172.21.15.98:0/757985024 >> 172.21.15.98:0/757985024 conn(0x563e456af4e0 msgr2=0x563e45a622f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 -- 172.21.15.98:0/757985024 shutdown_connections 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.466+0000 7fd91658d740 1 -- 172.21.15.98:0/757985024 wait complete. 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 1 -- start start 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 1 Processor -- start 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 0x563e45a6b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45adb750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fd9081ade10 con 0x563e456924d0 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fd9081ad740 con 0x563e4569a6b0 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fd9081aa980 con 0x563e45679f60 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd91658d740 10 client.0 _ll_register_callbacks cb 0x7fd900009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd90cc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd90cc7d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44436/0 (socket says 172.21.15.98:44436) 2024-04-25T16:33:11.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.467+0000 7fd90cc7d640 1 -- 172.21.15.98:0/2723310692 learned_addr learned my addr 172.21.15.98:0/2723310692 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd90dc7f640 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45adb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd90d47e640 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 0x563e45a6b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd90cc7d640 1 -- 172.21.15.98:0/2723310692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 msgr2=0x563e45a6b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 0x563e45a6b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd90cc7d640 1 -- 172.21.15.98:0/2723310692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 msgr2=0x563e45adb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45adb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd90cc7d640 1 -- 172.21.15.98:0/2723310692 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e45a6eb40 con 0x563e456924d0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd90cc7d640 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6bbd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80dc5c0 tx=0x7fd8f80a6b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1053ac71096dfe10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd8f0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/2723310692 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f801d040 con 0x563e456924d0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/2723310692 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fd8f80dcb50 con 0x563e456924d0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 1 -- 172.21.15.98:0/2723310692 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8f80a6f10 con 0x563e456924d0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/2723310692 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd8f80dcd50 con 0x563e456924d0 2024-04-25T16:33:11.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 10 client.15774 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 1 -- 172.21.15.98:0/2723310692 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e45adcbf0 con 0x563e456924d0 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/2723310692 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd8f80a6f10 con 0x563e456924d0 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/2723310692 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fd8f80a2880 con 0x563e456924d0 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 10 client.15774 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 10 client.15774 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 20 client.15774 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x563e456bb390 0x563e45adf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 4 client.15774 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.468+0000 7fd91658d740 1 -- 172.21.15.98:0/2723310692 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e459c9800 con 0x563e456bb390 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.469+0000 7fd90d47e640 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x563e456bb390 0x563e45adf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.469+0000 7fd8f0ff9640 10 client.15774 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.469+0000 7fd90d47e640 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x563e456bb390 0x563e45adf580 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=680b5a99e2a6b17c server_cookie=3dbf089b9aa113bb in_seq=0 out_seq=0 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.469+0000 7fd8f0ff9640 1 -- 172.21.15.98:0/2723310692 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e459c9800 con 0x563e456bb390 2024-04-25T16:33:11.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.469+0000 7fd8f0ff9640 10 client.15774 handle_command_reply: tid=0 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.469+0000 7fd91658d740 1 client.15774 shutdown 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 -- 172.21.15.98:0/2723310692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 msgr2=0x563e45a6bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6bbd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80dc5c0 tx=0x7fd8f80a6b60 comp rx=0 tx=0).stop 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 -- 172.21.15.98:0/2723310692 shutdown_connections 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x563e456bb390 0x563e45adf580 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563e4569a6b0 0x563e45adb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563e456924d0 0x563e45a6bbd0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 --2- 172.21.15.98:0/2723310692 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563e45679f60 0x563e45a6b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd8f0ff9640 0 client.15774 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 -- 172.21.15.98:0/2723310692 >> 172.21.15.98:0/2723310692 conn(0x563e456af4e0 msgr2=0x563e45a58950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 -- 172.21.15.98:0/2723310692 shutdown_connections 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 1 -- 172.21.15.98:0/2723310692 wait complete. 2024-04-25T16:33:11.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd91658d740 20 client.15774 trim_cache size 0 max 16384 2024-04-25T16:33:11.476 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 -- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd8f40afb80 msgr2=0x7fd8f40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd8f40afb80 0x7fd8f40b2090 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd904049000 tx=0x7fd90404a000 comp rx=0 tx=0).stop 2024-04-25T16:33:11.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 -- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908103c40 msgr2=0x7fd90819a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908103c40 0x7fd90819a690 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800c970 tx=0x7fd8f800ce30 comp rx=0 tx=0).stop 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 -- 172.21.15.98:0/1102867837 shutdown_connections 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fd8f40afb80 0x7fd8f40b2090 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd908108940 0x7fd9081a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fd908103c40 0x7fd90819a690 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 --2- 172.21.15.98:0/1102867837 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd908102a00 0x7fd90819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 -- 172.21.15.98:0/1102867837 >> 172.21.15.98:0/1102867837 conn(0x7fd9080fdbf0 msgr2=0x7fd908104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 -- 172.21.15.98:0/1102867837 shutdown_connections 2024-04-25T16:33:11.477 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.470+0000 7fd90f6e2640 1 -- 172.21.15.98:0/1102867837 wait complete. 2024-04-25T16:33:11.481 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:33:11.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:11 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1430376167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:33:11.597 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.595+0000 7f1b003a8640 1 -- 172.21.15.98:0/4281524451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8103940 msgr2=0x7f1af8103df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.595+0000 7f1b003a8640 1 --2- 172.21.15.98:0/4281524451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8103940 0x7f1af8103df0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1aec009900 tx=0x7f1aec02a410 comp rx=0 tx=0).stop 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 -- 172.21.15.98:0/4281524451 shutdown_connections 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 --2- 172.21.15.98:0/4281524451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1af8104480 0x7f1af810abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 --2- 172.21.15.98:0/4281524451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8103940 0x7f1af8103df0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 --2- 172.21.15.98:0/4281524451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1af8102f60 0x7f1af8103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 -- 172.21.15.98:0/4281524451 >> 172.21.15.98:0/4281524451 conn(0x7f1af80fd380 msgr2=0x7f1af80ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 -- 172.21.15.98:0/4281524451 shutdown_connections 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 -- 172.21.15.98:0/4281524451 wait complete. 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 Processor -- start 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 -- start start 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8102f60 0x7f1af8199030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1af8103940 0x7f1af8199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.598 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1af8104480 0x7f1af81a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810e400 con 0x7f1af8102f60 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810e280 con 0x7f1af8104480 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.596+0000 7f1b003a8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1af810e580 con 0x7f1af8103940 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8102f60 0x7f1af8199030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8102f60 0x7f1af8199030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44448/0 (socket says 172.21.15.98:44448) 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 -- 172.21.15.98:0/3027493641 learned_addr learned my addr 172.21.15.98:0/3027493641 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afd943640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1af8103940 0x7f1af8199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 -- 172.21.15.98:0/3027493641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1af8103940 msgr2=0x7f1af8199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1af8103940 0x7f1af8199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 -- 172.21.15.98:0/3027493641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1af8104480 msgr2=0x7f1af81a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1af8104480 0x7f1af81a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 -- 172.21.15.98:0/3027493641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af81a0b70 con 0x7f1af8102f60 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afd943640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1af8103940 0x7f1af8199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:11.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1afe144640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8102f60 0x7f1af8199030 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1af000db20 tx=0x7f1af000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1af77fe640 1 -- 172.21.15.98:0/3027493641 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0017070 con 0x7f1af8102f60 2024-04-25T16:33:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1af77fe640 1 -- 172.21.15.98:0/3027493641 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1af0003e00 con 0x7f1af8102f60 2024-04-25T16:33:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1af77fe640 1 -- 172.21.15.98:0/3027493641 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0010460 con 0x7f1af8102f60 2024-04-25T16:33:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1b003a8640 1 -- 172.21.15.98:0/3027493641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af81a0e60 con 0x7f1af8102f60 2024-04-25T16:33:11.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.597+0000 7f1b003a8640 1 -- 172.21.15.98:0/3027493641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af81a12d0 con 0x7f1af8102f60 2024-04-25T16:33:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1b0725a740 1 Processor -- start 2024-04-25T16:33:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1af77fe640 1 -- 172.21.15.98:0/3027493641 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f1af0004140 con 0x7f1af8102f60 2024-04-25T16:33:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1b0725a740 1 -- start start 2024-04-25T16:33:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5ff06390 0x562b602a8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fedd4d0 0x562b602a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fee56b0 0x562b602a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1b0725a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562b6020cd80 con 0x562b5fee56b0 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1b0725a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562b601c6a20 con 0x562b5ff06390 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1b0725a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562b60208d40 con 0x562b5fedd4d0 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1afd943640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fee56b0 0x562b602a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1afd943640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fee56b0 0x562b602a9a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44454/0 (socket says 172.21.15.98:44454) 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1afd943640 1 -- 172.21.15.98:0/3944502863 learned_addr learned my addr 172.21.15.98:0/3944502863 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.600+0000 7f1afe144640 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fedd4d0 0x562b602a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1afe945640 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5ff06390 0x562b602a8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1af77fe640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ae00afc00 0x7f1ae00b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1afd943640 1 -- 172.21.15.98:0/3944502863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fedd4d0 msgr2=0x562b602a8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1afd943640 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fedd4d0 0x562b602a8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1afd943640 1 -- 172.21.15.98:0/3944502863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5ff06390 msgr2=0x562b602a8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1afd943640 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5ff06390 0x562b602a8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1afd943640 1 -- 172.21.15.98:0/3944502863 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b601d1c30 con 0x562b5fee56b0 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1afe945640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ae00afc00 0x7f1ae00b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1afd943640 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fee56b0 0x562b602a9a20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1aec005760 tx=0x7f1aec031500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2beac85fd667bc19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1af77fe640 1 -- 172.21.15.98:0/3027493641 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f1af00defa0 con 0x7f1af8102f60 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1af4bfe640 1 -- 172.21.15.98:0/3944502863 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec035070 con 0x562b5fee56b0 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1af4bfe640 1 -- 172.21.15.98:0/3944502863 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1aec0316f0 con 0x562b5fee56b0 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.601+0000 7f1af4bfe640 1 -- 172.21.15.98:0/3944502863 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec003830 con 0x562b5fee56b0 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 -- 172.21.15.98:0/3944502863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fee56b0 msgr2=0x562b602a9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fee56b0 0x562b602a9a20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1aec005760 tx=0x7f1aec031500 comp rx=0 tx=0).stop 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1afe945640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ae00afc00 0x7f1ae00b2110 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x562b602a9860 tx=0x7f1ae80046c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 -- 172.21.15.98:0/3944502863 shutdown_connections 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fee56b0 0x562b602a9a20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fedd4d0 0x562b602a8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 --2- 172.21.15.98:0/3944502863 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5ff06390 0x562b602a8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 -- 172.21.15.98:0/3944502863 >> 172.21.15.98:0/3944502863 conn(0x562b5fefa4e0 msgr2=0x562b602a4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 -- 172.21.15.98:0/3944502863 shutdown_connections 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 -- 172.21.15.98:0/3944502863 wait complete. 2024-04-25T16:33:11.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.602+0000 7f1b0725a740 1 -- start start 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1b0725a740 1 Processor -- start 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fedd4d0 0x562b602b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fee56b0 0x562b602b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5ff06390 0x562b602b9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1b0725a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562b602aa4d0 con 0x562b5fedd4d0 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1b0725a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562b60193dc0 con 0x562b5fee56b0 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1b0725a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562b602aa650 con 0x562b5ff06390 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1b0725a740 10 client.0 _ll_register_callbacks cb 0x562b601af600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fedd4d0 0x562b602b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fedd4d0 0x562b602b25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44458/0 (socket says 172.21.15.98:44458) 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 -- 172.21.15.98:0/1399092971 learned_addr learned my addr 172.21.15.98:0/1399092971 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe945640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5ff06390 0x562b602b9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afd943640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fee56b0 0x562b602b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 -- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5ff06390 msgr2=0x562b602b9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5ff06390 0x562b602b9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 -- 172.21.15.98:0/1399092971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fee56b0 msgr2=0x562b602b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fee56b0 0x562b602b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 -- 172.21.15.98:0/1399092971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b5ff1e3f0 con 0x562b5fedd4d0 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe945640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5ff06390 0x562b602b9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afd943640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fee56b0 0x562b602b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1afe144640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fedd4d0 0x562b602b25c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1af00a6c70 tx=0x7f1af00dca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf359ecf55c77b84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1addffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.603+0000 7f1addffb640 1 -- 172.21.15.98:0/1399092971 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af0020070 con 0x562b5fedd4d0 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 1 -- 172.21.15.98:0/1399092971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1aec0316f0 con 0x562b5fedd4d0 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1addffb640 1 -- 172.21.15.98:0/1399092971 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f1af00e10d0 con 0x562b5fedd4d0 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1addffb640 1 -- 172.21.15.98:0/1399092971 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1af00a7670 con 0x562b5fedd4d0 2024-04-25T16:33:11.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 10 client.15786 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 1 -- 172.21.15.98:0/1399092971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562b5ffe87d0 con 0x562b5fedd4d0 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1addffb640 1 -- 172.21.15.98:0/1399092971 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1aec0316f0 con 0x562b5fedd4d0 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1addffb640 1 -- 172.21.15.98:0/1399092971 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1af00dcbb0 con 0x562b5fedd4d0 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 10 client.15786 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 10 client.15786 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 20 client.15786 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562b5fec4f60 0x562b602c03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 4 client.15786 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1b0725a740 1 -- 172.21.15.98:0/1399092971 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562b6021a990 con 0x562b5fec4f60 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.604+0000 7f1afd943640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562b5fec4f60 0x562b602c03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.605+0000 7f1addffb640 10 client.15786 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.605+0000 7f1afd943640 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562b5fec4f60 0x562b602c03b0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b72b2175049953e5 server_cookie=aca395071c8e674c in_seq=0 out_seq=0 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.605+0000 7f1addffb640 1 -- 172.21.15.98:0/1399092971 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562b6021a990 con 0x562b5fec4f60 2024-04-25T16:33:11.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.605+0000 7f1addffb640 10 client.15786 handle_command_reply: tid=0 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 client.15786 shutdown 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 -- 172.21.15.98:0/1399092971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fedd4d0 msgr2=0x562b602b25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fedd4d0 0x562b602b25c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1af00a6c70 tx=0x7f1af00dca20 comp rx=0 tx=0).stop 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 -- 172.21.15.98:0/1399092971 shutdown_connections 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562b5fec4f60 0x562b602c03b0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5ff06390 0x562b602b9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fee56b0 0x562b602b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1addffb640 0 client.15786 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 --2- 172.21.15.98:0/1399092971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5fedd4d0 0x562b602b25c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 -- 172.21.15.98:0/1399092971 >> 172.21.15.98:0/1399092971 conn(0x562b5fefa4e0 msgr2=0x562b602a6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 -- 172.21.15.98:0/1399092971 shutdown_connections 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 1 -- 172.21.15.98:0/1399092971 wait complete. 2024-04-25T16:33:11.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.606+0000 7f1b0725a740 20 client.15786 trim_cache size 0 max 16384 2024-04-25T16:33:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.619+0000 7f1b0725a740 1 Processor -- start 2024-04-25T16:33:11.624 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.619+0000 7f1b0725a740 1 -- start start 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.619+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 0x562b602b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.619+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 0x562b602b7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.619+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b602b8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.619+0000 7f1b0725a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81aa740 con 0x562b5ff06390 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.619+0000 7f1b0725a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81ac280 con 0x562b5fedd4d0 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.619+0000 7f1b0725a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81aae10 con 0x562b5fee56b0 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b602b8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b602b8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44466/0 (socket says 172.21.15.98:44466) 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 -- 172.21.15.98:0/3435177872 learned_addr learned my addr 172.21.15.98:0/3435177872 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 -- 172.21.15.98:0/3435177872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 msgr2=0x562b602b7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe144640 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 0x562b602b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afd943640 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 0x562b602b7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 0x562b602b7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 -- 172.21.15.98:0/3435177872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 msgr2=0x562b602b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 0x562b602b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 -- 172.21.15.98:0/3435177872 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562b602acdc0 con 0x562b5ff06390 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1afe945640 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b602b8310 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8011e20 tx=0x7f1ae8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f68c81e52bdb27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1adffff640 1 -- 172.21.15.98:0/3435177872 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8013be0 con 0x562b5ff06390 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1adffff640 1 -- 172.21.15.98:0/3435177872 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ae8013d80 con 0x562b5ff06390 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.620+0000 7f1adffff640 1 -- 172.21.15.98:0/3435177872 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae801d710 con 0x562b5ff06390 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 -- 172.21.15.98:0/3435177872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 msgr2=0x562b602b8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b602b8310 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8011e20 tx=0x7f1ae8014000 comp rx=0 tx=0).stop 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 -- 172.21.15.98:0/3435177872 shutdown_connections 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b602b8310 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 0x562b602b7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 --2- 172.21.15.98:0/3435177872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 0x562b602b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 -- 172.21.15.98:0/3435177872 >> 172.21.15.98:0/3435177872 conn(0x562b5fefa4e0 msgr2=0x562b602b2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 -- 172.21.15.98:0/3435177872 shutdown_connections 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 -- 172.21.15.98:0/3435177872 wait complete. 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 -- start start 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.621+0000 7f1b0725a740 1 Processor -- start 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 0x562b603234c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 0x562b60323a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b6032a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81abbb0 con 0x562b5ff06390 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81ab4e0 con 0x562b5fedd4d0 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f1af81a8720 con 0x562b5fee56b0 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 10 client.0 _ll_register_callbacks cb 0x7f1af00a2e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b6032a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b6032a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44480/0 (socket says 172.21.15.98:44480) 2024-04-25T16:33:11.627 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 -- 172.21.15.98:0/2606519860 learned_addr learned my addr 172.21.15.98:0/2606519860 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 -- 172.21.15.98:0/2606519860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 msgr2=0x562b60323a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 0x562b60323a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 -- 172.21.15.98:0/2606519860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 msgr2=0x562b603234c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 0x562b603234c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 -- 172.21.15.98:0/2606519860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562b6032af20 con 0x562b5ff06390 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1afe945640 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b6032a960 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005940 tx=0x7f1ae8027670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7ffdb76fd80452 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1addffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1addffb640 1 -- 172.21.15.98:0/2606519860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8035020 con 0x562b5ff06390 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1addffb640 1 -- 172.21.15.98:0/2606519860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f1ae801d490 con 0x562b5ff06390 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 1 -- 172.21.15.98:0/2606519860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ae8013d80 con 0x562b5ff06390 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1addffb640 1 -- 172.21.15.98:0/2606519860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1ae803d050 con 0x562b5ff06390 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 10 client.15792 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.622+0000 7f1b0725a740 1 -- 172.21.15.98:0/2606519860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562b6032c1a0 con 0x562b5ff06390 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1addffb640 1 -- 172.21.15.98:0/2606519860 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1ae8013d80 con 0x562b5ff06390 2024-04-25T16:33:11.628 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1addffb640 1 -- 172.21.15.98:0/2606519860 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f1ae8030070 con 0x562b5ff06390 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1b0725a740 10 client.15792 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1b0725a740 10 client.15792 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1b0725a740 20 client.15792 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1b0725a740 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562b5fe9a3a0 0x562b6032eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1b0725a740 4 client.15792 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1b0725a740 1 -- 172.21.15.98:0/2606519860 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562b60214800 con 0x562b5fe9a3a0 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.623+0000 7f1afe144640 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562b5fe9a3a0 0x562b6032eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1addffb640 10 client.15792 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1afe144640 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562b5fe9a3a0 0x562b6032eb90 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=81a375b51e7df5d7 server_cookie=e9b181dd62c7a460 in_seq=0 out_seq=0 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1addffb640 1 -- 172.21.15.98:0/2606519860 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562b60214800 con 0x562b5fe9a3a0 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1addffb640 10 client.15792 handle_command_reply: tid=0 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 client.15792 shutdown 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 -- 172.21.15.98:0/2606519860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 msgr2=0x562b6032a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b6032a960 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005940 tx=0x7f1ae8027670 comp rx=0 tx=0).stop 2024-04-25T16:33:11.629 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 -- 172.21.15.98:0/2606519860 shutdown_connections 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562b5fe9a3a0 0x562b6032eb90 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562b5ff06390 0x562b6032a960 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562b5fee56b0 0x562b60323a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1addffb640 0 client.15792 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 --2- 172.21.15.98:0/2606519860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562b5fedd4d0 0x562b603234c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 -- 172.21.15.98:0/2606519860 >> 172.21.15.98:0/2606519860 conn(0x562b5fefa4e0 msgr2=0x562b602a75b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 -- 172.21.15.98:0/2606519860 shutdown_connections 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 1 -- 172.21.15.98:0/2606519860 wait complete. 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b0725a740 20 client.15792 trim_cache size 0 max 16384 2024-04-25T16:33:11.630 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b003a8640 1 -- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ae00afc00 msgr2=0x7f1ae00b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b003a8640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ae00afc00 0x7f1ae00b2110 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x562b602a9860 tx=0x7f1ae80046c0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b003a8640 1 -- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8102f60 msgr2=0x7f1af8199030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.624+0000 7f1b003a8640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8102f60 0x7f1af8199030 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f1af000db20 tx=0x7f1af000b260 comp rx=0 tx=0).stop 2024-04-25T16:33:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.625+0000 7f1b003a8640 1 -- 172.21.15.98:0/3027493641 shutdown_connections 2024-04-25T16:33:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.625+0000 7f1b003a8640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f1ae00afc00 0x7f1ae00b2110 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.625+0000 7f1b003a8640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1af8104480 0x7f1af81a0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.625+0000 7f1b003a8640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1af8103940 0x7f1af8199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.625+0000 7f1b003a8640 1 --2- 172.21.15.98:0/3027493641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1af8102f60 0x7f1af8199030 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.625+0000 7f1b003a8640 1 -- 172.21.15.98:0/3027493641 >> 172.21.15.98:0/3027493641 conn(0x7f1af80fd380 msgr2=0x7f1af8104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.625+0000 7f1b003a8640 1 -- 172.21.15.98:0/3027493641 shutdown_connections 2024-04-25T16:33:11.632 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.625+0000 7f1b003a8640 1 -- 172.21.15.98:0/3027493641 wait complete. 2024-04-25T16:33:11.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:33:11.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:11 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1430376167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 -- 172.21.15.98:0/1074587854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 msgr2=0x7f24d81036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 --2- 172.21.15.98:0/1074587854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 0x7f24d81036e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009900 tx=0x7f24cc02a410 comp rx=0 tx=0).stop 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 -- 172.21.15.98:0/1074587854 shutdown_connections 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 --2- 172.21.15.98:0/1074587854 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8108920 0x7f24d810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 --2- 172.21.15.98:0/1074587854 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24d8103c20 0x7f24d8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 --2- 172.21.15.98:0/1074587854 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 0x7f24d81036e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 -- 172.21.15.98:0/1074587854 >> 172.21.15.98:0/1074587854 conn(0x7f24d80fdbd0 msgr2=0x7f24d8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 -- 172.21.15.98:0/1074587854 shutdown_connections 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.746+0000 7f24dfc84640 1 -- 172.21.15.98:0/1074587854 wait complete. 2024-04-25T16:33:11.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dfc84640 1 Processor -- start 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dfc84640 1 -- start start 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dfc84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 0x7f24d8199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dfc84640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8103c20 0x7f24d81998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dfc84640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24d8108920 0x7f24d81a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dfc84640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24d810e390 con 0x7f24d81029e0 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dfc84640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24d810e210 con 0x7f24d8103c20 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dfc84640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24d810e510 con 0x7f24d8108920 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 0x7f24d8199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 0x7f24d8199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44500/0 (socket says 172.21.15.98:44500) 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 -- 172.21.15.98:0/2442394750 learned_addr learned my addr 172.21.15.98:0/2442394750 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dd21f640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8103c20 0x7f24d81998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 -- 172.21.15.98:0/2442394750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24d8108920 msgr2=0x7f24d81a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24d8108920 0x7f24d81a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 -- 172.21.15.98:0/2442394750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8103c20 msgr2=0x7f24d81998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8103c20 0x7f24d81998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 -- 172.21.15.98:0/2442394750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d81a0eb0 con 0x7f24d81029e0 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dd21f640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8103c20 0x7f24d81998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24dda20640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 0x7f24d8199370 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f24cc005760 tx=0x7f24cc005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.747+0000 7f24caffd640 1 -- 172.21.15.98:0/2442394750 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc046070 con 0x7f24d81029e0 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.748+0000 7f24dfc84640 1 -- 172.21.15.98:0/2442394750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d8069e90 con 0x7f24d81029e0 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.748+0000 7f24caffd640 1 -- 172.21.15.98:0/2442394750 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24cc004350 con 0x7f24d81029e0 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.748+0000 7f24caffd640 1 -- 172.21.15.98:0/2442394750 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc035050 con 0x7f24d81029e0 2024-04-25T16:33:11.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.748+0000 7f24dfc84640 1 -- 172.21.15.98:0/2442394750 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d806a380 con 0x7f24d81029e0 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24e6c08740 1 Processor -- start 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24e6c08740 1 -- start start 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24caffd640 1 -- 172.21.15.98:0/2442394750 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f24cc049070 con 0x7f24d81029e0 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 0x5558a41e5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 0x5558a41e5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a41e6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24e6c08740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5558a4149d80 con 0x5558a3e226b0 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24e6c08740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5558a4103a20 con 0x5558a3e43390 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24e6c08740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5558a4145d40 con 0x5558a3e1a4d0 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24dd21f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a41e6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.761 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24dd21f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a41e6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44506/0 (socket says 172.21.15.98:44506) 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24dd21f640 1 -- 172.21.15.98:0/2471201279 learned_addr learned my addr 172.21.15.98:0/2471201279 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24dda20640 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 0x5558a41e5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.751+0000 7f24de221640 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 0x5558a41e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24dd21f640 1 -- 172.21.15.98:0/2471201279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 msgr2=0x5558a41e5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24dd21f640 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 0x5558a41e5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24dd21f640 1 -- 172.21.15.98:0/2471201279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 msgr2=0x5558a41e5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24dd21f640 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 0x5558a41e5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24dd21f640 1 -- 172.21.15.98:0/2471201279 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558a410ec30 con 0x5558a3e226b0 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24dd21f640 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a41e6a20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009c20 tx=0x7f24d4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c295449a2143fa99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24caffd640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24c40afb80 0x7f24c40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24c3fff640 1 -- 172.21.15.98:0/2471201279 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d4004030 con 0x5558a3e226b0 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24de221640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24c40afb80 0x7f24c40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.762 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24c3fff640 1 -- 172.21.15.98:0/2471201279 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24d40035f0 con 0x5558a3e226b0 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24c3fff640 1 -- 172.21.15.98:0/2471201279 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d40126b0 con 0x5558a3e226b0 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.752+0000 7f24caffd640 1 -- 172.21.15.98:0/2442394750 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f24cc100c80 con 0x7f24d81029e0 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 -- 172.21.15.98:0/2471201279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 msgr2=0x5558a41e6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a41e6a20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009c20 tx=0x7f24d4010740 comp rx=0 tx=0).stop 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24de221640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24c40afb80 0x7f24c40b2090 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x5558a41e6860 tx=0x7f24d0009b60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 -- 172.21.15.98:0/2471201279 shutdown_connections 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a41e6a20 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 0x5558a41e5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 --2- 172.21.15.98:0/2471201279 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 0x5558a41e5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 -- 172.21.15.98:0/2471201279 >> 172.21.15.98:0/2471201279 conn(0x5558a3e374e0 msgr2=0x5558a41e1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 -- 172.21.15.98:0/2471201279 shutdown_connections 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 -- 172.21.15.98:0/2471201279 wait complete. 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.753+0000 7f24e6c08740 1 -- start start 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24e6c08740 1 Processor -- start 2024-04-25T16:33:11.763 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e1a4d0 0x5558a41ef510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 0x5558a41efa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 0x5558a41f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24e6c08740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5558a41e7530 con 0x5558a3e1a4d0 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24e6c08740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5558a40d0dc0 con 0x5558a3e43390 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24e6c08740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5558a41e76b0 con 0x5558a3e226b0 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24e6c08740 10 client.0 _ll_register_callbacks cb 0x5558a40ec600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dda20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e1a4d0 0x5558a41ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dda20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e1a4d0 0x5558a41ef510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44508/0 (socket says 172.21.15.98:44508) 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dda20640 1 -- 172.21.15.98:0/774720266 learned_addr learned my addr 172.21.15.98:0/774720266 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24de221640 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 0x5558a41f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dda20640 1 -- 172.21.15.98:0/774720266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 msgr2=0x5558a41efa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dd21f640 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 0x5558a41efa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.764 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dda20640 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 0x5558a41efa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dda20640 1 -- 172.21.15.98:0/774720266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 msgr2=0x5558a41f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dda20640 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 0x5558a41f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.754+0000 7f24dda20640 1 -- 172.21.15.98:0/774720266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558a3e5b3f0 con 0x5558a3e1a4d0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24dda20640 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e1a4d0 0x5558a41ef510 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f24cc003fd0 tx=0x7f24cc009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6e4359c459026e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24c17fa640 1 -- 172.21.15.98:0/774720266 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc0c83c0 con 0x5558a3e1a4d0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24c17fa640 1 -- 172.21.15.98:0/774720266 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f24cc0c90c0 con 0x5558a3e1a4d0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 1 -- 172.21.15.98:0/774720266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24d40035f0 con 0x5558a3e1a4d0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24c17fa640 1 -- 172.21.15.98:0/774720266 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24cc0d13f0 con 0x5558a3e1a4d0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 10 client.15804 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 1 -- 172.21.15.98:0/774720266 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5558a3f257d0 con 0x5558a3e1a4d0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24c17fa640 1 -- 172.21.15.98:0/774720266 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24d40035f0 con 0x5558a3e1a4d0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24c17fa640 1 -- 172.21.15.98:0/774720266 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24cc0d1620 con 0x5558a3e1a4d0 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 10 client.15804 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.765 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 10 client.15804 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 20 client.15804 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x5558a3e01f60 0x5558a41fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 4 client.15804 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.755+0000 7f24e6c08740 1 -- 172.21.15.98:0/774720266 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558a4157990 con 0x5558a3e01f60 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.756+0000 7f24dd21f640 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x5558a3e01f60 0x5558a41fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.756+0000 7f24c17fa640 10 client.15804 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.756+0000 7f24dd21f640 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x5558a3e01f60 0x5558a41fd1d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fb522a6231747be7 server_cookie=b54c53fdf70e0894 in_seq=0 out_seq=0 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24c17fa640 1 -- 172.21.15.98:0/774720266 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5558a4157990 con 0x5558a3e01f60 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24c17fa640 10 client.15804 handle_command_reply: tid=0 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 client.15804 shutdown 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 -- 172.21.15.98:0/774720266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e1a4d0 msgr2=0x5558a41ef510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e1a4d0 0x5558a41ef510 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f24cc003fd0 tx=0x7f24cc009260 comp rx=0 tx=0).stop 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 -- 172.21.15.98:0/774720266 shutdown_connections 2024-04-25T16:33:11.766 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x5558a3e01f60 0x5558a41fd1d0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e43390 0x5558a41f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 0x5558a41efa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24c17fa640 0 client.15804 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:33:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 --2- 172.21.15.98:0/774720266 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e1a4d0 0x5558a41ef510 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 -- 172.21.15.98:0/774720266 >> 172.21.15.98:0/774720266 conn(0x5558a3e374e0 msgr2=0x5558a41e3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 -- 172.21.15.98:0/774720266 shutdown_connections 2024-04-25T16:33:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 1 -- 172.21.15.98:0/774720266 wait complete. 2024-04-25T16:33:11.767 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.757+0000 7f24e6c08740 20 client.15804 trim_cache size 0 max 16384 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24e6c08740 1 Processor -- start 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24e6c08740 1 -- start start 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e01f60 0x5558a41ef7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e1a4d0 0x5558a42019c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 0x5558a4201f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24e6c08740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81aa6c0 con 0x5558a3e01f60 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24e6c08740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81ac200 con 0x5558a3e1a4d0 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24e6c08740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81aad90 con 0x5558a3e226b0 2024-04-25T16:33:11.772 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dda20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e01f60 0x5558a41ef7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dda20640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e01f60 0x5558a41ef7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44520/0 (socket says 172.21.15.98:44520) 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dda20640 1 -- 172.21.15.98:0/3534540383 learned_addr learned my addr 172.21.15.98:0/3534540383 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dd21f640 1 --2- 172.21.15.98:0/3534540383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e1a4d0 0x5558a42019c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dda20640 1 -- 172.21.15.98:0/3534540383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 msgr2=0x5558a4201f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dda20640 1 --2- 172.21.15.98:0/3534540383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 0x5558a4201f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dda20640 1 -- 172.21.15.98:0/3534540383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e1a4d0 msgr2=0x5558a42019c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dda20640 1 --2- 172.21.15.98:0/3534540383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e1a4d0 0x5558a42019c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.771+0000 7f24dda20640 1 -- 172.21.15.98:0/3534540383 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558a41ed600 con 0x5558a3e01f60 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.772+0000 7f24dda20640 1 --2- 172.21.15.98:0/3534540383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e01f60 0x5558a41ef7f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0c5ec0 tx=0x7f24cc0e32c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d677403c845af377 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.772+0000 7f24c37fe640 1 -- 172.21.15.98:0/3534540383 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc0c8a20 con 0x5558a3e01f60 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.772+0000 7f24c37fe640 1 -- 172.21.15.98:0/3534540383 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24cc0c8760 con 0x5558a3e01f60 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.772+0000 7f24c37fe640 1 -- 172.21.15.98:0/3534540383 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24cc0d1620 con 0x5558a3e01f60 2024-04-25T16:33:11.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- 172.21.15.98:0/3534540383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e01f60 msgr2=0x5558a41ef7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 --2- 172.21.15.98:0/3534540383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e01f60 0x5558a41ef7f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0c5ec0 tx=0x7f24cc0e32c0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- 172.21.15.98:0/3534540383 shutdown_connections 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 --2- 172.21.15.98:0/3534540383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e226b0 0x5558a4201f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 --2- 172.21.15.98:0/3534540383 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e1a4d0 0x5558a42019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 --2- 172.21.15.98:0/3534540383 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e01f60 0x5558a41ef7f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- 172.21.15.98:0/3534540383 >> 172.21.15.98:0/3534540383 conn(0x5558a3e374e0 msgr2=0x5558a41ef0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- 172.21.15.98:0/3534540383 shutdown_connections 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- 172.21.15.98:0/3534540383 wait complete. 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- start start 2024-04-25T16:33:11.774 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 Processor -- start 2024-04-25T16:33:11.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e01f60 0x5558a41f5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 0x5558a41f5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a4263850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81abb30 con 0x5558a3e226b0 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81ab460 con 0x5558a3e01f60 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f24d81a86a0 con 0x5558a3e1a4d0 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.773+0000 7f24e6c08740 10 client.0 _ll_register_callbacks cb 0x7f24cc104300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a4263850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a4263850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44526/0 (socket says 172.21.15.98:44526) 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 -- 172.21.15.98:0/4049402241 learned_addr learned my addr 172.21.15.98:0/4049402241 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24dda20640 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e01f60 0x5558a41f5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24dd21f640 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 0x5558a41f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 -- 172.21.15.98:0/4049402241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 msgr2=0x5558a41f5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 0x5558a41f5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 -- 172.21.15.98:0/4049402241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e01f60 msgr2=0x5558a41f5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e01f60 0x5558a41f5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 -- 172.21.15.98:0/4049402241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558a4263e40 con 0x5558a3e226b0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24de221640 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a4263850 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f24d0011c90 tx=0x7f24d0009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f92c65e1bf0c964 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24c17fa640 1 -- 172.21.15.98:0/4049402241 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d0011cc0 con 0x5558a3e226b0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24c17fa640 1 -- 172.21.15.98:0/4049402241 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f24d001f430 con 0x5558a3e226b0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24e6c08740 1 -- 172.21.15.98:0/4049402241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24cc0c8760 con 0x5558a3e226b0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24c17fa640 1 -- 172.21.15.98:0/4049402241 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f24d0011e60 con 0x5558a3e226b0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24e6c08740 10 client.15810 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24e6c08740 1 -- 172.21.15.98:0/4049402241 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5558a4265190 con 0x5558a3e226b0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.774+0000 7f24c17fa640 1 -- 172.21.15.98:0/4049402241 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f24cc0c8760 con 0x5558a3e226b0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24c17fa640 1 -- 172.21.15.98:0/4049402241 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f24d00287c0 con 0x5558a3e226b0 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24e6c08740 10 client.15810 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24e6c08740 10 client.15810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24e6c08740 20 client.15810 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24e6c08740 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x5558a3e43390 0x5558a4267c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24e6c08740 4 client.15810 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24e6c08740 1 -- 172.21.15.98:0/4049402241 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5558a4151800 con 0x5558a3e43390 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24dda20640 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x5558a3e43390 0x5558a4267c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24c17fa640 10 client.15810 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.775+0000 7f24dda20640 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x5558a3e43390 0x5558a4267c00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=34608c628bbfb4ff server_cookie=cc8ff4a04bcdf891 in_seq=0 out_seq=0 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24c17fa640 1 -- 172.21.15.98:0/4049402241 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5558a4151800 con 0x5558a3e43390 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24c17fa640 10 client.15810 handle_command_reply: tid=0 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 client.15810 shutdown 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 -- 172.21.15.98:0/4049402241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 msgr2=0x5558a4263850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a4263850 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f24d0011c90 tx=0x7f24d0009500 comp rx=0 tx=0).stop 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 -- 172.21.15.98:0/4049402241 shutdown_connections 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x5558a3e43390 0x5558a4267c00 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5558a3e226b0 0x5558a4263850 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5558a3e1a4d0 0x5558a41f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 --2- 172.21.15.98:0/4049402241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5558a3e01f60 0x5558a41f5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24c17fa640 0 client.15810 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:33:11.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 -- 172.21.15.98:0/4049402241 >> 172.21.15.98:0/4049402241 conn(0x5558a3e374e0 msgr2=0x5558a41e3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 -- 172.21.15.98:0/4049402241 shutdown_connections 2024-04-25T16:33:11.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 1 -- 172.21.15.98:0/4049402241 wait complete. 2024-04-25T16:33:11.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24e6c08740 20 client.15810 trim_cache size 0 max 16384 2024-04-25T16:33:11.782 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:11.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 -- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24c40afb80 msgr2=0x7f24c40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24c40afb80 0x7f24c40b2090 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x5558a41e6860 tx=0x7f24d0009b60 comp rx=0 tx=0).stop 2024-04-25T16:33:11.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 -- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 msgr2=0x7f24d8199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 0x7f24d8199370 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f24cc005760 tx=0x7f24cc005e20 comp rx=0 tx=0).stop 2024-04-25T16:33:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 -- 172.21.15.98:0/2442394750 shutdown_connections 2024-04-25T16:33:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f24c40afb80 0x7f24c40b2090 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24d8108920 0x7f24d81a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d8103c20 0x7f24d81998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 --2- 172.21.15.98:0/2442394750 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f24d81029e0 0x7f24d8199370 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.776+0000 7f24dfc84640 1 -- 172.21.15.98:0/2442394750 >> 172.21.15.98:0/2442394750 conn(0x7f24d80fdbd0 msgr2=0x7f24d8104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.777+0000 7f24dfc84640 1 -- 172.21.15.98:0/2442394750 shutdown_connections 2024-04-25T16:33:11.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.777+0000 7f24dfc84640 1 -- 172.21.15.98:0/2442394750 wait complete. 2024-04-25T16:33:11.787 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:33:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/89342101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58104d30 msgr2=0x7f9d58107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/89342101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58104d30 0x7f9d58107150 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009900 tx=0x7f9d4802a410 comp rx=0 tx=0).stop 2024-04-25T16:33:11.899 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/89342101 shutdown_connections 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/89342101 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d58107690 0x7f9d58107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/89342101 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58104d30 0x7f9d58107150 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/89342101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d581023d0 0x7f9d581047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/89342101 >> 172.21.15.98:0/89342101 conn(0x7f9d580fdbc0 msgr2=0x7f9d58100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/89342101 shutdown_connections 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/89342101 wait complete. 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 Processor -- start 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.898+0000 7f9d5fb4d640 1 -- start start 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5fb4d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d581023d0 0x7f9d581a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5fb4d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d58104d30 0x7f9d581a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5fb4d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58107690 0x7f9d581a7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.900 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5fb4d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58115ac0 con 0x7f9d58107690 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5fb4d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58115940 con 0x7f9d581023d0 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5fb4d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d58115c40 con 0x7f9d58104d30 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58107690 0x7f9d581a7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58107690 0x7f9d581a7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44542/0 (socket says 172.21.15.98:44542) 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 -- 172.21.15.98:0/2805102283 learned_addr learned my addr 172.21.15.98:0/2805102283 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d58104d30 0x7f9d581a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d581023d0 0x7f9d581a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 -- 172.21.15.98:0/2805102283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d58104d30 msgr2=0x7f9d581a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d58104d30 0x7f9d581a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 -- 172.21.15.98:0/2805102283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d581023d0 msgr2=0x7f9d581a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d581023d0 0x7f9d581a0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 -- 172.21.15.98:0/2805102283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d581a8290 con 0x7f9d58107690 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5e0ea640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58107690 0x7f9d581a7c20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9d5000e9d0 tx=0x7f9d5000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d4effd640 1 -- 172.21.15.98:0/2805102283 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5000cd80 con 0x7f9d58107690 2024-04-25T16:33:11.901 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d4effd640 1 -- 172.21.15.98:0/2805102283 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d50005020 con 0x7f9d58107690 2024-04-25T16:33:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/2805102283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d581a8580 con 0x7f9d58107690 2024-04-25T16:33:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d4effd640 1 -- 172.21.15.98:0/2805102283 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d500116d0 con 0x7f9d58107690 2024-04-25T16:33:11.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.899+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/2805102283 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d581a8ad0 con 0x7f9d58107690 2024-04-25T16:33:11.908 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d4effd640 1 -- 172.21.15.98:0/2805102283 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f9d50004a40 con 0x7f9d58107690 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d66a03740 1 Processor -- start 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d66a03740 1 -- start start 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6fb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a6fbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 0x55b89a6fca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d66a03740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b89a65fd80 con 0x55b89a3304d0 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d66a03740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b89a619a20 con 0x55b89a3386b0 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d66a03740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b89a65bd40 con 0x55b89a359390 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.902+0000 7f9d5d8e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a6fbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d8e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a6fbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44558/0 (socket says 172.21.15.98:44558) 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d8e9640 1 -- 172.21.15.98:0/3504352860 learned_addr learned my addr 172.21.15.98:0/3504352860 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6fb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d4effd640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9d440afb80 0x7f9d440b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 0x55b89a6fca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d8e9640 1 -- 172.21.15.98:0/3504352860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 msgr2=0x55b89a6fb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6fb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d8e9640 1 -- 172.21.15.98:0/3504352860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 msgr2=0x55b89a6fca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 0x55b89a6fca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d8e9640 1 -- 172.21.15.98:0/3504352860 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b89a624c30 con 0x55b89a3304d0 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5e0ea640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9d440afb80 0x7f9d440b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d4effd640 1 -- 172.21.15.98:0/2805102283 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f9d500df6e0 con 0x7f9d58107690 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a6fbd90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009b60 tx=0x7f9d5400e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36e3f5ac84dad24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d43fff640 1 -- 172.21.15.98:0/3504352860 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54016070 con 0x55b89a3304d0 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d43fff640 1 -- 172.21.15.98:0/3504352860 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d54009e50 con 0x55b89a3304d0 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.903+0000 7f9d43fff640 1 -- 172.21.15.98:0/3504352860 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54004d80 con 0x55b89a3304d0 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 -- 172.21.15.98:0/3504352860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 msgr2=0x55b89a6fbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d5e0ea640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9d440afb80 0x7f9d440b2090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9d50074fd0 tx=0x7f9d50076040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a6fbd90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009b60 tx=0x7f9d5400e6c0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 -- 172.21.15.98:0/3504352860 shutdown_connections 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 0x55b89a6fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a6fbd90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 --2- 172.21.15.98:0/3504352860 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6fb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 -- 172.21.15.98:0/3504352860 >> 172.21.15.98:0/3504352860 conn(0x55b89a34d4e0 msgr2=0x55b89a6f7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 -- 172.21.15.98:0/3504352860 shutdown_connections 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.904+0000 7f9d66a03740 1 -- 172.21.15.98:0/3504352860 wait complete. 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 1 -- start start 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 1 Processor -- start 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a705510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a3386b0 0x55b89a705a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a359390 0x55b89a70c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b89a6fd530 con 0x55b89a3304d0 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b89a5e6dc0 con 0x55b89a359390 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b89a6fd6b0 con 0x55b89a3386b0 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d66a03740 10 client.0 _ll_register_callbacks cb 0x55b89a602600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d5d0e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a705510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d5d0e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a705510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44562/0 (socket says 172.21.15.98:44562) 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d5d0e8640 1 -- 172.21.15.98:0/1237270373 learned_addr learned my addr 172.21.15.98:0/1237270373 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a3386b0 0x55b89a705a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d5d0e8640 1 -- 172.21.15.98:0/1237270373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a3386b0 msgr2=0x55b89a705a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a3386b0 0x55b89a705a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d5d0e8640 1 -- 172.21.15.98:0/1237270373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a359390 msgr2=0x55b89a70c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.905+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a359390 0x55b89a70c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d5d0e8640 1 -- 172.21.15.98:0/1237270373 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b89a3713f0 con 0x55b89a3304d0 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a705510 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009900 tx=0x7f9d48032d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7b3fd3574c7d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d417fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d417fa640 1 -- 172.21.15.98:0/1237270373 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d48035040 con 0x55b89a3304d0 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d417fa640 1 -- 172.21.15.98:0/1237270373 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f9d48004450 con 0x55b89a3304d0 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 1 -- 172.21.15.98:0/1237270373 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d54009e50 con 0x55b89a3304d0 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d417fa640 1 -- 172.21.15.98:0/1237270373 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9d480035f0 con 0x55b89a3304d0 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 10 client.15822 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 1 -- 172.21.15.98:0/1237270373 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b89a43b7d0 con 0x55b89a3304d0 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d417fa640 1 -- 172.21.15.98:0/1237270373 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d54009e50 con 0x55b89a3304d0 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d417fa640 1 -- 172.21.15.98:0/1237270373 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9d4803d910 con 0x55b89a3304d0 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 10 client.15822 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 10 client.15822 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 20 client.15822 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b89a317f60 0x55b89a713210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 4 client.15822 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.906+0000 7f9d66a03740 1 -- 172.21.15.98:0/1237270373 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b89a66d990 con 0x55b89a317f60 2024-04-25T16:33:11.913 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.907+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b89a317f60 0x55b89a713210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.907+0000 7f9d417fa640 10 client.15822 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.907+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b89a317f60 0x55b89a713210 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20f40eacea0f2876 server_cookie=a45df85f99455fe5 in_seq=0 out_seq=0 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d417fa640 1 -- 172.21.15.98:0/1237270373 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b89a66d990 con 0x55b89a317f60 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d417fa640 10 client.15822 handle_command_reply: tid=0 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 client.15822 shutdown 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 -- 172.21.15.98:0/1237270373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 msgr2=0x55b89a705510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a705510 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009900 tx=0x7f9d48032d20 comp rx=0 tx=0).stop 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 -- 172.21.15.98:0/1237270373 shutdown_connections 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b89a317f60 0x55b89a713210 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a359390 0x55b89a70c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a3386b0 0x55b89a705a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 --2- 172.21.15.98:0/1237270373 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a705510 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d417fa640 0 client.15822 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 -- 172.21.15.98:0/1237270373 >> 172.21.15.98:0/1237270373 conn(0x55b89a34d4e0 msgr2=0x55b89a6f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.914 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 -- 172.21.15.98:0/1237270373 shutdown_connections 2024-04-25T16:33:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 1 -- 172.21.15.98:0/1237270373 wait complete. 2024-04-25T16:33:11.915 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.908+0000 7f9d66a03740 20 client.15822 trim_cache size 0 max 16384 2024-04-25T16:33:11.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.921+0000 7f9d66a03740 1 Processor -- start 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.921+0000 7f9d66a03740 1 -- start start 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3304d0 0x55b89a6f6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3386b0 0x55b89a70c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6f9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d66a03740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581b3080 con 0x55b89a3386b0 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d66a03740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581b4bc0 con 0x55b89a3304d0 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d66a03740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581b3750 con 0x55b89a359390 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3386b0 0x55b89a70c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3386b0 0x55b89a70c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44570/0 (socket says 172.21.15.98:44570) 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 -- 172.21.15.98:0/642960919 learned_addr learned my addr 172.21.15.98:0/642960919 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3304d0 0x55b89a6f6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6f9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 -- 172.21.15.98:0/642960919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 msgr2=0x55b89a6f9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6f9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 -- 172.21.15.98:0/642960919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3304d0 msgr2=0x55b89a6f6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3304d0 0x55b89a6f6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 -- 172.21.15.98:0/642960919 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b89a705e60 con 0x55b89a3386b0 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6f9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3386b0 0x55b89a70c940 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x55b89a6fb530 tx=0x7f9d540058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec735d7e623486be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d437fe640 1 -- 172.21.15.98:0/642960919 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5402e070 con 0x55b89a3386b0 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d437fe640 1 -- 172.21.15.98:0/642960919 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d54004820 con 0x55b89a3386b0 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.922+0000 7f9d437fe640 1 -- 172.21.15.98:0/642960919 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d54016040 con 0x55b89a3386b0 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 -- 172.21.15.98:0/642960919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3386b0 msgr2=0x55b89a70c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3386b0 0x55b89a70c940 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x55b89a6fb530 tx=0x7f9d540058f0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 -- 172.21.15.98:0/642960919 shutdown_connections 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a6f9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3386b0 0x55b89a70c940 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 --2- 172.21.15.98:0/642960919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3304d0 0x55b89a6f6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 -- 172.21.15.98:0/642960919 >> 172.21.15.98:0/642960919 conn(0x55b89a34d4e0 msgr2=0x55b89a705750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 -- 172.21.15.98:0/642960919 shutdown_connections 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 -- 172.21.15.98:0/642960919 wait complete. 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 -- start start 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.923+0000 7f9d66a03740 1 Processor -- start 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a70a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 0x55b89a70af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d66a03740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a70be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d66a03740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581b44f0 con 0x55b89a3304d0 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d66a03740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581b3e20 con 0x55b89a3386b0 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d66a03740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f9d581b1060 con 0x55b89a359390 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d66a03740 10 client.0 _ll_register_callbacks cb 0x7f9d48005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:11.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a70a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a70a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44578/0 (socket says 172.21.15.98:44578) 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 -- 172.21.15.98:0/822520777 learned_addr learned my addr 172.21.15.98:0/822520777 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 0x55b89a70af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a70be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 -- 172.21.15.98:0/822520777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 msgr2=0x55b89a70be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a70be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 -- 172.21.15.98:0/822520777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 msgr2=0x55b89a70af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 0x55b89a70af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 -- 172.21.15.98:0/822520777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b89a70c410 con 0x55b89a3304d0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d5d0e8640 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a70a9e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9d48032ac0 tx=0x7f9d48009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ef477f12c2077b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.924+0000 7f9d417fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d417fa640 1 -- 172.21.15.98:0/822520777 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d4805d030 con 0x55b89a3304d0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d417fa640 1 -- 172.21.15.98:0/822520777 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f9d48031a70 con 0x55b89a3304d0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 1 -- 172.21.15.98:0/822520777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d54004820 con 0x55b89a3304d0 2024-04-25T16:33:11.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d417fa640 1 -- 172.21.15.98:0/822520777 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9d48067050 con 0x55b89a3304d0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 10 client.15828 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 1 -- 172.21.15.98:0/822520777 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b89a77bd10 con 0x55b89a3304d0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d417fa640 1 -- 172.21.15.98:0/822520777 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9d54004820 con 0x55b89a3304d0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d417fa640 1 -- 172.21.15.98:0/822520777 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f9d48046510 con 0x55b89a3304d0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 10 client.15828 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 10 client.15828 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 20 client.15828 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b89a2ed3a0 0x55b89a77e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 4 client.15828 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d66a03740 1 -- 172.21.15.98:0/822520777 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b89a667800 con 0x55b89a2ed3a0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.925+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b89a2ed3a0 0x55b89a77e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d417fa640 10 client.15828 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d5d8e9640 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b89a2ed3a0 0x55b89a77e6c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dbb6d656129e62c server_cookie=74e486973af02e71 in_seq=0 out_seq=0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d417fa640 1 -- 172.21.15.98:0/822520777 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b89a667800 con 0x55b89a2ed3a0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d417fa640 10 client.15828 handle_command_reply: tid=0 2024-04-25T16:33:11.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 client.15828 shutdown 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 -- 172.21.15.98:0/822520777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 msgr2=0x55b89a70a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a70a9e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f9d48032ac0 tx=0x7f9d48009260 comp rx=0 tx=0).stop 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 -- 172.21.15.98:0/822520777 shutdown_connections 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b89a2ed3a0 0x55b89a77e6c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b89a359390 0x55b89a70be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d417fa640 0 client.15828 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b89a3386b0 0x55b89a70af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 --2- 172.21.15.98:0/822520777 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b89a3304d0 0x55b89a70a9e0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 -- 172.21.15.98:0/822520777 >> 172.21.15.98:0/822520777 conn(0x55b89a34d4e0 msgr2=0x55b89a705750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 -- 172.21.15.98:0/822520777 shutdown_connections 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 1 -- 172.21.15.98:0/822520777 wait complete. 2024-04-25T16:33:11.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.926+0000 7f9d66a03740 20 client.15828 trim_cache size 0 max 16384 2024-04-25T16:33:11.934 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:11.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9d440afb80 msgr2=0x7f9d440b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9d440afb80 0x7f9d440b2090 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9d50074fd0 tx=0x7f9d50076040 comp rx=0 tx=0).stop 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58107690 msgr2=0x7f9d581a7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58107690 0x7f9d581a7c20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9d5000e9d0 tx=0x7f9d5000ee90 comp rx=0 tx=0).stop 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/2805102283 shutdown_connections 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f9d440afb80 0x7f9d440b2090 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9d58107690 0x7f9d581a7c20 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9d58104d30 0x7f9d581a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 --2- 172.21.15.98:0/2805102283 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9d581023d0 0x7f9d581a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/2805102283 >> 172.21.15.98:0/2805102283 conn(0x7f9d580fdbc0 msgr2=0x7f9d58103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/2805102283 shutdown_connections 2024-04-25T16:33:11.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:11.927+0000 7f9d5fb4d640 1 -- 172.21.15.98:0/2805102283 wait complete. 2024-04-25T16:33:12.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:12 smithi098 ceph-mon[29162]: pgmap v377: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:33:12.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:12 smithi173 ceph-mon[32556]: pgmap v377: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:33:12.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:12 smithi203 ceph-mon[33115]: pgmap v377: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:33:12.938 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:33:12.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:33:12.938 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:33:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 -- 172.21.15.98:0/4257591075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4103c20 msgr2=0x7f45d4104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 --2- 172.21.15.98:0/4257591075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4103c20 0x7f45d4104030 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009900 tx=0x7f45d002a410 comp rx=0 tx=0).stop 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 -- 172.21.15.98:0/4257591075 shutdown_connections 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 --2- 172.21.15.98:0/4257591075 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d4108920 0x7f45d410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 --2- 172.21.15.98:0/4257591075 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4103c20 0x7f45d4104030 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 --2- 172.21.15.98:0/4257591075 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d41029e0 0x7f45d41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 -- 172.21.15.98:0/4257591075 >> 172.21.15.98:0/4257591075 conn(0x7f45d40fdbd0 msgr2=0x7f45d4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 -- 172.21.15.98:0/4257591075 shutdown_connections 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 -- 172.21.15.98:0/4257591075 wait complete. 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.067+0000 7f45db9ce640 1 Processor -- start 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45db9ce640 1 -- start start 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45db9ce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d41029e0 0x7f45d419a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45db9ce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d4103c20 0x7f45d419a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45db9ce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4108920 0x7f45d41a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45db9ce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45d410f4c0 con 0x7f45d4108920 2024-04-25T16:33:13.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45db9ce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45d410f340 con 0x7f45d41029e0 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45db9ce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45d410f640 con 0x7f45d4103c20 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4108920 0x7f45d41a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4108920 0x7f45d41a1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44592/0 (socket says 172.21.15.98:44592) 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 -- 172.21.15.98:0/1574657240 learned_addr learned my addr 172.21.15.98:0/1574657240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d976a640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d41029e0 0x7f45d419a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d8f69640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d4103c20 0x7f45d419a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 -- 172.21.15.98:0/1574657240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d4103c20 msgr2=0x7f45d419a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d4103c20 0x7f45d419a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 -- 172.21.15.98:0/1574657240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d41029e0 msgr2=0x7f45d419a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d41029e0 0x7f45d419a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 -- 172.21.15.98:0/1574657240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d41a1ff0 con 0x7f45d4108920 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d976a640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d41029e0 0x7f45d419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:13.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45d9f6b640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4108920 0x7f45d41a1980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009fd0 tx=0x7f45cc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.068+0000 7f45c67fc640 1 -- 172.21.15.98:0/1574657240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc017070 con 0x7f45d4108920 2024-04-25T16:33:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.069+0000 7f45db9ce640 1 -- 172.21.15.98:0/1574657240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d41a22e0 con 0x7f45d4108920 2024-04-25T16:33:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.069+0000 7f45c67fc640 1 -- 172.21.15.98:0/1574657240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45cc003e00 con 0x7f45d4108920 2024-04-25T16:33:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.069+0000 7f45c67fc640 1 -- 172.21.15.98:0/1574657240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc010460 con 0x7f45d4108920 2024-04-25T16:33:13.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.069+0000 7f45db9ce640 1 -- 172.21.15.98:0/1574657240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d41a2830 con 0x7f45d4108920 2024-04-25T16:33:13.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45e287e740 1 Processor -- start 2024-04-25T16:33:13.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45c67fc640 1 -- 172.21.15.98:0/1574657240 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f45cc003970 con 0x7f45d4108920 2024-04-25T16:33:13.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45e287e740 1 -- start start 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45e287e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993a9f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45e287e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993a9fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45e287e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45e287e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564993a04b00 con 0x5649936fe410 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45e287e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5649939a0920 con 0x5649936dd730 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45e287e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564993a00ac0 con 0x5649936d5550 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45d8f69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993a9f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45d8f69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993a9f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44604/0 (socket says 172.21.15.98:44604) 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45d8f69640 1 -- 172.21.15.98:0/3879764371 learned_addr learned my addr 172.21.15.98:0/3879764371 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.072+0000 7f45d976a640 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993a9fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45d8f69640 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45c67fc640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45c00afb80 0x7f45c00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45d8f69640 1 -- 172.21.15.98:0/3879764371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 msgr2=0x564993a9fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45d8f69640 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993a9fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45d8f69640 1 -- 172.21.15.98:0/3879764371 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 msgr2=0x564993aa6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45d8f69640 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45d8f69640 1 -- 172.21.15.98:0/3879764371 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649937de7e0 con 0x5649936fe410 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45d9f6b640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45c00afb80 0x7f45c00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45c67fc640 1 -- 172.21.15.98:0/1574657240 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f45cc0df430 con 0x7f45d4108920 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45d8f69640 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993a9f4a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f45d0006010 tx=0x7f45d00058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b29c9e8dc684e85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45bf7fe640 1 -- 172.21.15.98:0/3879764371 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0035070 con 0x5649936fe410 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45bf7fe640 1 -- 172.21.15.98:0/3879764371 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45d00044c0 con 0x5649936fe410 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.073+0000 7f45bf7fe640 1 -- 172.21.15.98:0/3879764371 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0003b10 con 0x5649936fe410 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 -- 172.21.15.98:0/3879764371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 msgr2=0x564993a9f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45d9f6b640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45c00afb80 0x7f45c00b2090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0a70a0 tx=0x7f45cc074c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993a9f4a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f45d0006010 tx=0x7f45d00058d0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 -- 172.21.15.98:0/3879764371 shutdown_connections 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993a9fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 --2- 172.21.15.98:0/3879764371 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993a9f4a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 -- 172.21.15.98:0/3879764371 >> 172.21.15.98:0/3879764371 conn(0x5649936f2560 msgr2=0x564993a9ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 -- 172.21.15.98:0/3879764371 shutdown_connections 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 -- 172.21.15.98:0/3879764371 wait complete. 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.074+0000 7f45e287e740 1 -- start start 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45e287e740 1 Processor -- start 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45e287e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993aa9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45e287e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa9860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45e287e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993ab07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45e287e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564993aa7310 con 0x5649936fe410 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45e287e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56499399e5d0 con 0x5649936dd730 2024-04-25T16:33:13.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45e287e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564993aa7490 con 0x5649936d5550 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45e287e740 10 client.0 _ll_register_callbacks cb 0x56499398c080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d8f69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993ab07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d8f69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993ab07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44616/0 (socket says 172.21.15.98:44616) 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d8f69640 1 -- 172.21.15.98:0/550154386 learned_addr learned my addr 172.21.15.98:0/550154386 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d976a640 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d8f69640 1 -- 172.21.15.98:0/550154386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 msgr2=0x564993aa9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d8f69640 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993aa9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d8f69640 1 -- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 msgr2=0x564993aa9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d8f69640 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d8f69640 1 -- 172.21.15.98:0/550154386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56499383ef70 con 0x5649936fe410 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.075+0000 7f45d976a640 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45d8f69640 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993ab07c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f45d0031460 tx=0x7f45d0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50f8f3f1ccccb39c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45bcff9640 1 -- 172.21.15.98:0/550154386 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0035030 con 0x5649936fe410 2024-04-25T16:33:13.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45bcff9640 1 -- 172.21.15.98:0/550154386 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f45d003b740 con 0x5649936fe410 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 1 -- 172.21.15.98:0/550154386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564993a12710 con 0x5649936fe410 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45bcff9640 1 -- 172.21.15.98:0/550154386 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f45d0059050 con 0x5649936fe410 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 10 client.15840 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 1 -- 172.21.15.98:0/550154386 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5649936d6170 con 0x5649936fe410 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45bcff9640 1 -- 172.21.15.98:0/550154386 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x564993a12710 con 0x5649936fe410 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45bcff9640 1 -- 172.21.15.98:0/550154386 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f45d0047a60 con 0x5649936fe410 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 10 client.15840 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 20 client.15840 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5649936bd1f0 0x564993ab6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 4 client.15840 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.076+0000 7f45e287e740 1 -- 172.21.15.98:0/550154386 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564993a0c580 con 0x5649936bd1f0 2024-04-25T16:33:13.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.077+0000 7f45d976a640 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5649936bd1f0 0x564993ab6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.077+0000 7f45bcff9640 10 client.15840 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.077+0000 7f45d976a640 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5649936bd1f0 0x564993ab6880 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cbee30c3d5dc8b9 server_cookie=308eb20e8e5d98cc in_seq=0 out_seq=0 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45bcff9640 1 -- 172.21.15.98:0/550154386 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564993a0c580 con 0x5649936bd1f0 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45bcff9640 10 client.15840 handle_command_reply: tid=0 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 client.15840 shutdown 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 -- 172.21.15.98:0/550154386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 msgr2=0x564993ab07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993ab07c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f45d0031460 tx=0x7f45d0005d50 comp rx=0 tx=0).stop 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 -- 172.21.15.98:0/550154386 shutdown_connections 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5649936bd1f0 0x564993ab6880 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936fe410 0x564993ab07c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936dd730 0x564993aa9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45bcff9640 0 client.15840 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 --2- 172.21.15.98:0/550154386 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993aa9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 -- 172.21.15.98:0/550154386 >> 172.21.15.98:0/550154386 conn(0x5649936f2560 msgr2=0x564993a99ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 -- 172.21.15.98:0/550154386 shutdown_connections 2024-04-25T16:33:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 1 -- 172.21.15.98:0/550154386 wait complete. 2024-04-25T16:33:13.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.078+0000 7f45e287e740 20 client.15840 trim_cache size 0 max 16384 2024-04-25T16:33:13.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.096+0000 7f45e287e740 1 Processor -- start 2024-04-25T16:33:13.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.096+0000 7f45e287e740 1 -- start start 2024-04-25T16:33:13.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45e287e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936d5550 0x564993aa3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45e287e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936dd730 0x564993a9e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45e287e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936fe410 0x564993a9e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45e287e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41ac870 con 0x5649936d5550 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45e287e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41ae3b0 con 0x5649936fe410 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45e287e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41acf40 con 0x5649936dd730 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936d5550 0x564993aa3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936d5550 0x564993aa3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44618/0 (socket says 172.21.15.98:44618) 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 -- 172.21.15.98:0/1147803073 learned_addr learned my addr 172.21.15.98:0/1147803073 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d976a640 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936dd730 0x564993a9e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 -- 172.21.15.98:0/1147803073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936dd730 msgr2=0x564993a9e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936dd730 0x564993a9e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 -- 172.21.15.98:0/1147803073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936fe410 msgr2=0x564993a9e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936fe410 0x564993a9e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 -- 172.21.15.98:0/1147803073 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564993a96600 con 0x5649936d5550 2024-04-25T16:33:13.101 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d976a640 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936dd730 0x564993a9e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45d8f69640 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936d5550 0x564993aa3e40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f45d003e420 tx=0x7f45d0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3baf2a0de3281291 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45beffd640 1 -- 172.21.15.98:0/1147803073 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0035040 con 0x5649936d5550 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45beffd640 1 -- 172.21.15.98:0/1147803073 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45d003eb60 con 0x5649936d5550 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.097+0000 7f45beffd640 1 -- 172.21.15.98:0/1147803073 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d0047820 con 0x5649936d5550 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 -- 172.21.15.98:0/1147803073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936d5550 msgr2=0x564993aa3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936d5550 0x564993aa3e40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f45d003e420 tx=0x7f45d0005720 comp rx=0 tx=0).stop 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 -- 172.21.15.98:0/1147803073 shutdown_connections 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936fe410 0x564993a9e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936dd730 0x564993a9e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 --2- 172.21.15.98:0/1147803073 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936d5550 0x564993aa3e40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 -- 172.21.15.98:0/1147803073 >> 172.21.15.98:0/1147803073 conn(0x5649936f2560 msgr2=0x564993a9dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 -- 172.21.15.98:0/1147803073 shutdown_connections 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 -- 172.21.15.98:0/1147803073 wait complete. 2024-04-25T16:33:13.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 -- start start 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.098+0000 7f45e287e740 1 Processor -- start 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45e287e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993aafb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45e287e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936dd730 0x564993ab00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45e287e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936fe410 0x564993b1cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45e287e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41adce0 con 0x5649936dd730 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45e287e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41ad610 con 0x5649936fe410 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45e287e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41aa850 con 0x5649936d5550 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45e287e740 10 client.0 _ll_register_callbacks cb 0x7f45d002aa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936dd730 0x564993ab00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936dd730 0x564993ab00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44626/0 (socket says 172.21.15.98:44626) 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 -- 172.21.15.98:0/3272331884 learned_addr learned my addr 172.21.15.98:0/3272331884 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d8f69640 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993aafb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 -- 172.21.15.98:0/3272331884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 msgr2=0x564993aafb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993aafb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 -- 172.21.15.98:0/3272331884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936fe410 msgr2=0x564993b1cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936fe410 0x564993b1cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 -- 172.21.15.98:0/3272331884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564993b1d110 con 0x5649936dd730 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d8f69640 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993aafb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45d976a640 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936dd730 0x564993ab00b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f45c8005fd0 tx=0x7f45c80128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b133ae30ad520d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.099+0000 7f45bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45bcff9640 1 -- 172.21.15.98:0/3272331884 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c801b790 con 0x5649936dd730 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45bcff9640 1 -- 172.21.15.98:0/3272331884 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f45c801b930 con 0x5649936dd730 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 1 -- 172.21.15.98:0/3272331884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45d003eb60 con 0x5649936dd730 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45bcff9640 1 -- 172.21.15.98:0/3272331884 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f45c802c050 con 0x5649936dd730 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 10 client.15846 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 1 -- 172.21.15.98:0/3272331884 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564993b1e250 con 0x5649936dd730 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45bcff9640 1 -- 172.21.15.98:0/3272331884 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f45d003eb60 con 0x5649936dd730 2024-04-25T16:33:13.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45bcff9640 1 -- 172.21.15.98:0/3272331884 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f45c800f040 con 0x5649936dd730 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 10 client.15846 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 10 client.15846 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 20 client.15846 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x564993692630 0x564993b20cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 4 client.15846 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.100+0000 7f45e287e740 1 -- 172.21.15.98:0/3272331884 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564993987ac0 con 0x564993692630 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.101+0000 7f45d8f69640 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x564993692630 0x564993b20cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.101+0000 7f45d8f69640 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x564993692630 0x564993b20cc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e658d7bb8c170d1d server_cookie=865c8c3fb2875e38 in_seq=0 out_seq=0 2024-04-25T16:33:13.105 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.101+0000 7f45bcff9640 10 client.15846 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45bcff9640 1 -- 172.21.15.98:0/3272331884 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564993987ac0 con 0x564993692630 2024-04-25T16:33:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45bcff9640 10 client.15846 handle_command_reply: tid=0 2024-04-25T16:33:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 client.15846 shutdown 2024-04-25T16:33:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 -- 172.21.15.98:0/3272331884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936dd730 msgr2=0x564993ab00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936dd730 0x564993ab00b0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f45c8005fd0 tx=0x7f45c80128e0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 -- 172.21.15.98:0/3272331884 shutdown_connections 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x564993692630 0x564993b20cc0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5649936fe410 0x564993b1cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5649936dd730 0x564993ab00b0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45bcff9640 0 client.15846 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 --2- 172.21.15.98:0/3272331884 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5649936d5550 0x564993aafb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 -- 172.21.15.98:0/3272331884 >> 172.21.15.98:0/3272331884 conn(0x5649936f2560 msgr2=0x564993a9e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.102+0000 7f45e287e740 1 -- 172.21.15.98:0/3272331884 shutdown_connections 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45e287e740 1 -- 172.21.15.98:0/3272331884 wait complete. 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45e287e740 20 client.15846 trim_cache size 0 max 16384 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:33:13.107 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-04-25T16:33:13.108 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "d57a327a-2484-4875-a9d7-f2740f5af6a0", 2024-04-25T16:33:13.108 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-04-25T16:33:13.108 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 -- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45c00afb80 msgr2=0x7f45c00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45c00afb80 0x7f45c00b2090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f45cc0a70a0 tx=0x7f45cc074c50 comp rx=0 tx=0).stop 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 -- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4108920 msgr2=0x7f45d41a1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4108920 0x7f45d41a1980 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f45cc009fd0 tx=0x7f45cc00b260 comp rx=0 tx=0).stop 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 -- 172.21.15.98:0/1574657240 shutdown_connections 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f45c00afb80 0x7f45c00b2090 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f45d4108920 0x7f45d41a1980 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f45d4103c20 0x7f45d419a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 --2- 172.21.15.98:0/1574657240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45d41029e0 0x7f45d419a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 -- 172.21.15.98:0/1574657240 >> 172.21.15.98:0/1574657240 conn(0x7f45d40fdbd0 msgr2=0x7f45d4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 -- 172.21.15.98:0/1574657240 shutdown_connections 2024-04-25T16:33:13.109 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.103+0000 7f45db9ce640 1 -- 172.21.15.98:0/1574657240 wait complete. 2024-04-25T16:33:13.113 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:33:13.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:33:13.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 -- 172.21.15.98:0/2814371127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674104d30 msgr2=0x7fc674105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 --2- 172.21.15.98:0/2814371127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674104d30 0x7fc674105140 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc664009900 tx=0x7fc66402a410 comp rx=0 tx=0).stop 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 -- 172.21.15.98:0/2814371127 shutdown_connections 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 --2- 172.21.15.98:0/2814371127 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc674109a30 0x7fc67410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 --2- 172.21.15.98:0/2814371127 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674104d30 0x7fc674105140 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 --2- 172.21.15.98:0/2814371127 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc674103af0 0x7fc6741047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 -- 172.21.15.98:0/2814371127 >> 172.21.15.98:0/2814371127 conn(0x7fc6740fece0 msgr2=0x7fc674101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 -- 172.21.15.98:0/2814371127 shutdown_connections 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.223+0000 7fc67bb84640 1 -- 172.21.15.98:0/2814371127 wait complete. 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67bb84640 1 Processor -- start 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67bb84640 1 -- start start 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67bb84640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc674103af0 0x7fc67419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67bb84640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc674104d30 0x7fc67419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67bb84640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674109a30 0x7fc6741a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67bb84640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc67410f4d0 con 0x7fc674109a30 2024-04-25T16:33:13.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67bb84640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc67410f350 con 0x7fc674103af0 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67bb84640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc67410f650 con 0x7fc674104d30 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67a121640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674109a30 0x7fc6741a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67a121640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674109a30 0x7fc6741a1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44634/0 (socket says 172.21.15.98:44634) 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67a121640 1 -- 172.21.15.98:0/1416484188 learned_addr learned my addr 172.21.15.98:0/1416484188 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc679920640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc674103af0 0x7fc67419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67911f640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc674104d30 0x7fc67419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67a121640 1 -- 172.21.15.98:0/1416484188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc674104d30 msgr2=0x7fc67419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67a121640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc674104d30 0x7fc67419a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67a121640 1 -- 172.21.15.98:0/1416484188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc674103af0 msgr2=0x7fc67419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67a121640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc674103af0 0x7fc67419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.224+0000 7fc67a121640 1 -- 172.21.15.98:0/1416484188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6741a1c90 con 0x7fc674109a30 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.225+0000 7fc67a121640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674109a30 0x7fc6741a1620 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc66c00eb10 tx=0x7fc66c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.225+0000 7fc66affd640 1 -- 172.21.15.98:0/1416484188 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c004b20 con 0x7fc674109a30 2024-04-25T16:33:13.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.225+0000 7fc66affd640 1 -- 172.21.15.98:0/1416484188 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc66c004cc0 con 0x7fc674109a30 2024-04-25T16:33:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.225+0000 7fc67bb84640 1 -- 172.21.15.98:0/1416484188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6741a1f80 con 0x7fc674109a30 2024-04-25T16:33:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.225+0000 7fc66affd640 1 -- 172.21.15.98:0/1416484188 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc66c010690 con 0x7fc674109a30 2024-04-25T16:33:13.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.225+0000 7fc67bb84640 1 -- 172.21.15.98:0/1416484188 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6741a23e0 con 0x7fc674109a30 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.227+0000 7fc682a3a740 1 Processor -- start 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.227+0000 7fc66affd640 1 -- 172.21.15.98:0/1416484188 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fc66c005020 con 0x7fc674109a30 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc682a3a740 1 -- start start 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead30390 0x560feb0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead074d0 0x560feb0d1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 0x560feb0d2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc682a3a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560feb036d80 con 0x560fead074d0 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc682a3a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560feaff08b0 con 0x560fead30390 2024-04-25T16:33:13.234 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc682a3a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560feb032d40 con 0x560fead0f6b0 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc679920640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead074d0 0x560feb0d1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc679920640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead074d0 0x560feb0d1fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44650/0 (socket says 172.21.15.98:44650) 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc679920640 1 -- 172.21.15.98:0/969399696 learned_addr learned my addr 172.21.15.98:0/969399696 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc67911f640 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead30390 0x560feb0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc66affd640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc6600afb80 0x7fc6600b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc67911f640 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 0x560feb0d2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc67a121640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc6600afb80 0x7fc6600b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc679920640 1 -- 172.21.15.98:0/969399696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 msgr2=0x560feb0d2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc679920640 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 0x560feb0d2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc679920640 1 -- 172.21.15.98:0/969399696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead30390 msgr2=0x560feb0d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc679920640 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead30390 0x560feb0d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc679920640 1 -- 172.21.15.98:0/969399696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560feafc1a40 con 0x560fead074d0 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.228+0000 7fc66affd640 1 -- 172.21.15.98:0/1416484188 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fc66c0df7a0 con 0x7fc674109a30 2024-04-25T16:33:13.235 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.229+0000 7fc679920640 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead074d0 0x560feb0d1fe0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc670009c60 tx=0x7fc670011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe4dba4948fbeb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.229+0000 7fc67a121640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc6600afb80 0x7fc6600b2090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0a7470 tx=0x7fc66c0ca040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.229+0000 7fc65ffff640 1 -- 172.21.15.98:0/969399696 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670009e10 con 0x560fead074d0 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.229+0000 7fc65ffff640 1 -- 172.21.15.98:0/969399696 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc670005020 con 0x560fead074d0 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.229+0000 7fc65ffff640 1 -- 172.21.15.98:0/969399696 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc670004a00 con 0x560fead074d0 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 -- 172.21.15.98:0/969399696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead074d0 msgr2=0x560feb0d1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead074d0 0x560feb0d1fe0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc670009c60 tx=0x7fc670011810 comp rx=0 tx=0).stop 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 -- 172.21.15.98:0/969399696 shutdown_connections 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 0x560feb0d2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead074d0 0x560feb0d1fe0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 --2- 172.21.15.98:0/969399696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead30390 0x560feb0d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 -- 172.21.15.98:0/969399696 >> 172.21.15.98:0/969399696 conn(0x560fead244e0 msgr2=0x560feb0cd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 -- 172.21.15.98:0/969399696 shutdown_connections 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 -- 172.21.15.98:0/969399696 wait complete. 2024-04-25T16:33:13.236 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 -- start start 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.230+0000 7fc682a3a740 1 Processor -- start 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead074d0 0x560feb0db810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 0x560feb0dbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead30390 0x560feb0e2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc682a3a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560feb0d3780 con 0x560fead30390 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc682a3a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560feafbdd10 con 0x560fead074d0 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc682a3a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560feb0d3900 con 0x560fead0f6b0 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc682a3a740 10 client.0 _ll_register_callbacks cb 0x560feafea210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead30390 0x560feb0e2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead30390 0x560feb0e2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44656/0 (socket says 172.21.15.98:44656) 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 -- 172.21.15.98:0/1695466785 learned_addr learned my addr 172.21.15.98:0/1695466785 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc679920640 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 0x560feb0dbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead074d0 0x560feb0db810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.237 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 -- 172.21.15.98:0/1695466785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 msgr2=0x560feb0dbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 0x560feb0dbd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 -- 172.21.15.98:0/1695466785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead074d0 msgr2=0x560feb0db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead074d0 0x560feb0db810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 -- 172.21.15.98:0/1695466785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560feae127d0 con 0x560fead30390 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc67911f640 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead30390 0x560feb0e2cb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc664009900 tx=0x7fc664040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea18492b987a2b55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc65d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc65d7fa640 1 -- 172.21.15.98:0/1695466785 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664035030 con 0x560fead30390 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc65d7fa640 1 -- 172.21.15.98:0/1695466785 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fc664032d30 con 0x560fead30390 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.231+0000 7fc682a3a740 1 -- 172.21.15.98:0/1695466785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc670005020 con 0x560fead30390 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc65d7fa640 1 -- 172.21.15.98:0/1695466785 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc664004020 con 0x560fead30390 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc682a3a740 10 client.15858 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc682a3a740 1 -- 172.21.15.98:0/1695466785 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560fead483f0 con 0x560fead30390 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc65d7fa640 1 -- 172.21.15.98:0/1695466785 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc670005020 con 0x560fead30390 2024-04-25T16:33:13.238 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc65d7fa640 1 -- 172.21.15.98:0/1695466785 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc66403a040 con 0x560fead30390 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc682a3a740 10 client.15858 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc682a3a740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc682a3a740 20 client.15858 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc682a3a740 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560feaceef60 0x560feb0e90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc682a3a740 4 client.15858 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc682a3a740 1 -- 172.21.15.98:0/1695466785 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560feb044990 con 0x560feaceef60 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.232+0000 7fc679920640 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560feaceef60 0x560feb0e90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.233+0000 7fc65d7fa640 10 client.15858 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.233+0000 7fc679920640 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560feaceef60 0x560feb0e90f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99d3dcc8613ad4ca server_cookie=984aea9a5bb84e9 in_seq=0 out_seq=0 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.233+0000 7fc65d7fa640 1 -- 172.21.15.98:0/1695466785 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560feb044990 con 0x560feaceef60 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.233+0000 7fc65d7fa640 10 client.15858 handle_command_reply: tid=0 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.233+0000 7fc682a3a740 1 client.15858 shutdown 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.233+0000 7fc682a3a740 1 -- 172.21.15.98:0/1695466785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead30390 msgr2=0x560feb0e2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.233+0000 7fc682a3a740 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead30390 0x560feb0e2cb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc664009900 tx=0x7fc664040040 comp rx=0 tx=0).stop 2024-04-25T16:33:13.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 1 -- 172.21.15.98:0/1695466785 shutdown_connections 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560feaceef60 0x560feb0e90f0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead30390 0x560feb0e2cb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead0f6b0 0x560feb0dbd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc65d7fa640 0 client.15858 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 1 --2- 172.21.15.98:0/1695466785 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560fead074d0 0x560feb0db810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 1 -- 172.21.15.98:0/1695466785 >> 172.21.15.98:0/1695466785 conn(0x560fead244e0 msgr2=0x560feb0d0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 1 -- 172.21.15.98:0/1695466785 shutdown_connections 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 1 -- 172.21.15.98:0/1695466785 wait complete. 2024-04-25T16:33:13.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.234+0000 7fc682a3a740 20 client.15858 trim_cache size 0 max 16384 2024-04-25T16:33:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.251+0000 7fc682a3a740 1 Processor -- start 2024-04-25T16:33:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc682a3a740 1 -- start start 2024-04-25T16:33:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 0x560feb0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 0x560feb0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb0e2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc682a3a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741acac0 con 0x560fead0f6b0 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc682a3a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741ae600 con 0x560feaceef60 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc682a3a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741ad190 con 0x560fead074d0 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb0e2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb0e2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44662/0 (socket says 172.21.15.98:44662) 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 -- 172.21.15.98:0/180573667 learned_addr learned my addr 172.21.15.98:0/180573667 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc679920640 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 0x560feb0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 0x560feb0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 -- 172.21.15.98:0/180573667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 msgr2=0x560feb0cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 0x560feb0cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 -- 172.21.15.98:0/180573667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 msgr2=0x560feb0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 0x560feb0ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 -- 172.21.15.98:0/180573667 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560fead080f0 con 0x560fead0f6b0 2024-04-25T16:33:13.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.252+0000 7fc67911f640 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb0e2c40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc664001eb0 tx=0x7fc664005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6b8a7fdef8470d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc65f7fe640 1 -- 172.21.15.98:0/180573667 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664035070 con 0x560fead0f6b0 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc65f7fe640 1 -- 172.21.15.98:0/180573667 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc664032c90 con 0x560fead0f6b0 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc65f7fe640 1 -- 172.21.15.98:0/180573667 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664031970 con 0x560fead0f6b0 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc682a3a740 1 -- 172.21.15.98:0/180573667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 msgr2=0x560feb0e2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc682a3a740 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb0e2c40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fc664001eb0 tx=0x7fc664005fd0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc682a3a740 1 -- 172.21.15.98:0/180573667 shutdown_connections 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc682a3a740 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb0e2c40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc682a3a740 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 0x560feb0cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc682a3a740 1 --2- 172.21.15.98:0/180573667 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 0x560feb0ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc682a3a740 1 -- 172.21.15.98:0/180573667 >> 172.21.15.98:0/180573667 conn(0x560fead244e0 msgr2=0x560feb0d6540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.253+0000 7fc682a3a740 1 -- 172.21.15.98:0/180573667 shutdown_connections 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 -- 172.21.15.98:0/180573667 wait complete. 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 -- start start 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 Processor -- start 2024-04-25T16:33:13.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 0x560feb0e0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 0x560feb0e0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb14f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741adf30 con 0x560fead0f6b0 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741ad860 con 0x560feaceef60 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc6741aaaa0 con 0x560fead074d0 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc682a3a740 10 client.0 _ll_register_callbacks cb 0x7fc664009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc67911f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb14f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc67911f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb14f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44672/0 (socket says 172.21.15.98:44672) 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.254+0000 7fc67911f640 1 -- 172.21.15.98:0/70922245 learned_addr learned my addr 172.21.15.98:0/70922245 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc679920640 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 0x560feb0e0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc67911f640 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 0x560feb0e0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc67911f640 1 -- 172.21.15.98:0/70922245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 msgr2=0x560feb0e0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc67911f640 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 0x560feb0e0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc67911f640 1 -- 172.21.15.98:0/70922245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 msgr2=0x560feb0e0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc67911f640 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 0x560feb0e0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc67911f640 1 -- 172.21.15.98:0/70922245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560feaff74b0 con 0x560fead0f6b0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc67911f640 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb14f450 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc6640059d0 tx=0x7fc664009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd16127793fb851 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc65d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc65d7fa640 1 -- 172.21.15.98:0/70922245 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc664035040 con 0x560fead0f6b0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc682a3a740 1 -- 172.21.15.98:0/70922245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc664032c90 con 0x560fead0f6b0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc65d7fa640 1 -- 172.21.15.98:0/70922245 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fc66404d110 con 0x560fead0f6b0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc65d7fa640 1 -- 172.21.15.98:0/70922245 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc66404d820 con 0x560fead0f6b0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc682a3a740 10 client.15864 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc682a3a740 1 -- 172.21.15.98:0/70922245 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560feb150890 con 0x560fead0f6b0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc65d7fa640 1 -- 172.21.15.98:0/70922245 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc664061070 con 0x560fead0f6b0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc65d7fa640 1 -- 172.21.15.98:0/70922245 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fc664047030 con 0x560fead0f6b0 2024-04-25T16:33:13.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc682a3a740 10 client.15864 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.255+0000 7fc682a3a740 10 client.15864 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.256+0000 7fc682a3a740 20 client.15864 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.256+0000 7fc682a3a740 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560fead30390 0x560feb153170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.256+0000 7fc682a3a740 4 client.15864 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.256+0000 7fc682a3a740 1 -- 172.21.15.98:0/70922245 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560feb03e800 con 0x560fead30390 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.256+0000 7fc679920640 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560fead30390 0x560feb153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.256+0000 7fc65d7fa640 10 client.15864 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.256+0000 7fc679920640 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560fead30390 0x560feb153170 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=156c4088165096f0 server_cookie=d989719eb6e2a497 in_seq=0 out_seq=0 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc65d7fa640 1 -- 172.21.15.98:0/70922245 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x560feb03e800 con 0x560fead30390 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc65d7fa640 10 client.15864 handle_command_reply: tid=0 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 client.15864 shutdown 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 -- 172.21.15.98:0/70922245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 msgr2=0x560feb14f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb14f450 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fc6640059d0 tx=0x7fc664009830 comp rx=0 tx=0).stop 2024-04-25T16:33:13.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 -- 172.21.15.98:0/70922245 shutdown_connections 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560fead30390 0x560feb153170 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560fead0f6b0 0x560feb14f450 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560fead074d0 0x560feb0e0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 --2- 172.21.15.98:0/70922245 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560feaceef60 0x560feb0e0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc65d7fa640 0 client.15864 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 -- 172.21.15.98:0/70922245 >> 172.21.15.98:0/70922245 conn(0x560fead244e0 msgr2=0x560feb0ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 -- 172.21.15.98:0/70922245 shutdown_connections 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 1 -- 172.21.15.98:0/70922245 wait complete. 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc682a3a740 20 client.15864 trim_cache size 0 max 16384 2024-04-25T16:33:13.264 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "d57a327a-2484-4875-a9d7-f2740f5af6a0": { 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "d57a327a-2484-4875-a9d7-f2740f5af6a0", 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:33:13.265 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:33:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 -- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc6600afb80 msgr2=0x7fc6600b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc6600afb80 0x7fc6600b2090 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc66c0a7470 tx=0x7fc66c0ca040 comp rx=0 tx=0).stop 2024-04-25T16:33:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 -- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674109a30 msgr2=0x7fc6741a1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674109a30 0x7fc6741a1620 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc66c00eb10 tx=0x7fc66c00c2d0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 -- 172.21.15.98:0/1416484188 shutdown_connections 2024-04-25T16:33:13.266 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc6600afb80 0x7fc6600b2090 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc674109a30 0x7fc6741a1620 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc674104d30 0x7fc67419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 --2- 172.21.15.98:0/1416484188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc674103af0 0x7fc67419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.257+0000 7fc67bb84640 1 -- 172.21.15.98:0/1416484188 >> 172.21.15.98:0/1416484188 conn(0x7fc6740fece0 msgr2=0x7fc674105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.258+0000 7fc67bb84640 1 -- 172.21.15.98:0/1416484188 shutdown_connections 2024-04-25T16:33:13.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:13.258+0000 7fc67bb84640 1 -- 172.21.15.98:0/1416484188 wait complete. 2024-04-25T16:33:13.268 INFO:tasks.cephfs.filesystem:scrub status for tag:d57a327a-2484-4875-a9d7-f2740f5af6a0 - {'path': '/', 'tag': 'd57a327a-2484-4875-a9d7-f2740f5af6a0', 'options': 'recursive,force'} 2024-04-25T16:33:14.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:13 smithi098 ceph-mon[29162]: scrub queued for path: / 2024-04-25T16:33:14.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:13 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:14.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:13 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:33:14.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:13 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:14.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:13 smithi173 ceph-mon[32556]: scrub queued for path: / 2024-04-25T16:33:14.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:13 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:14.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:13 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:33:14.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:13 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:14.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:13 smithi203 ceph-mon[33115]: scrub queued for path: / 2024-04-25T16:33:14.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:13 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:14.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:13 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:33:14.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:13 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:15.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:14 smithi098 ceph-mon[29162]: pgmap v378: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:15.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:14 smithi098 ceph-mon[29162]: scrub summary: idle 2024-04-25T16:33:15.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:14 smithi173 ceph-mon[32556]: pgmap v378: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:15.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:14 smithi173 ceph-mon[32556]: scrub summary: idle 2024-04-25T16:33:15.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:14 smithi203 ceph-mon[33115]: pgmap v378: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:15.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:14 smithi203 ceph-mon[33115]: scrub summary: idle 2024-04-25T16:33:16.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:16 smithi203 ceph-mon[33115]: pgmap v379: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:16.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:16 smithi098 ceph-mon[29162]: pgmap v379: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:16.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:16 smithi173 ceph-mon[32556]: pgmap v379: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:18.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:18 smithi203 ceph-mon[33115]: pgmap v380: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:33:18.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:18 smithi098 ceph-mon[29162]: pgmap v380: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:33:18.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:18 smithi173 ceph-mon[32556]: pgmap v380: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:33:20.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:20 smithi203 ceph-mon[33115]: pgmap v381: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T16:33:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:20 smithi098 ceph-mon[29162]: pgmap v381: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T16:33:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:20 smithi173 ceph-mon[32556]: pgmap v381: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T16:33:22.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:22 smithi203 ceph-mon[33115]: pgmap v382: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:33:22.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:22 smithi098 ceph-mon[29162]: pgmap v382: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:33:22.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:22 smithi173 ceph-mon[32556]: pgmap v382: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:33:24.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:24 smithi203 ceph-mon[33115]: pgmap v383: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T16:33:24.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:24 smithi098 ceph-mon[29162]: pgmap v383: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T16:33:24.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:24 smithi173 ceph-mon[32556]: pgmap v383: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-04-25T16:33:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:26 smithi203 ceph-mon[33115]: pgmap v384: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:26.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:26 smithi098 ceph-mon[29162]: pgmap v384: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:26.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:26 smithi173 ceph-mon[32556]: pgmap v384: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:28.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:28 smithi203 ceph-mon[33115]: pgmap v385: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T16:33:28.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:28 smithi098 ceph-mon[29162]: pgmap v385: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T16:33:28.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:28 smithi173 ceph-mon[32556]: pgmap v385: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T16:33:30.318 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:30 smithi098 ceph-mon[29162]: pgmap v386: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:33:30.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:30 smithi203 ceph-mon[33115]: pgmap v386: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:33:30.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:30 smithi173 ceph-mon[32556]: pgmap v386: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:33:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:32 smithi203 ceph-mon[33115]: pgmap v387: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:33:32.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:32 smithi098 ceph-mon[29162]: pgmap v387: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:33:32.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:32 smithi173 ceph-mon[32556]: pgmap v387: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-04-25T16:33:34.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:34 smithi203 ceph-mon[33115]: pgmap v388: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:34.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:34 smithi098 ceph-mon[29162]: pgmap v388: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:34.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:34 smithi173 ceph-mon[32556]: pgmap v388: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:33:36.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:36 smithi203 ceph-mon[33115]: pgmap v389: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:33:36.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:36 smithi098 ceph-mon[29162]: pgmap v389: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:33:36.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:36 smithi173 ceph-mon[32556]: pgmap v389: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:33:38.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:38 smithi203 ceph-mon[33115]: pgmap v390: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:33:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:38 smithi098 ceph-mon[29162]: pgmap v390: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:33:38.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:38 smithi173 ceph-mon[32556]: pgmap v390: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:33:40.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:40 smithi203 ceph-mon[33115]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T16:33:40.439 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:40 smithi203 ceph-mon[33115]: pgmap v391: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T16:33:40.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:40 smithi098 ceph-mon[29162]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T16:33:40.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:40 smithi098 ceph-mon[29162]: pgmap v391: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T16:33:40.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:40 smithi173 ceph-mon[32556]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-25T16:33:40.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:40 smithi173 ceph-mon[32556]: pgmap v391: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 170 B/s wr, 10 op/s 2024-04-25T16:33:42.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:42 smithi203 ceph-mon[33115]: pgmap v392: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T16:33:42.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:42 smithi098 ceph-mon[29162]: pgmap v392: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T16:33:42.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:42 smithi173 ceph-mon[32556]: pgmap v392: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T16:33:43.269 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:33:43.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:33:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 -- 172.21.15.98:0/1255387827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c108920 msgr2=0x7f716c10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 --2- 172.21.15.98:0/1255387827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c108920 0x7f716c10ad40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7164009ca0 tx=0x7f716402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 -- 172.21.15.98:0/1255387827 shutdown_connections 2024-04-25T16:33:43.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 --2- 172.21.15.98:0/1255387827 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c108920 0x7f716c10ad40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 --2- 172.21.15.98:0/1255387827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f716c103c20 0x7f716c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 --2- 172.21.15.98:0/1255387827 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f716c1029e0 0x7f716c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 -- 172.21.15.98:0/1255387827 >> 172.21.15.98:0/1255387827 conn(0x7f716c0fdbd0 msgr2=0x7f716c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:43.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 -- 172.21.15.98:0/1255387827 shutdown_connections 2024-04-25T16:33:43.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.442+0000 7f7171c9f640 1 -- 172.21.15.98:0/1255387827 wait complete. 2024-04-25T16:33:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f7171c9f640 1 Processor -- start 2024-04-25T16:33:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f7171c9f640 1 -- start start 2024-04-25T16:33:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f7171c9f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1029e0 0x7f716c199130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f7171c9f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f716c103c20 0x7f716c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f7171c9f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f716c108920 0x7f716c1a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1029e0 0x7f716c199130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1029e0 0x7f716c199130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42214/0 (socket says 172.21.15.98:42214) 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f716b7fe640 1 -- 172.21.15.98:0/500882675 learned_addr learned my addr 172.21.15.98:0/500882675 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f716c10e3c0 con 0x7f716c1029e0 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f716c10e240 con 0x7f716c103c20 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.443+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f716c10e540 con 0x7f716c108920 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716bfff640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f716c108920 0x7f716c1a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716affd640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f716c103c20 0x7f716c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716b7fe640 1 -- 172.21.15.98:0/500882675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f716c108920 msgr2=0x7f716c1a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716b7fe640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f716c108920 0x7f716c1a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716b7fe640 1 -- 172.21.15.98:0/500882675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f716c103c20 msgr2=0x7f716c199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716b7fe640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f716c103c20 0x7f716c199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716b7fe640 1 -- 172.21.15.98:0/500882675 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716c1a0c70 con 0x7f716c1029e0 2024-04-25T16:33:43.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716affd640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f716c103c20 0x7f716c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f716b7fe640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1029e0 0x7f716c199130 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f715c005fd0 tx=0x7f715c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.444+0000 7f7168ff9640 1 -- 172.21.15.98:0/500882675 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c017070 con 0x7f716c1029e0 2024-04-25T16:33:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.445+0000 7f7168ff9640 1 -- 172.21.15.98:0/500882675 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f715c009c30 con 0x7f716c1029e0 2024-04-25T16:33:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.445+0000 7f7168ff9640 1 -- 172.21.15.98:0/500882675 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c003c90 con 0x7f716c1029e0 2024-04-25T16:33:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.445+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716c1a0f60 con 0x7f716c1029e0 2024-04-25T16:33:43.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.445+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f716c1a1370 con 0x7f716c1029e0 2024-04-25T16:33:43.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.448+0000 7f7178c2e740 1 Processor -- start 2024-04-25T16:33:43.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.448+0000 7f7178c2e740 1 -- start start 2024-04-25T16:33:43.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.448+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.448+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070b7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.448+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06cf56b0 0x563c070b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.448+0000 7f7178c2e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563c0701cd80 con 0x563c06d16390 2024-04-25T16:33:43.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.448+0000 7f7178c2e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563c06fd68b0 con 0x563c06ced4d0 2024-04-25T16:33:43.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.448+0000 7f7178c2e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563c07018d40 con 0x563c06cf56b0 2024-04-25T16:33:43.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06cf56b0 0x563c070b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06cf56b0 0x563c070b7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:35404/0 (socket says 172.21.15.98:35404) 2024-04-25T16:33:43.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 -- 172.21.15.98:0/2462477936 learned_addr learned my addr 172.21.15.98:0/2462477936 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:43.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 -- 172.21.15.98:0/2462477936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 msgr2=0x563c070b7630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:33:43.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 --2- 172.21.15.98:0/2462477936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070b7630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 -- 172.21.15.98:0/2462477936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 msgr2=0x563c070b9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:43.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 --2- 172.21.15.98:0/2462477936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 -- 172.21.15.98:0/2462477936 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c06fa7a40 con 0x563c06cf56b0 2024-04-25T16:33:43.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716bfff640 1 --2- 172.21.15.98:0/2462477936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:43.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f716affd640 1 --2- 172.21.15.98:0/2462477936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06cf56b0 0x563c070b7d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7160009c50 tx=0x7f71600117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a47421b4f6ef41f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:43.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.449+0000 7f7168ff9640 1 -- 172.21.15.98:0/500882675 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f715c003e30 con 0x7f716c1029e0 2024-04-25T16:33:43.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.450+0000 7f7151ffb640 1 -- 172.21.15.98:0/2462477936 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160012450 con 0x563c06cf56b0 2024-04-25T16:33:43.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.450+0000 7f7151ffb640 1 -- 172.21.15.98:0/2462477936 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7160012a80 con 0x563c06cf56b0 2024-04-25T16:33:43.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.450+0000 7f7151ffb640 1 -- 172.21.15.98:0/2462477936 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716000faf0 con 0x563c06cf56b0 2024-04-25T16:33:43.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 -- 172.21.15.98:0/2462477936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06cf56b0 msgr2=0x563c070b7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 --2- 172.21.15.98:0/2462477936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06cf56b0 0x563c070b7d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7160009c50 tx=0x7f71600117b0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 -- 172.21.15.98:0/2462477936 shutdown_connections 2024-04-25T16:33:43.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 --2- 172.21.15.98:0/2462477936 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06cf56b0 0x563c070b7d80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 --2- 172.21.15.98:0/2462477936 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070b7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 --2- 172.21.15.98:0/2462477936 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 -- 172.21.15.98:0/2462477936 >> 172.21.15.98:0/2462477936 conn(0x563c06d0a4e0 msgr2=0x563c070b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7168ff9640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f71540afbb0 0x7f71540b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 -- 172.21.15.98:0/2462477936 shutdown_connections 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7178c2e740 1 -- 172.21.15.98:0/2462477936 wait complete. 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f716affd640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f71540afbb0 0x7f71540b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.451+0000 7f7168ff9640 1 -- 172.21.15.98:0/500882675 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f715c0dec60 con 0x7f716c1029e0 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.452+0000 7f7178c2e740 1 -- start start 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.452+0000 7f716affd640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f71540afbb0 0x7f71540b20c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f716001c040 tx=0x7f7160014040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.452+0000 7f7178c2e740 1 Processor -- start 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.452+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.452+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070c6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.452+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06d16390 0x563c070cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f7178c2e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563c070c3e00 con 0x563c06cf56b0 2024-04-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f7178c2e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563c06fa3d10 con 0x563c06d16390 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f7178c2e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563c070c3f80 con 0x563c06ced4d0 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f7178c2e740 10 client.0 _ll_register_callbacks cb 0x563c06fd0210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070c6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42238/0 (socket says 172.21.15.98:42238) 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 -- 172.21.15.98:0/4118405211 learned_addr learned my addr 172.21.15.98:0/4118405211 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716b7fe640 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716bfff640 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06d16390 0x563c070cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 -- 172.21.15.98:0/4118405211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 msgr2=0x563c070c5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 -- 172.21.15.98:0/4118405211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06d16390 msgr2=0x563c070cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06d16390 0x563c070cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 -- 172.21.15.98:0/4118405211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c06df87d0 con 0x563c06cf56b0 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716b7fe640 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f716affd640 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070c6220 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f716000a010 tx=0x7f71600117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3368b83372be6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:43.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f714affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:43.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.453+0000 7f714affd640 1 -- 172.21.15.98:0/4118405211 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716002c040 con 0x563c06cf56b0 2024-04-25T16:33:43.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.454+0000 7f7178c2e740 1 -- 172.21.15.98:0/4118405211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7160012a80 con 0x563c06cf56b0 2024-04-25T16:33:43.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.454+0000 7f714affd640 1 -- 172.21.15.98:0/4118405211 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f716000f810 con 0x563c06cf56b0 2024-04-25T16:33:43.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.454+0000 7f714affd640 1 -- 172.21.15.98:0/4118405211 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f716003e050 con 0x563c06cf56b0 2024-04-25T16:33:43.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.454+0000 7f7178c2e740 10 client.15876 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:43.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.454+0000 7f7178c2e740 1 -- 172.21.15.98:0/4118405211 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c06d2e3f0 con 0x563c06cf56b0 2024-04-25T16:33:43.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.454+0000 7f714affd640 1 -- 172.21.15.98:0/4118405211 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7160012a80 con 0x563c06cf56b0 2024-04-25T16:33:43.457 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.455+0000 7f714affd640 1 -- 172.21.15.98:0/4118405211 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f716002f030 con 0x563c06cf56b0 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.455+0000 7f7178c2e740 10 client.15876 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.455+0000 7f7178c2e740 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.455+0000 7f7178c2e740 20 client.15876 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.455+0000 7f7178c2e740 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x563c06cd4f60 0x563c070d3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.455+0000 7f7178c2e740 4 client.15876 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.455+0000 7f7178c2e740 1 -- 172.21.15.98:0/4118405211 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c0702a990 con 0x563c06cd4f60 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.455+0000 7f716b7fe640 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x563c06cd4f60 0x563c070d3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.456+0000 7f714affd640 10 client.15876 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.456+0000 7f716b7fe640 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x563c06cd4f60 0x563c070d3ac0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b808ddbdbe9051 server_cookie=17b099bfb7ba0b74 in_seq=0 out_seq=0 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f714affd640 1 -- 172.21.15.98:0/4118405211 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563c0702a990 con 0x563c06cd4f60 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f714affd640 10 client.15876 handle_command_reply: tid=0 2024-04-25T16:33:43.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 client.15876 shutdown 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 -- 172.21.15.98:0/4118405211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 msgr2=0x563c070c6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070c6220 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f716000a010 tx=0x7f71600117b0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 -- 172.21.15.98:0/4118405211 shutdown_connections 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x563c06cd4f60 0x563c070d3ac0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06d16390 0x563c070cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070c6220 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 --2- 172.21.15.98:0/4118405211 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f714affd640 0 client.15876 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 -- 172.21.15.98:0/4118405211 >> 172.21.15.98:0/4118405211 conn(0x563c06d0a4e0 msgr2=0x563c070b23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 -- 172.21.15.98:0/4118405211 shutdown_connections 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 1 -- 172.21.15.98:0/4118405211 wait complete. 2024-04-25T16:33:43.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.457+0000 7f7178c2e740 20 client.15876 trim_cache size 0 max 16384 2024-04-25T16:33:43.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f7178c2e740 1 Processor -- start 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f7178c2e740 1 -- start start 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06cf56b0 0x563c070c4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f7178c2e740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1aa5c0 con 0x563c06d16390 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f7178c2e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1ac100 con 0x563c06cf56b0 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f7178c2e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1aac90 con 0x563c06ced4d0 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f716bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f716affd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06cf56b0 0x563c070c4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f716bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42254/0 (socket says 172.21.15.98:42254) 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f716affd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06cf56b0 0x563c070c4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40074/0 (socket says 172.21.15.98:40074) 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f716bfff640 1 -- 172.21.15.98:0/3884549436 learned_addr learned my addr 172.21.15.98:0/3884549436 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.493+0000 7f716b7fe640 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f716b7fe640 1 -- 172.21.15.98:0/3884549436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06cf56b0 msgr2=0x563c070c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f716b7fe640 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06cf56b0 0x563c070c4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f716b7fe640 1 -- 172.21.15.98:0/3884549436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 msgr2=0x563c070b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f716b7fe640 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f716b7fe640 1 -- 172.21.15.98:0/3884549436 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c06cee0f0 con 0x563c06ced4d0 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f716bfff640 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f716b7fe640 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c42e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x563c070b8e60 tx=0x7f715c0de040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37a04b599ae550ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:43.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f71517fa640 1 -- 172.21.15.98:0/3884549436 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c0a6d90 con 0x563c06ced4d0 2024-04-25T16:33:43.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f71517fa640 1 -- 172.21.15.98:0/3884549436 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f715c0ab140 con 0x563c06ced4d0 2024-04-25T16:33:43.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.494+0000 7f71517fa640 1 -- 172.21.15.98:0/3884549436 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c004120 con 0x563c06ced4d0 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 -- 172.21.15.98:0/3884549436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 msgr2=0x563c070c42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c42e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x563c070b8e60 tx=0x7f715c0de040 comp rx=0 tx=0).stop 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 -- 172.21.15.98:0/3884549436 shutdown_connections 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06d16390 0x563c070b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06cf56b0 0x563c070c4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 --2- 172.21.15.98:0/3884549436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06ced4d0 0x563c070c42e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 -- 172.21.15.98:0/3884549436 >> 172.21.15.98:0/3884549436 conn(0x563c06d0a4e0 msgr2=0x563c070c03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 -- 172.21.15.98:0/3884549436 shutdown_connections 2024-04-25T16:33:43.501 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 -- 172.21.15.98:0/3884549436 wait complete. 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.496+0000 7f7178c2e740 1 -- start start 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.497+0000 7f7178c2e740 1 Processor -- start 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.497+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.497+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070bd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.497+0000 7f7178c2e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06d16390 0x563c070cd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.497+0000 7f716affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070bd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.497+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.497+0000 7f716affd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070bd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42260/0 (socket says 172.21.15.98:42260) 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716b7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070bcbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40080/0 (socket says 172.21.15.98:40080) 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716b7fe640 1 -- 172.21.15.98:0/1942396044 learned_addr learned my addr 172.21.15.98:0/1942396044 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1aba30 con 0x563c06cf56b0 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1ab360 con 0x563c06ced4d0 2024-04-25T16:33:43.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f716c1a85a0 con 0x563c06d16390 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716bfff640 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06d16390 0x563c070cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716affd640 1 -- 172.21.15.98:0/1942396044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06d16390 msgr2=0x563c070cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716affd640 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06d16390 0x563c070cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716affd640 1 -- 172.21.15.98:0/1942396044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 msgr2=0x563c070bcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716affd640 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070bcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716affd640 1 -- 172.21.15.98:0/1942396044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c06fdd4b0 con 0x563c06cf56b0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716b7fe640 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070bcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f7178c2e740 10 client.0 _ll_register_callbacks cb 0x7f7164009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f716affd640 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070bd100 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7160005940 tx=0x7f7160012b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5723335833748110 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f714affd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.498+0000 7f714affd640 1 -- 172.21.15.98:0/1942396044 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160004870 con 0x563c06cf56b0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f714affd640 1 -- 172.21.15.98:0/1942396044 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7160004b00 con 0x563c06cf56b0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f715c0ab140 con 0x563c06cf56b0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f714affd640 1 -- 172.21.15.98:0/1942396044 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7160054050 con 0x563c06cf56b0 2024-04-25T16:33:43.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f7178c2e740 10 client.15882 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=29}) -- 0x563c0713f090 con 0x563c06cf56b0 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f714affd640 1 -- 172.21.15.98:0/1942396044 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f716002f040 con 0x563c06cf56b0 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f7178c2e740 10 client.15882 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f7178c2e740 10 client.15882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f7178c2e740 20 client.15882 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f7178c2e740 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x563c06caa3a0 0x563c071418e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.499+0000 7f7178c2e740 4 client.15882 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.500+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f715c0ab140 con 0x563c06caa3a0 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.500+0000 7f716b7fe640 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x563c06caa3a0 0x563c071418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.500+0000 7f714affd640 10 client.15882 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.500+0000 7f716b7fe640 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x563c06caa3a0 0x563c071418e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebb2fac038107f67 server_cookie=a6819bf0f4461ea6 in_seq=0 out_seq=0 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f714affd640 1 -- 172.21.15.98:0/1942396044 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f715c0ab140 con 0x563c06caa3a0 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f714affd640 10 client.15882 handle_command_reply: tid=0 2024-04-25T16:33:43.504 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 client.15882 shutdown 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 msgr2=0x563c070bd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070bd100 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f7160005940 tx=0x7f7160012b30 comp rx=0 tx=0).stop 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 shutdown_connections 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x563c06caa3a0 0x563c071418e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x563c06d16390 0x563c070cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f714affd640 0 client.15882 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x563c06cf56b0 0x563c070bd100 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 --2- 172.21.15.98:0/1942396044 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x563c06ced4d0 0x563c070bcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 >> 172.21.15.98:0/1942396044 conn(0x563c06d0a4e0 msgr2=0x563c070b6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 shutdown_connections 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 1 -- 172.21.15.98:0/1942396044 wait complete. 2024-04-25T16:33:43.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.501+0000 7f7178c2e740 20 client.15882 trim_cache size 0 max 16384 2024-04-25T16:33:43.506 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:33:43.506 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-04-25T16:33:43.506 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-04-25T16:33:43.506 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:33:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f71540afbb0 msgr2=0x7f71540b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f71540afbb0 0x7f71540b20c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f716001c040 tx=0x7f7160014040 comp rx=0 tx=0).stop 2024-04-25T16:33:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1029e0 msgr2=0x7f716c199130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1029e0 0x7f716c199130 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f715c005fd0 tx=0x7f715c00dde0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 shutdown_connections 2024-04-25T16:33:43.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f71540afbb0 0x7f71540b20c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f716c108920 0x7f716c1a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f716c103c20 0x7f716c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.502+0000 7f7171c9f640 1 --2- 172.21.15.98:0/500882675 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f716c1029e0 0x7f716c199130 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.503+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 >> 172.21.15.98:0/500882675 conn(0x7f716c0fdbd0 msgr2=0x7f716c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.503+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 shutdown_connections 2024-04-25T16:33:43.508 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.503+0000 7f7171c9f640 1 -- 172.21.15.98:0/500882675 wait complete. 2024-04-25T16:33:43.517 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:33:43.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:33:43.630 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.631+0000 7fb8cdffb640 1 -- 172.21.15.98:0/2675277464 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8031800 con 0x7fb8d0101730 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 -- 172.21.15.98:0/2675277464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0101730 msgr2=0x7fb8d0101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/2675277464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0101730 0x7fb8d0101ba0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8009900 tx=0x7fb8b802a410 comp rx=0 tx=0).stop 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 -- 172.21.15.98:0/2675277464 shutdown_connections 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/2675277464 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8d0102230 0x7fb8d010bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/2675277464 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0101730 0x7fb8d0101ba0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/2675277464 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8d0105080 0x7fb8d0105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 -- 172.21.15.98:0/2675277464 >> 172.21.15.98:0/2675277464 conn(0x7fb8d00fd330 msgr2=0x7fb8d00ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 -- 172.21.15.98:0/2675277464 shutdown_connections 2024-04-25T16:33:43.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 -- 172.21.15.98:0/2675277464 wait complete. 2024-04-25T16:33:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.632+0000 7fb8d52f2640 1 Processor -- start 2024-04-25T16:33:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8d52f2640 1 -- start start 2024-04-25T16:33:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8d52f2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8d0101730 0x7fb8d01a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8d52f2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0102230 0x7fb8d01a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8d52f2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8d0105080 0x7fb8d01a7c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8d52f2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d0103ca0 con 0x7fb8d0102230 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8d52f2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d0103b20 con 0x7fb8d0101730 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8ce7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0102230 0x7fb8d01a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8d52f2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb8d0103e20 con 0x7fb8d0105080 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8ce7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0102230 0x7fb8d01a0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42286/0 (socket says 172.21.15.98:42286) 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8ce7fc640 1 -- 172.21.15.98:0/1220537142 learned_addr learned my addr 172.21.15.98:0/1220537142 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8ceffd640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8d0101730 0x7fb8d01a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.633+0000 7fb8cf7fe640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8d0105080 0x7fb8d01a7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8ceffd640 1 -- 172.21.15.98:0/1220537142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8d0105080 msgr2=0x7fb8d01a7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8ceffd640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8d0105080 0x7fb8d01a7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8ceffd640 1 -- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0102230 msgr2=0x7fb8d01a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8ceffd640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0102230 0x7fb8d01a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8ceffd640 1 -- 172.21.15.98:0/1220537142 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8d01a82e0 con 0x7fb8d0101730 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8cf7fe640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8d0105080 0x7fb8d01a7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:43.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8ce7fc640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0102230 0x7fb8d01a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:43.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8ceffd640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8d0101730 0x7fb8d01a07a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0069bf0 tx=0x7fb8c000dbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:43.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8c7fff640 1 -- 172.21.15.98:0/1220537142 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0017070 con 0x7fb8d0101730 2024-04-25T16:33:43.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8c7fff640 1 -- 172.21.15.98:0/1220537142 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb8c0009e30 con 0x7fb8d0101730 2024-04-25T16:33:43.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8d01a85d0 con 0x7fb8d0101730 2024-04-25T16:33:43.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8c7fff640 1 -- 172.21.15.98:0/1220537142 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c000f3c0 con 0x7fb8d0101730 2024-04-25T16:33:43.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.634+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8d01a8ad0 con 0x7fb8d0101730 2024-04-25T16:33:43.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.635+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8d0108300 con 0x7fb8d0101730 2024-04-25T16:33:43.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.638+0000 7fb8c7fff640 1 -- 172.21.15.98:0/1220537142 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fb8c001a070 con 0x7fb8d0101730 2024-04-25T16:33:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.638+0000 7fb8c7fff640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc0afaf0 0x7fb8bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.639+0000 7fb8c7fff640 1 -- 172.21.15.98:0/1220537142 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fb8c00de9b0 con 0x7fb8d0101730 2024-04-25T16:33:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.639+0000 7fb8ce7fc640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc0afaf0 0x7fb8bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:43.638 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.639+0000 7fb8ce7fc640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc0afaf0 0x7fb8bc0b2000 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8000c00 tx=0x7fb8b8031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:43.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.642+0000 7fb8c7fff640 1 -- 172.21.15.98:0/1220537142 <== mon.2 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8c00a7180 con 0x7fb8d0101730 2024-04-25T16:33:43.963 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.964+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8d002db50 con 0x7fb8d0101730 2024-04-25T16:33:43.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.965+0000 7fb8c7fff640 1 -- 172.21.15.98:0/1220537142 <== mon.2 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11935 (secure 0 0 0) 0x7fb8c00af070 con 0x7fb8d0101730 2024-04-25T16:33:43.965 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:33:43.965 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:32:32.844737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24494,24473,24457,24484,24539]},"id":1}]} 2024-04-25T16:33:43.965 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T16:33:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc0afaf0 msgr2=0x7fb8bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc0afaf0 0x7fb8bc0b2000 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8000c00 tx=0x7fb8b8031400 comp rx=0 tx=0).stop 2024-04-25T16:33:43.968 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8d0101730 msgr2=0x7fb8d01a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8d0101730 0x7fb8d01a07a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0069bf0 tx=0x7fb8c000dbb0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 shutdown_connections 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb8bc0afaf0 0x7fb8bc0b2000 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb8d0105080 0x7fb8d01a7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb8d0102230 0x7fb8d01a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 --2- 172.21.15.98:0/1220537142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8d0101730 0x7fb8d01a07a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 >> 172.21.15.98:0/1220537142 conn(0x7fb8d00fd330 msgr2=0x7fb8d00ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 shutdown_connections 2024-04-25T16:33:43.969 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:43.969+0000 7fb8d52f2640 1 -- 172.21.15.98:0/1220537142 wait complete. 2024-04-25T16:33:43.982 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:33:44.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 -- 172.21.15.98:0/2580991313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc1029e0 msgr2=0x7f43fc1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 --2- 172.21.15.98:0/2580991313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc1029e0 0x7f43fc1036e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f43ec009900 tx=0x7f43ec02a410 comp rx=0 tx=0).stop 2024-04-25T16:33:44.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 -- 172.21.15.98:0/2580991313 shutdown_connections 2024-04-25T16:33:44.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 --2- 172.21.15.98:0/2580991313 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43fc108920 0x7f43fc10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 --2- 172.21.15.98:0/2580991313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43fc103c20 0x7f43fc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 --2- 172.21.15.98:0/2580991313 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc1029e0 0x7f43fc1036e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 -- 172.21.15.98:0/2580991313 >> 172.21.15.98:0/2580991313 conn(0x7f43fc0fdbd0 msgr2=0x7f43fc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 -- 172.21.15.98:0/2580991313 shutdown_connections 2024-04-25T16:33:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 -- 172.21.15.98:0/2580991313 wait complete. 2024-04-25T16:33:44.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.100+0000 7f44013f3640 1 Processor -- start 2024-04-25T16:33:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f44013f3640 1 -- start start 2024-04-25T16:33:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f44013f3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43fc103c20 0x7f43fc198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f44013f3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43fc108920 0x7f43fc198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f44013f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc19d810 0x7f43fc19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f44013f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc10d290 con 0x7f43fc19d810 2024-04-25T16:33:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f44013f3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc10d110 con 0x7f43fc108920 2024-04-25T16:33:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f44013f3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc10d410 con 0x7f43fc103c20 2024-04-25T16:33:44.102 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc19d810 0x7f43fc19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc19d810 0x7f43fc19fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42314/0 (socket says 172.21.15.98:42314) 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fb7fe640 1 -- 172.21.15.98:0/2634903251 learned_addr learned my addr 172.21.15.98:0/2634903251 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43fc108920 0x7f43fc198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43faffd640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43fc103c20 0x7f43fc198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fb7fe640 1 -- 172.21.15.98:0/2634903251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43fc103c20 msgr2=0x7f43fc198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fb7fe640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43fc103c20 0x7f43fc198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fb7fe640 1 -- 172.21.15.98:0/2634903251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43fc108920 msgr2=0x7f43fc198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fb7fe640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43fc108920 0x7f43fc198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fb7fe640 1 -- 172.21.15.98:0/2634903251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc1a0360 con 0x7f43fc19d810 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.101+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43fc108920 0x7f43fc198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.102+0000 7f43fb7fe640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc19d810 0x7f43fc19fc30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f43f400eb10 tx=0x7f43f400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.102+0000 7f43e3fff640 1 -- 172.21.15.98:0/2634903251 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4004b20 con 0x7f43fc19d810 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.102+0000 7f43e3fff640 1 -- 172.21.15.98:0/2634903251 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43f4004cc0 con 0x7f43fc19d810 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.102+0000 7f44013f3640 1 -- 172.21.15.98:0/2634903251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43fc1a0650 con 0x7f43fc19d810 2024-04-25T16:33:44.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.102+0000 7f44013f3640 1 -- 172.21.15.98:0/2634903251 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43fc1a0bc0 con 0x7f43fc19d810 2024-04-25T16:33:44.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.103+0000 7f43e3fff640 1 -- 172.21.15.98:0/2634903251 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4010690 con 0x7f43fc19d810 2024-04-25T16:33:44.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.105+0000 7f43e3fff640 1 -- 172.21.15.98:0/2634903251 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f43f4010830 con 0x7f43fc19d810 2024-04-25T16:33:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.105+0000 7f440837b740 1 Processor -- start 2024-04-25T16:33:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f440837b740 1 -- start start 2024-04-25T16:33:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43e3fff640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43e40b3f80 0x7f43e40b6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f440837b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 0x5645ff0fa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f440837b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0fad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f440837b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed376b0 0x5645ff0fba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f440837b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5645ff05ed80 con 0x5645fed2f4d0 2024-04-25T16:33:44.106 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f440837b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5645ff018a20 con 0x5645fed376b0 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f440837b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5645ff05ad40 con 0x5645fed58390 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43e40b3f80 0x7f43e40b6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43e3fff640 1 -- 172.21.15.98:0/2634903251 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f43f40df460 con 0x7f43fc19d810 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0fad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0fad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42322/0 (socket says 172.21.15.98:42322) 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 -- 172.21.15.98:0/1908295631 learned_addr learned my addr 172.21.15.98:0/1908295631 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fb7fe640 1 --2- 172.21.15.98:0/1908295631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed376b0 0x5645ff0fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 -- 172.21.15.98:0/1908295631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 msgr2=0x5645ff0fa810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/1908295631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 0x5645ff0fa810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 -- 172.21.15.98:0/1908295631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed376b0 msgr2=0x5645ff0fba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/1908295631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed376b0 0x5645ff0fba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.106+0000 7f43fa7fc640 1 -- 172.21.15.98:0/1908295631 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645ff023c30 con 0x5645fed2f4d0 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.107+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/1908295631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0fad90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f43f0010f20 tx=0x7f43f000f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f028ad263e7e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.107 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.107+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43e40b3f80 0x7f43e40b6490 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f43f000fd50 tx=0x7f43f001e000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.107+0000 7f43e17fa640 1 -- 172.21.15.98:0/1908295631 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0019040 con 0x5645fed2f4d0 2024-04-25T16:33:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.107+0000 7f43e17fa640 1 -- 172.21.15.98:0/1908295631 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43f000eca0 con 0x5645fed2f4d0 2024-04-25T16:33:44.108 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.107+0000 7f43e17fa640 1 -- 172.21.15.98:0/1908295631 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f001ed60 con 0x5645fed2f4d0 2024-04-25T16:33:44.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 -- 172.21.15.98:0/1908295631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 msgr2=0x5645ff0fad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 --2- 172.21.15.98:0/1908295631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0fad90 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f43f0010f20 tx=0x7f43f000f850 comp rx=0 tx=0).stop 2024-04-25T16:33:44.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 -- 172.21.15.98:0/1908295631 shutdown_connections 2024-04-25T16:33:44.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 --2- 172.21.15.98:0/1908295631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed376b0 0x5645ff0fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 --2- 172.21.15.98:0/1908295631 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0fad90 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.111 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 --2- 172.21.15.98:0/1908295631 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 0x5645ff0fa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 -- 172.21.15.98:0/1908295631 >> 172.21.15.98:0/1908295631 conn(0x5645fed4c4e0 msgr2=0x5645ff0f6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 -- 172.21.15.98:0/1908295631 shutdown_connections 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.108+0000 7f440837b740 1 -- 172.21.15.98:0/1908295631 wait complete. 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f440837b740 1 -- start start 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f440837b740 1 Processor -- start 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f440837b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed2f4d0 0x5645ff104560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f440837b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed376b0 0x5645ff104aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f440837b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed58390 0x5645ff10ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f43fa7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed376b0 0x5645ff104aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f43fa7fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed376b0 0x5645ff104aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42338/0 (socket says 172.21.15.98:42338) 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f43fa7fc640 1 -- 172.21.15.98:0/1500397729 learned_addr learned my addr 172.21.15.98:0/1500397729 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5645ff0fc4d0 con 0x5645fed376b0 2024-04-25T16:33:44.112 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5645fefe5dc0 con 0x5645fed58390 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5645ff0fc650 con 0x5645fed2f4d0 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.109+0000 7f43faffd640 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed2f4d0 0x5645ff104560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43fb7fe640 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed58390 0x5645ff10ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f440837b740 10 client.0 _ll_register_callbacks cb 0x5645ff001600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43fa7fc640 1 -- 172.21.15.98:0/1500397729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed2f4d0 msgr2=0x5645ff104560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed2f4d0 0x5645ff104560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43fa7fc640 1 -- 172.21.15.98:0/1500397729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed58390 msgr2=0x5645ff10ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed58390 0x5645ff10ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43fa7fc640 1 -- 172.21.15.98:0/1500397729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645fed703f0 con 0x5645fed376b0 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43fb7fe640 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed58390 0x5645ff10ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed376b0 0x5645ff104aa0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009df0 tx=0x7f43f001ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=746f322c8ed3060a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43deff5640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:44.113 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43deff5640 1 -- 172.21.15.98:0/1500397729 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0035060 con 0x5645fed376b0 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43deff5640 1 -- 172.21.15.98:0/1500397729 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f43f0005040 con 0x5645fed376b0 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43f000eca0 con 0x5645fed376b0 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f43deff5640 1 -- 172.21.15.98:0/1500397729 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43f003c050 con 0x5645fed376b0 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f440837b740 10 client.15900 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.110+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5645fee3a7d0 con 0x5645fed376b0 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f43deff5640 1 -- 172.21.15.98:0/1500397729 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43f000ee50 con 0x5645fed376b0 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f43deff5640 1 -- 172.21.15.98:0/1500397729 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f43f0004ad0 con 0x5645fed376b0 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f440837b740 10 client.15900 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f440837b740 10 client.15900 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f440837b740 20 client.15900 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f440837b740 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x5645fed16f60 0x5645ff112040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f440837b740 4 client.15900 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:44.114 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f43f000eca0 con 0x5645fed16f60 2024-04-25T16:33:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.111+0000 7f43faffd640 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x5645fed16f60 0x5645ff112040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.112+0000 7f43deff5640 10 client.15900 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:33:44.115 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.112+0000 7f43faffd640 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x5645fed16f60 0x5645ff112040 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=62bb9b1f8a9da320 server_cookie=171fbb746c19268c in_seq=0 out_seq=0 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f43deff5640 1 -- 172.21.15.98:0/1500397729 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f43f000eca0 con 0x5645fed16f60 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f43deff5640 10 client.15900 handle_command_reply: tid=0 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 client.15900 shutdown 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed376b0 msgr2=0x5645ff104aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed376b0 0x5645ff104aa0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f43f0009df0 tx=0x7f43f001ede0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 shutdown_connections 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x5645fed16f60 0x5645ff112040 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed58390 0x5645ff10ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed376b0 0x5645ff104aa0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.116 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f43deff5640 0 client.15900 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:33:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 --2- 172.21.15.98:0/1500397729 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed2f4d0 0x5645ff104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 >> 172.21.15.98:0/1500397729 conn(0x5645fed4c4e0 msgr2=0x5645ff0f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 shutdown_connections 2024-04-25T16:33:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 1 -- 172.21.15.98:0/1500397729 wait complete. 2024-04-25T16:33:44.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.113+0000 7f440837b740 20 client.15900 trim_cache size 0 max 16384 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f440837b740 1 Processor -- start 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f440837b740 1 -- start start 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f440837b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0ff2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f440837b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 0x5645ff10b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f440837b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fecec3a0 0x5645ff0ff830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f440837b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43e800fc50 con 0x5645fed2f4d0 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f440837b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f43e8011790 con 0x5645fecec3a0 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f440837b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f43e8010320 con 0x5645fed58390 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43faffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0ff2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43faffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0ff2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42340/0 (socket says 172.21.15.98:42340) 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43faffd640 1 -- 172.21.15.98:0/1744617397 learned_addr learned my addr 172.21.15.98:0/1744617397 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43fb7fe640 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fecec3a0 0x5645ff0ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 0x5645ff10b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43faffd640 1 -- 172.21.15.98:0/1744617397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 msgr2=0x5645ff10b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43faffd640 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 0x5645ff10b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43faffd640 1 -- 172.21.15.98:0/1744617397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fecec3a0 msgr2=0x5645ff0ff830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43faffd640 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fecec3a0 0x5645ff0ff830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.127+0000 7f43faffd640 1 -- 172.21.15.98:0/1744617397 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645ff105160 con 0x5645fed2f4d0 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.128+0000 7f43faffd640 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0ff2f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0057c0 tx=0x7f43ec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5167a0618f3c2cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.128+0000 7f43e0ff9640 1 -- 172.21.15.98:0/1744617397 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec035040 con 0x5645fed2f4d0 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.128+0000 7f43e0ff9640 1 -- 172.21.15.98:0/1744617397 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43ec004040 con 0x5645fed2f4d0 2024-04-25T16:33:44.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.128+0000 7f43e0ff9640 1 -- 172.21.15.98:0/1744617397 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec045750 con 0x5645fed2f4d0 2024-04-25T16:33:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- 172.21.15.98:0/1744617397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 msgr2=0x5645ff0ff2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0ff2f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0057c0 tx=0x7f43ec000bc0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- 172.21.15.98:0/1744617397 shutdown_connections 2024-04-25T16:33:44.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fecec3a0 0x5645ff0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fed58390 0x5645ff10b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 --2- 172.21.15.98:0/1744617397 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed2f4d0 0x5645ff0ff2f0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- 172.21.15.98:0/1744617397 >> 172.21.15.98:0/1744617397 conn(0x5645fed4c4e0 msgr2=0x5645ff0fa790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- 172.21.15.98:0/1744617397 shutdown_connections 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- 172.21.15.98:0/1744617397 wait complete. 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- start start 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 Processor -- start 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fecec3a0 0x5645ff10ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed2f4d0 0x5645ff10b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed58390 0x5645ff178590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f43e80110c0 con 0x5645fed58390 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f43e80109f0 con 0x5645fed2f4d0 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f43e800dc30 con 0x5645fecec3a0 2024-04-25T16:33:44.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.129+0000 7f440837b740 10 client.0 _ll_register_callbacks cb 0x7f43f0010cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fb7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed58390 0x5645ff178590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed2f4d0 0x5645ff10b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed2f4d0 0x5645ff10b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40158/0 (socket says 172.21.15.98:40158) 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43faffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fecec3a0 0x5645ff10ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 -- 172.21.15.98:0/2363153783 learned_addr learned my addr 172.21.15.98:0/2363153783 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 -- 172.21.15.98:0/2363153783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fecec3a0 msgr2=0x5645ff10ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fecec3a0 0x5645ff10ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 -- 172.21.15.98:0/2363153783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed58390 msgr2=0x5645ff178590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed58390 0x5645ff178590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 -- 172.21.15.98:0/2363153783 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645ff178b80 con 0x5645fed2f4d0 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fb7fe640 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed58390 0x5645ff178590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43fa7fc640 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed2f4d0 0x5645ff10b290 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f43f00304f0 tx=0x7f43f000ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df05af15604011f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43deff5640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:33:44.135 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43deff5640 1 -- 172.21.15.98:0/2363153783 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0044070 con 0x5645fed2f4d0 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f440837b740 1 -- 172.21.15.98:0/2363153783 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43ec004040 con 0x5645fed2f4d0 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.130+0000 7f43deff5640 1 -- 172.21.15.98:0/2363153783 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f43f001e620 con 0x5645fed2f4d0 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f43deff5640 1 -- 172.21.15.98:0/2363153783 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43f0050050 con 0x5645fed2f4d0 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f440837b740 10 client.25433 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f440837b740 1 -- 172.21.15.98:0/2363153783 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5645ff179e00 con 0x5645fed2f4d0 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f43deff5640 1 -- 172.21.15.98:0/2363153783 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f43ec004040 con 0x5645fed2f4d0 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f43deff5640 1 -- 172.21.15.98:0/2363153783 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f43f0005040 con 0x5645fed2f4d0 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f440837b740 10 client.25433 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f440837b740 10 client.25433 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f440837b740 20 client.25433 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f440837b740 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x5645ff17a750 0x5645ff17cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f440837b740 4 client.25433 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f440837b740 1 -- 172.21.15.98:0/2363153783 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5645ff06c990 con 0x5645ff17a750 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.131+0000 7f43faffd640 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x5645ff17a750 0x5645ff17cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.136 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f43faffd640 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x5645ff17a750 0x5645ff17cb70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1c670a8ef030991f server_cookie=aed4ab475d21d9b1 in_seq=0 out_seq=0 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f43deff5640 10 client.25433 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f43deff5640 1 -- 172.21.15.98:0/2363153783 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5645ff06c990 con 0x5645ff17a750 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f43deff5640 10 client.25433 handle_command_reply: tid=0 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 client.25433 shutdown 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 -- 172.21.15.98:0/2363153783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed2f4d0 msgr2=0x5645ff10b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed2f4d0 0x5645ff10b290 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f43f00304f0 tx=0x7f43f000ef00 comp rx=0 tx=0).stop 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 -- 172.21.15.98:0/2363153783 shutdown_connections 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x5645ff17a750 0x5645ff17cb70 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5645fed58390 0x5645ff178590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5645fed2f4d0 0x5645ff10b290 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 --2- 172.21.15.98:0/2363153783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5645fecec3a0 0x5645ff10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f440837b740 1 -- 172.21.15.98:0/2363153783 >> 172.21.15.98:0/2363153783 conn(0x5645fed4c4e0 msgr2=0x5645ff1044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.132+0000 7f43deff5640 0 client.25433 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:33:44.137 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f440837b740 1 -- 172.21.15.98:0/2363153783 shutdown_connections 2024-04-25T16:33:44.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f440837b740 1 -- 172.21.15.98:0/2363153783 wait complete. 2024-04-25T16:33:44.138 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f440837b740 20 client.25433 trim_cache size 0 max 16384 2024-04-25T16:33:44.138 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 -- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43e40b3f80 msgr2=0x7f43e40b6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43e40b3f80 0x7f43e40b6490 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f43f000fd50 tx=0x7f43f001e000 comp rx=0 tx=0).stop 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 -- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc19d810 msgr2=0x7f43fc19fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc19d810 0x7f43fc19fc30 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f43f400eb10 tx=0x7f43f400c2d0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 -- 172.21.15.98:0/2634903251 shutdown_connections 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43e40b3f80 0x7f43e40b6490 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f43fc19d810 0x7f43fc19fc30 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f43fc108920 0x7f43fc198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 --2- 172.21.15.98:0/2634903251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43fc103c20 0x7f43fc198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 -- 172.21.15.98:0/2634903251 >> 172.21.15.98:0/2634903251 conn(0x7f43fc0fdbd0 msgr2=0x7f43fc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 -- 172.21.15.98:0/2634903251 shutdown_connections 2024-04-25T16:33:44.139 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.133+0000 7f44013f3640 1 -- 172.21.15.98:0/2634903251 wait complete. 2024-04-25T16:33:44.143 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:33:44.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:44 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1220537142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:33:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2713767898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68102240 msgr2=0x7f7a6810bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2713767898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68102240 0x7f7a6810bca0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7a60009ca0 tx=0x7f7a6002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2713767898 shutdown_connections 2024-04-25T16:33:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2713767898 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68102240 0x7f7a6810bca0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2713767898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a68101740 0x7f7a68101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2713767898 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a68105090 0x7f7a681054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2713767898 >> 172.21.15.98:0/2713767898 conn(0x7f7a680fd340 msgr2=0x7f7a680ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2713767898 shutdown_connections 2024-04-25T16:33:44.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.256+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2713767898 wait complete. 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6edb2640 1 Processor -- start 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6edb2640 1 -- start start 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6edb2640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68101740 0x7f7a6819a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6edb2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a68102240 0x7f7a6819a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6edb2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a68105090 0x7f7a681a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6edb2640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6810f500 con 0x7f7a68101740 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6edb2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6810f380 con 0x7f7a68102240 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6edb2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a6810f680 con 0x7f7a68105090 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6cb4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68101740 0x7f7a6819a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6cb4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68101740 0x7f7a6819a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42368/0 (socket says 172.21.15.98:42368) 2024-04-25T16:33:44.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/2952306514 learned_addr learned my addr 172.21.15.98:0/2952306514 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a67fff640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a68102240 0x7f7a6819a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.257+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a68105090 0x7f7a681a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/2952306514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a68105090 msgr2=0x7f7a681a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a68105090 0x7f7a681a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/2952306514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a68102240 msgr2=0x7f7a6819a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a68102240 0x7f7a6819a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/2952306514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a681a1c20 con 0x7f7a68101740 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68101740 0x7f7a6819a0e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00e9f0 tx=0x7f7a5c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a65ffb640 1 -- 172.21.15.98:0/2952306514 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c00ccc0 con 0x7f7a68101740 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2952306514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a681a1f10 con 0x7f7a68101740 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a65ffb640 1 -- 172.21.15.98:0/2952306514 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7a5c0044b0 con 0x7f7a68101740 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.258+0000 7f7a65ffb640 1 -- 172.21.15.98:0/2952306514 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c010430 con 0x7f7a68101740 2024-04-25T16:33:44.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.259+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2952306514 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a681a23d0 con 0x7f7a68101740 2024-04-25T16:33:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a75c63740 1 Processor -- start 2024-04-25T16:33:44.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a75c63740 1 -- start start 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a65ffb640 1 -- 172.21.15.98:0/2952306514 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f7a5c0105d0 con 0x7f7a68101740 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 0x56546b81ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b81f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b81fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a75c63740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56546b77ed80 con 0x56546b44f4d0 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a75c63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56546b738a20 con 0x56546b478390 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a75c63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56546b77ad40 con 0x56546b4576b0 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.262+0000 7f7a6cb4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b81f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6cb4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b81f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42372/0 (socket says 172.21.15.98:42372) 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/2395017924 learned_addr learned my addr 172.21.15.98:0/2395017924 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a67fff640 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b81fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 0x56546b81ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/2395017924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 msgr2=0x56546b81fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b81fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.263 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/2395017924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 msgr2=0x56546b81ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 0x56546b81ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/2395017924 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56546b743c30 con 0x56546b44f4d0 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b81f1a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0e72f0 tx=0x7f7a5c0e77b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f32f4bb582f9de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.263+0000 7f7a65ffb640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a580afb70 0x7f7a580b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.264+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a580afb70 0x7f7a580b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.264+0000 7f7a52ffd640 1 -- 172.21.15.98:0/2395017924 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c0f2070 con 0x56546b44f4d0 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.264+0000 7f7a52ffd640 1 -- 172.21.15.98:0/2395017924 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7a5c0a6eb0 con 0x56546b44f4d0 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.264+0000 7f7a52ffd640 1 -- 172.21.15.98:0/2395017924 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c0a6b10 con 0x56546b44f4d0 2024-04-25T16:33:44.264 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.264+0000 7f7a65ffb640 1 -- 172.21.15.98:0/2952306514 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7a5c0df270 con 0x7f7a68101740 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 -- 172.21.15.98:0/2395017924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 msgr2=0x56546b81f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a580afb70 0x7f7a580b2080 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x56546b81fc70 tx=0x7f7a60031000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b81f1a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c0e72f0 tx=0x7f7a5c0e77b0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 -- 172.21.15.98:0/2395017924 shutdown_connections 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b81fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b81f1a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 --2- 172.21.15.98:0/2395017924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 0x56546b81ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 -- 172.21.15.98:0/2395017924 >> 172.21.15.98:0/2395017924 conn(0x56546b46c4e0 msgr2=0x56546b816020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 -- 172.21.15.98:0/2395017924 shutdown_connections 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.265+0000 7f7a75c63740 1 -- 172.21.15.98:0/2395017924 wait complete. 2024-04-25T16:33:44.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 1 -- start start 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 1 Processor -- start 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b8289f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b828f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 0x56546b82fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56546b820940 con 0x56546b44f4d0 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56546b705dc0 con 0x56546b478390 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56546b820ac0 con 0x56546b4576b0 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a75c63740 10 client.0 _ll_register_callbacks cb 0x56546b721600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.266+0000 7f7a6cb4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b8289f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a67fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b828f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a6cb4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b8289f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42374/0 (socket says 172.21.15.98:42374) 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/3946349848 learned_addr learned my addr 172.21.15.98:0/3946349848 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 0x56546b82fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a67fff640 1 -- 172.21.15.98:0/3946349848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 msgr2=0x56546b82fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a67fff640 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 0x56546b82fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a67fff640 1 -- 172.21.15.98:0/3946349848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 msgr2=0x56546b8289f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a67fff640 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b8289f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a67fff640 1 -- 172.21.15.98:0/3946349848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56546b4903f0 con 0x56546b4576b0 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b8289f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a67fff640 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b828f30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7a6819b180 tx=0x7f7a54012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47c93c8cba28b658 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a47fff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a47fff640 1 -- 172.21.15.98:0/3946349848 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5400f8c0 con 0x56546b4576b0 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a47fff640 1 -- 172.21.15.98:0/3946349848 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7a5400fa60 con 0x56546b4576b0 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.267+0000 7f7a75c63740 1 -- 172.21.15.98:0/3946349848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a5c0a6eb0 con 0x56546b4576b0 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a47fff640 1 -- 172.21.15.98:0/3946349848 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7a54025050 con 0x56546b4576b0 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a75c63740 10 client.25381 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a75c63740 1 -- 172.21.15.98:0/3946349848 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56546b55a7d0 con 0x56546b4576b0 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a47fff640 1 -- 172.21.15.98:0/3946349848 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7a5c0a6eb0 con 0x56546b4576b0 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a47fff640 1 -- 172.21.15.98:0/3946349848 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7a54010430 con 0x56546b4576b0 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a75c63740 10 client.25381 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a75c63740 10 client.25381 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a75c63740 20 client.25381 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.268+0000 7f7a75c63740 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x56546b436f60 0x56546b8367c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.269+0000 7f7a75c63740 4 client.25381 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.269+0000 7f7a75c63740 1 -- 172.21.15.98:0/3946349848 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56546b78c990 con 0x56546b436f60 2024-04-25T16:33:44.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.269+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x56546b436f60 0x56546b8367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.269+0000 7f7a47fff640 10 client.25381 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:33:44.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.269+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x56546b436f60 0x56546b8367c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=714a70c4573ae2c1 server_cookie=96bd82b58cad0f21 in_seq=0 out_seq=0 2024-04-25T16:33:44.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.270+0000 7f7a47fff640 1 -- 172.21.15.98:0/3946349848 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56546b78c990 con 0x56546b436f60 2024-04-25T16:33:44.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.270+0000 7f7a47fff640 10 client.25381 handle_command_reply: tid=0 2024-04-25T16:33:44.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.270+0000 7f7a75c63740 1 client.25381 shutdown 2024-04-25T16:33:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 -- 172.21.15.98:0/3946349848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 msgr2=0x56546b828f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b828f30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7a6819b180 tx=0x7f7a54012b60 comp rx=0 tx=0).stop 2024-04-25T16:33:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 -- 172.21.15.98:0/3946349848 shutdown_connections 2024-04-25T16:33:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x56546b436f60 0x56546b8367c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b478390 0x56546b82fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b828f30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a47fff640 0 client.25381 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:33:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 --2- 172.21.15.98:0/3946349848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b8289f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 -- 172.21.15.98:0/3946349848 >> 172.21.15.98:0/3946349848 conn(0x56546b46c4e0 msgr2=0x56546b818f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 -- 172.21.15.98:0/3946349848 shutdown_connections 2024-04-25T16:33:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 1 -- 172.21.15.98:0/3946349848 wait complete. 2024-04-25T16:33:44.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.274+0000 7f7a75c63740 20 client.25381 trim_cache size 0 max 16384 2024-04-25T16:33:44.301 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a75c63740 1 Processor -- start 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a75c63740 1 -- start start 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b436f60 0x56546b815ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b82fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b4576b0 0x56546b818e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a75c63740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a681aa850 con 0x56546b44f4d0 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a75c63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7a681ac390 con 0x56546b4576b0 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a75c63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a681aaf20 con 0x56546b436f60 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a67fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b82fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a67fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b82fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42388/0 (socket says 172.21.15.98:42388) 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.301+0000 7f7a67fff640 1 -- 172.21.15.98:0/746288847 learned_addr learned my addr 172.21.15.98:0/746288847 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b436f60 0x56546b815ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.302 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b4576b0 0x56546b818e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a67fff640 1 -- 172.21.15.98:0/746288847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b436f60 msgr2=0x56546b815ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a67fff640 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b436f60 0x56546b815ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a67fff640 1 -- 172.21.15.98:0/746288847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b4576b0 msgr2=0x56546b818e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a67fff640 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b4576b0 0x56546b818e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a67fff640 1 -- 172.21.15.98:0/746288847 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56546b8302f0 con 0x56546b44f4d0 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b436f60 0x56546b815ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b4576b0 0x56546b818e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a67fff640 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b82fdb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7a540109c0 tx=0x7f7a54009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c836841fc7e4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a527fc640 1 -- 172.21.15.98:0/746288847 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54010570 con 0x56546b44f4d0 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a527fc640 1 -- 172.21.15.98:0/746288847 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7a54010a40 con 0x56546b44f4d0 2024-04-25T16:33:44.303 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.302+0000 7f7a527fc640 1 -- 172.21.15.98:0/746288847 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5400fc90 con 0x56546b44f4d0 2024-04-25T16:33:44.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 -- 172.21.15.98:0/746288847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 msgr2=0x56546b82fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b82fdb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f7a540109c0 tx=0x7f7a54009f00 comp rx=0 tx=0).stop 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 -- 172.21.15.98:0/746288847 shutdown_connections 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b4576b0 0x56546b818e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b44f4d0 0x56546b82fdb0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 --2- 172.21.15.98:0/746288847 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b436f60 0x56546b815ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 -- 172.21.15.98:0/746288847 >> 172.21.15.98:0/746288847 conn(0x56546b46c4e0 msgr2=0x56546b8157c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 -- 172.21.15.98:0/746288847 shutdown_connections 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 -- 172.21.15.98:0/746288847 wait complete. 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 -- start start 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.304+0000 7f7a75c63740 1 Processor -- start 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b436f60 0x56546b82cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b44f4d0 0x56546b82d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a75c63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b89cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a75c63740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a681abcc0 con 0x56546b436f60 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a6cb4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b436f60 0x56546b82cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a6cb4e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b436f60 0x56546b82cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42400/0 (socket says 172.21.15.98:42400) 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a6cb4e640 1 -- 172.21.15.98:0/4149774939 learned_addr learned my addr 172.21.15.98:0/4149774939 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a75c63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7a681ab5f0 con 0x56546b44f4d0 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a67fff640 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b44f4d0 0x56546b82d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a681a8830 con 0x56546b4576b0 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.305+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b89cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a75c63740 10 client.0 _ll_register_callbacks cb 0x7f7a54009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a6d34f640 1 -- 172.21.15.98:0/4149774939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b44f4d0 msgr2=0x56546b82d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b44f4d0 0x56546b82d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a6d34f640 1 -- 172.21.15.98:0/4149774939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b436f60 msgr2=0x56546b82cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b436f60 0x56546b82cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a6d34f640 1 -- 172.21.15.98:0/4149774939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56546b82ff80 con 0x56546b4576b0 2024-04-25T16:33:44.309 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b436f60 0x56546b82cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a6d34f640 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b89cb20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a60031920 tx=0x7f7a60041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eb107d77630f285 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a47fff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.306+0000 7f7a47fff640 1 -- 172.21.15.98:0/4149774939 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a60059060 con 0x56546b4576b0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.307+0000 7f7a47fff640 1 -- 172.21.15.98:0/4149774939 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7a60030a10 con 0x56546b4576b0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.307+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a54010a40 con 0x56546b4576b0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.307+0000 7f7a47fff640 1 -- 172.21.15.98:0/4149774939 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7a60060050 con 0x56546b4576b0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.307+0000 7f7a75c63740 10 client.25387 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.307+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56546b89dfa0 con 0x56546b4576b0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.307+0000 7f7a47fff640 1 -- 172.21.15.98:0/4149774939 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7a54010a40 con 0x56546b4576b0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.308+0000 7f7a47fff640 1 -- 172.21.15.98:0/4149774939 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f7a60049420 con 0x56546b4576b0 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.308+0000 7f7a75c63740 10 client.25387 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.308+0000 7f7a75c63740 10 client.25387 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T16:33:44.310 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.308+0000 7f7a75c63740 20 client.25387 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.308+0000 7f7a75c63740 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x56546b478390 0x56546b8a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.308+0000 7f7a75c63740 4 client.25387 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:44.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.308+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56546b786800 con 0x56546b478390 2024-04-25T16:33:44.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.308+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x56546b478390 0x56546b8a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a47fff640 10 client.25387 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:33:44.311 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a6cb4e640 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x56546b478390 0x56546b8a09c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=450a4dffbae597c5 server_cookie=1e9d8b5af7da3a5c in_seq=0 out_seq=0 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a47fff640 1 -- 172.21.15.98:0/4149774939 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56546b786800 con 0x56546b478390 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a47fff640 10 client.25387 handle_command_reply: tid=0 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a75c63740 1 client.25387 shutdown 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 msgr2=0x56546b89cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a75c63740 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b89cb20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7a60031920 tx=0x7f7a60041000 comp rx=0 tx=0).stop 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 shutdown_connections 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.309+0000 7f7a75c63740 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x56546b478390 0x56546b8a09c0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.310+0000 7f7a75c63740 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56546b4576b0 0x56546b89cb20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.310+0000 7f7a47fff640 0 client.25387 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.310+0000 7f7a75c63740 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56546b44f4d0 0x56546b82d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.310+0000 7f7a75c63740 1 --2- 172.21.15.98:0/4149774939 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56546b436f60 0x56546b82cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.310+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 >> 172.21.15.98:0/4149774939 conn(0x56546b46c4e0 msgr2=0x56546b815760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.313 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.310+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 shutdown_connections 2024-04-25T16:33:44.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.310+0000 7f7a75c63740 1 -- 172.21.15.98:0/4149774939 wait complete. 2024-04-25T16:33:44.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.310+0000 7f7a75c63740 20 client.25387 trim_cache size 0 max 16384 2024-04-25T16:33:44.314 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:44.314 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a580afb70 msgr2=0x7f7a580b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a580afb70 0x7f7a580b2080 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x56546b81fc70 tx=0x7f7a60031000 comp rx=0 tx=0).stop 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68101740 msgr2=0x7f7a6819a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68101740 0x7f7a6819a0e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00e9f0 tx=0x7f7a5c00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2952306514 shutdown_connections 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a580afb70 0x7f7a580b2080 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a68105090 0x7f7a681a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a68102240 0x7f7a6819a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 --2- 172.21.15.98:0/2952306514 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a68101740 0x7f7a6819a0e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2952306514 >> 172.21.15.98:0/2952306514 conn(0x7f7a680fd340 msgr2=0x7f7a68107c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2952306514 shutdown_connections 2024-04-25T16:33:44.315 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.311+0000 7f7a6edb2640 1 -- 172.21.15.98:0/2952306514 wait complete. 2024-04-25T16:33:44.327 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:33:44.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:44 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1220537142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:33:44.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.502+0000 7f64b8bfe640 1 -- 172.21.15.98:0/3073736135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b41043c0 msgr2=0x7f64b4102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.502+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/3073736135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b41043c0 0x7f64b4102dc0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009900 tx=0x7f64a802a410 comp rx=0 tx=0).stop 2024-04-25T16:33:44.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.503+0000 7f64b8bfe640 1 -- 172.21.15.98:0/3073736135 shutdown_connections 2024-04-25T16:33:44.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.503+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/3073736135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b4108a10 0x7f64b410ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.503+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/3073736135 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b41043c0 0x7f64b4102dc0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.502 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.503+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/3073736135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64b4103ee0 0x7f64b41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.503+0000 7f64b8bfe640 1 -- 172.21.15.98:0/3073736135 >> 172.21.15.98:0/3073736135 conn(0x7f64b40fd340 msgr2=0x7f64b40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.503+0000 7f64b8bfe640 1 -- 172.21.15.98:0/3073736135 shutdown_connections 2024-04-25T16:33:44.503 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.503+0000 7f64b8bfe640 1 -- 172.21.15.98:0/3073736135 wait complete. 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.504+0000 7f64b8bfe640 1 Processor -- start 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.504+0000 7f64b8bfe640 1 -- start start 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b8bfe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b4103ee0 0x7f64b4199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b8bfe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b4108a10 0x7f64b419a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b8bfe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64b419f1d0 0x7f64b41a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b8bfe640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64b410f4d0 con 0x7f64b4103ee0 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b4103ee0 0x7f64b4199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b4103ee0 0x7f64b4199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42412/0 (socket says 172.21.15.98:42412) 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f64b410f350 con 0x7f64b419f1d0 2024-04-25T16:33:44.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b259c640 1 -- 172.21.15.98:0/2877529088 learned_addr learned my addr 172.21.15.98:0/2877529088 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64b410f650 con 0x7f64b4108a10 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64b419f1d0 0x7f64b41a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b1d9b640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b4108a10 0x7f64b419a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b2d9d640 1 -- 172.21.15.98:0/2877529088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b4108a10 msgr2=0x7f64b419a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b4108a10 0x7f64b419a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b2d9d640 1 -- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b4103ee0 msgr2=0x7f64b4199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b4103ee0 0x7f64b4199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b2d9d640 1 -- 172.21.15.98:0/2877529088 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b41a1c60 con 0x7f64b419f1d0 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.505+0000 7f64b259c640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b4103ee0 0x7f64b4199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.506+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64b419f1d0 0x7f64b41a15f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009fd0 tx=0x7f64ac00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.506+0000 7f64a37fe640 1 -- 172.21.15.98:0/2877529088 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00ce20 con 0x7f64b419f1d0 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.506+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b41a1f50 con 0x7f64b419f1d0 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.506+0000 7f64a37fe640 1 -- 172.21.15.98:0/2877529088 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64ac005020 con 0x7f64b419f1d0 2024-04-25T16:33:44.506 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.506+0000 7f64a37fe640 1 -- 172.21.15.98:0/2877529088 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac010690 con 0x7f64b419f1d0 2024-04-25T16:33:44.507 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.507+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b41a2410 con 0x7f64b419f1d0 2024-04-25T16:33:44.509 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.510+0000 7f64a37fe640 1 -- 172.21.15.98:0/2877529088 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f64ac004a40 con 0x7f64b419f1d0 2024-04-25T16:33:44.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.511+0000 7f64a37fe640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f649c0afba0 0x7f649c0b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.511+0000 7f64bfb83740 1 Processor -- start 2024-04-25T16:33:44.510 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.511+0000 7f64a37fe640 1 -- 172.21.15.98:0/2877529088 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f64ac0df290 con 0x7f64b419f1d0 2024-04-25T16:33:44.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.511+0000 7f64bfb83740 1 -- start start 2024-04-25T16:33:44.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.511+0000 7f64b259c640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f649c0afba0 0x7f649c0b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.511+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a75390 0x560e25e17740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.511 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e17ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e18960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64bfb83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e25d7bd80 con 0x560e25a4c4d0 2024-04-25T16:33:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64bfb83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560e25d35a20 con 0x560e25a546b0 2024-04-25T16:33:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64bfb83740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560e25d77d40 con 0x560e25a75390 2024-04-25T16:33:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e17ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.512 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e17ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42426/0 (socket says 172.21.15.98:42426) 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b1d9b640 1 -- 172.21.15.98:0/2531777952 learned_addr learned my addr 172.21.15.98:0/2531777952 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e18960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b2d9d640 1 -- 172.21.15.98:0/2531777952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a75390 msgr2=0x560e25e17740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a75390 0x560e25e17740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b2d9d640 1 -- 172.21.15.98:0/2531777952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 msgr2=0x560e25e17ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e17ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b2d9d640 1 -- 172.21.15.98:0/2531777952 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e25d40c30 con 0x560e25a546b0 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b1d9b640 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e17ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.512+0000 7f64b259c640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f649c0afba0 0x7f649c0b20b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f64b419b140 tx=0x7f64a4005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.513+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e18960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0dc9e0 tx=0x7f64ac0a6c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad7245819c9c1b09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.513+0000 7f64a0ff9640 1 -- 172.21.15.98:0/2531777952 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac0ab770 con 0x560e25a546b0 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.513+0000 7f64a0ff9640 1 -- 172.21.15.98:0/2531777952 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64ac0b0220 con 0x560e25a546b0 2024-04-25T16:33:44.513 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.513+0000 7f64a0ff9640 1 -- 172.21.15.98:0/2531777952 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac0a6620 con 0x560e25a546b0 2024-04-25T16:33:44.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:44 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1220537142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:33:44.517 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 -- 172.21.15.98:0/2531777952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 msgr2=0x560e25e18960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e18960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0dc9e0 tx=0x7f64ac0a6c60 comp rx=0 tx=0).stop 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 -- 172.21.15.98:0/2531777952 shutdown_connections 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e18960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e17ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 --2- 172.21.15.98:0/2531777952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a75390 0x560e25e17740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 -- 172.21.15.98:0/2531777952 >> 172.21.15.98:0/2531777952 conn(0x560e25a694e0 msgr2=0x560e25e13020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 -- 172.21.15.98:0/2531777952 shutdown_connections 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.514+0000 7f64bfb83740 1 -- 172.21.15.98:0/2531777952 wait complete. 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.515+0000 7f64bfb83740 1 -- start start 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.515+0000 7f64bfb83740 1 Processor -- start 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.515+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a4c4d0 0x560e25e29e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.518 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.515+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e2a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a75390 0x560e25e312d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64bfb83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560e25e19480 con 0x560e25a75390 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a75390 0x560e25e312d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64bfb83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560e25d02dc0 con 0x560e25a546b0 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a75390 0x560e25e312d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42428/0 (socket says 172.21.15.98:42428) 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b2d9d640 1 -- 172.21.15.98:0/1746251111 learned_addr learned my addr 172.21.15.98:0/1746251111 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a4c4d0 0x560e25e29e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b1d9b640 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e2a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560e25e19600 con 0x560e25a4c4d0 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b2d9d640 1 -- 172.21.15.98:0/1746251111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a4c4d0 msgr2=0x560e25e29e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a4c4d0 0x560e25e29e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b2d9d640 1 -- 172.21.15.98:0/1746251111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 msgr2=0x560e25e2a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e2a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.519 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.516+0000 7f64b2d9d640 1 -- 172.21.15.98:0/1746251111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e25a8d3f0 con 0x560e25a75390 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a75390 0x560e25e312d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0df5b0 tx=0x7f64ac004e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee997c0082d5e086 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f6499ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f6499ffb640 1 -- 172.21.15.98:0/1746251111 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac0054d0 con 0x560e25a75390 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f6499ffb640 1 -- 172.21.15.98:0/1746251111 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f64ac0a1b50 con 0x560e25a75390 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f64bfb83740 10 client.0 _ll_register_callbacks cb 0x560e25d1e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64ac0b0220 con 0x560e25a75390 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f6499ffb640 1 -- 172.21.15.98:0/1746251111 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f64ac0a1d50 con 0x560e25a75390 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f64bfb83740 10 client.15936 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.517+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({fsmap=29}) -- 0x560e25b577d0 con 0x560e25a75390 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.518+0000 7f6499ffb640 1 -- 172.21.15.98:0/1746251111 <== mon.0 v2:172.21.15.98:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f64ac01f020 con 0x560e25a75390 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.518+0000 7f64bfb83740 10 client.15936 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.518+0000 7f64bfb83740 10 client.15936 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-25T16:33:44.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.518+0000 7f64bfb83740 20 client.15936 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.518+0000 7f64bfb83740 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x560e25a33f60 0x560e25e37310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.518+0000 7f64bfb83740 4 client.15936 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.518+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e25d89990 con 0x560e25a33f60 2024-04-25T16:33:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.518+0000 7f64b259c640 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x560e25a33f60 0x560e25e37310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.519+0000 7f6499ffb640 10 client.15936 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:33:44.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.519+0000 7f64b259c640 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x560e25a33f60 0x560e25e37310 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dda95e033e146b5f server_cookie=8495162c9d7381e7 in_seq=0 out_seq=0 2024-04-25T16:33:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.519+0000 7f6499ffb640 1 -- 172.21.15.98:0/1746251111 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560e25d89990 con 0x560e25a33f60 2024-04-25T16:33:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.519+0000 7f6499ffb640 10 client.15936 handle_command_reply: tid=0 2024-04-25T16:33:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.519+0000 7f64bfb83740 1 client.15936 shutdown 2024-04-25T16:33:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a75390 msgr2=0x560e25e312d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.522 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a75390 0x560e25e312d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f64ac0df5b0 tx=0x7f64ac004e80 comp rx=0 tx=0).stop 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 shutdown_connections 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x560e25a33f60 0x560e25e37310 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a75390 0x560e25e312d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f6499ffb640 0 client.15936 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a546b0 0x560e25e2a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 --2- 172.21.15.98:0/1746251111 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a4c4d0 0x560e25e29e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 >> 172.21.15.98:0/1746251111 conn(0x560e25a694e0 msgr2=0x560e25e12040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 shutdown_connections 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 1 -- 172.21.15.98:0/1746251111 wait complete. 2024-04-25T16:33:44.523 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.520+0000 7f64bfb83740 20 client.15936 trim_cache size 0 max 16384 2024-04-25T16:33:44.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64bfb83740 1 Processor -- start 2024-04-25T16:33:44.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64bfb83740 1 -- start start 2024-04-25T16:33:44.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e28450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e28990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e311f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64bfb83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41aa710 con 0x560e25a4c4d0 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64bfb83740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41ac250 con 0x560e25a694e0 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64b2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e311f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64b2d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e311f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42432/0 (socket says 172.21.15.98:42432) 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64b2d9d640 1 -- 172.21.15.98:0/2252562579 learned_addr learned my addr 172.21.15.98:0/2252562579 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64bfb83740 1 -- 172.21.15.98:0/2252562579 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41aade0 con 0x560e25a33f60 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.547+0000 7f64b259c640 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e28450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b1d9b640 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e28990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b259c640 1 -- 172.21.15.98:0/2252562579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 msgr2=0x560e25e28990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b259c640 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e28990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b259c640 1 -- 172.21.15.98:0/2252562579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 msgr2=0x560e25e311f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.548 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b259c640 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e311f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b259c640 1 -- 172.21.15.98:0/2252562579 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e25e31730 con 0x560e25a694e0 2024-04-25T16:33:44.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b1d9b640 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e28990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:44.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e311f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f64b259c640 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e28450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x560e25e37a10 tx=0x7f64a4014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9209effb4cdc116 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f649bfff640 1 -- 172.21.15.98:0/2252562579 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4003d40 con 0x560e25a694e0 2024-04-25T16:33:44.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f649bfff640 1 -- 172.21.15.98:0/2252562579 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f64a402f070 con 0x560e25a694e0 2024-04-25T16:33:44.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.548+0000 7f649bfff640 1 -- 172.21.15.98:0/2252562579 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a401d680 con 0x560e25a694e0 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 -- 172.21.15.98:0/2252562579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 msgr2=0x560e25e28450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e28450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x560e25e37a10 tx=0x7f64a4014000 comp rx=0 tx=0).stop 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 -- 172.21.15.98:0/2252562579 shutdown_connections 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e311f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e28990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 --2- 172.21.15.98:0/2252562579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e28450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 -- 172.21.15.98:0/2252562579 >> 172.21.15.98:0/2252562579 conn(0x560e25e0e2e0 msgr2=0x560e25e24d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 -- 172.21.15.98:0/2252562579 shutdown_connections 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 -- 172.21.15.98:0/2252562579 wait complete. 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 -- start start 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.550+0000 7f64bfb83740 1 Processor -- start 2024-04-25T16:33:44.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e2e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e2eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64bfb83740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e2fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64bfb83740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41abb80 con 0x560e25a4c4d0 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e2eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b1d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e2eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42444/0 (socket says 172.21.15.98:42444) 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b1d9b640 1 -- 172.21.15.98:0/4253161661 learned_addr learned my addr 172.21.15.98:0/4253161661 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e2fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b259c640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e2e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41ab4b0 con 0x560e25a694e0 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41a86f0 con 0x560e25a33f60 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b259c640 1 -- 172.21.15.98:0/4253161661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 msgr2=0x560e25e2fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b259c640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e2fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b259c640 1 -- 172.21.15.98:0/4253161661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 msgr2=0x560e25e2eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b259c640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e2eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b259c640 1 -- 172.21.15.98:0/4253161661 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e25e30040 con 0x560e25a33f60 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b1d9b640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e2eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.551+0000 7f64b2d9d640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e2fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f64bfb83740 10 client.0 _ll_register_callbacks cb 0x7f64a4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64a402f070 con 0x560e25a33f60 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f64b259c640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e2e610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64a4004190 tx=0x7f64a4005760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2836179ecdb99bdc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f6499ffb640 10 client.25402 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f6499ffb640 1 -- 172.21.15.98:0/4253161661 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a4003d40 con 0x560e25a33f60 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f6499ffb640 1 -- 172.21.15.98:0/4253161661 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f64a400f4e0 con 0x560e25a33f60 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f6499ffb640 1 -- 172.21.15.98:0/4253161661 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f64a400f6e0 con 0x560e25a33f60 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f64bfb83740 10 client.25402 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.552+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({fsmap=29}) -- 0x560e25e31000 con 0x560e25a33f60 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.553+0000 7f6499ffb640 1 -- 172.21.15.98:0/4253161661 <== mon.1 v2:172.21.15.203:3300/0 4 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f64a400f880 con 0x560e25a33f60 2024-04-25T16:33:44.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.553+0000 7f64bfb83740 10 client.25402 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.553+0000 7f64bfb83740 10 client.25402 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.553+0000 7f64bfb83740 20 client.25402 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.553+0000 7f64bfb83740 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x560e25a546b0 0x560e25ea1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.553+0000 7f64bfb83740 4 client.25402 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.553+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e25d83800 con 0x560e25a546b0 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.553+0000 7f64b1d9b640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x560e25a546b0 0x560e25ea1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f6499ffb640 10 client.25402 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64b1d9b640 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x560e25a546b0 0x560e25ea1960 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b8bed1d38c8aac54 server_cookie=2c64474f257abaab in_seq=0 out_seq=0 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f6499ffb640 1 -- 172.21.15.98:0/4253161661 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e25d83800 con 0x560e25a546b0 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f6499ffb640 10 client.25402 handle_command_reply: tid=0 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 client.25402 shutdown 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 msgr2=0x560e25e2e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e2e610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f64a4004190 tx=0x7f64a4005760 comp rx=0 tx=0).stop 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 shutdown_connections 2024-04-25T16:33:44.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x560e25a546b0 0x560e25ea1960 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560e25a694e0 0x560e25e2fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560e25a4c4d0 0x560e25e2eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 --2- 172.21.15.98:0/4253161661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560e25a33f60 0x560e25e2e610 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 >> 172.21.15.98:0/4253161661 conn(0x560e25e0e2e0 msgr2=0x560e25e15e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f6499ffb640 0 client.25402 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 shutdown_connections 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 1 -- 172.21.15.98:0/4253161661 wait complete. 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.554+0000 7f64bfb83740 20 client.25402 trim_cache size 0 max 16384 2024-04-25T16:33:44.558 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:44.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f649c0afba0 msgr2=0x7f649c0b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f649c0afba0 0x7f649c0b20b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f64b419b140 tx=0x7f64a4005f90 comp rx=0 tx=0).stop 2024-04-25T16:33:44.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64b419f1d0 msgr2=0x7f64b41a15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64b419f1d0 0x7f64b41a15f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009fd0 tx=0x7f64ac00ef10 comp rx=0 tx=0).stop 2024-04-25T16:33:44.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 shutdown_connections 2024-04-25T16:33:44.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f649c0afba0 0x7f649c0b20b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64b419f1d0 0x7f64b41a15f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f64b4108a10 0x7f64b419a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 --2- 172.21.15.98:0/2877529088 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64b4103ee0 0x7f64b4199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 >> 172.21.15.98:0/2877529088 conn(0x7f64b40fd340 msgr2=0x7f64b40ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 shutdown_connections 2024-04-25T16:33:44.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.555+0000 7f64b8bfe640 1 -- 172.21.15.98:0/2877529088 wait complete. 2024-04-25T16:33:44.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.741+0000 7f66f6dce640 1 -- 172.21.15.98:0/3830692605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f01029e0 msgr2=0x7f66f01036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.741+0000 7f66f6dce640 1 --2- 172.21.15.98:0/3830692605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f01029e0 0x7f66f01036e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009900 tx=0x7f66e402a410 comp rx=0 tx=0).stop 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.741+0000 7f66f6dce640 1 -- 172.21.15.98:0/3830692605 shutdown_connections 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.741+0000 7f66f6dce640 1 --2- 172.21.15.98:0/3830692605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66f0108920 0x7f66f010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.741+0000 7f66f6dce640 1 --2- 172.21.15.98:0/3830692605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66f0103c20 0x7f66f0104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.741+0000 7f66f6dce640 1 --2- 172.21.15.98:0/3830692605 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f01029e0 0x7f66f01036e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.741+0000 7f66f6dce640 1 -- 172.21.15.98:0/3830692605 >> 172.21.15.98:0/3830692605 conn(0x7f66f00fdbd0 msgr2=0x7f66f0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.741+0000 7f66f6dce640 1 -- 172.21.15.98:0/3830692605 shutdown_connections 2024-04-25T16:33:44.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.742+0000 7f66f6dce640 1 -- 172.21.15.98:0/3830692605 wait complete. 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f6dce640 1 Processor -- start 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f6dce640 1 -- start start 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f6dce640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66f01029e0 0x7f66f0199340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f6dce640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f0103c20 0x7f66f0199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f6dce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66f0108920 0x7f66f01a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f6dce640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010e3c0 con 0x7f66f0103c20 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f6dce640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010e240 con 0x7f66f01029e0 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f6dce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010e540 con 0x7f66f0108920 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66effff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f0103c20 0x7f66f0199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f4b6a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66f01029e0 0x7f66f0199340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66effff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f0103c20 0x7f66f0199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42466/0 (socket says 172.21.15.98:42466) 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66effff640 1 -- 172.21.15.98:0/972126427 learned_addr learned my addr 172.21.15.98:0/972126427 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.743+0000 7f66f4b6a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66f01029e0 0x7f66f0199340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40284/0 (socket says 172.21.15.98:40284) 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.744+0000 7f66f536b640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66f0108920 0x7f66f01a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.744+0000 7f66f4b6a640 1 -- 172.21.15.98:0/972126427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66f0108920 msgr2=0x7f66f01a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.744+0000 7f66f4b6a640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66f0108920 0x7f66f01a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.744+0000 7f66f4b6a640 1 -- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f0103c20 msgr2=0x7f66f0199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.744+0000 7f66f4b6a640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f0103c20 0x7f66f0199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.744+0000 7f66f4b6a640 1 -- 172.21.15.98:0/972126427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f01a0e80 con 0x7f66f01029e0 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.744+0000 7f66f4b6a640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66f01029e0 0x7f66f0199340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f66e402aa00 tx=0x7f66e4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.744+0000 7f66edffb640 1 -- 172.21.15.98:0/972126427 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e403f070 con 0x7f66f01029e0 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.745+0000 7f66edffb640 1 -- 172.21.15.98:0/972126427 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66e4031ba0 con 0x7f66f01029e0 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.745+0000 7f66edffb640 1 -- 172.21.15.98:0/972126427 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4035030 con 0x7f66f01029e0 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.745+0000 7f66f6dce640 1 -- 172.21.15.98:0/972126427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f01a1110 con 0x7f66f01029e0 2024-04-25T16:33:44.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.745+0000 7f66f6dce640 1 -- 172.21.15.98:0/972126427 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f01a1580 con 0x7f66f01029e0 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.748+0000 7f66fdc81740 1 Processor -- start 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.748+0000 7f66fdc81740 1 -- start start 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472117a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 0x558472117fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d526b0 0x558472118c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66fdc81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558472079d80 con 0x558471d73390 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66fdc81740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558472033a20 con 0x558471d526b0 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66fdc81740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x558472075d40 con 0x558471d4a4d0 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66f536b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472117a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66f536b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472117a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42476/0 (socket says 172.21.15.98:42476) 2024-04-25T16:33:44.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66f536b640 1 -- 172.21.15.98:0/1443434002 learned_addr learned my addr 172.21.15.98:0/1443434002 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66edffb640 1 -- 172.21.15.98:0/972126427 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f66e4042070 con 0x7f66f01029e0 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66effff640 1 --2- 172.21.15.98:0/1443434002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d526b0 0x558472118c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66f536b640 1 -- 172.21.15.98:0/1443434002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 msgr2=0x558472117fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66f536b640 1 --2- 172.21.15.98:0/1443434002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 0x558472117fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66f536b640 1 -- 172.21.15.98:0/1443434002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d526b0 msgr2=0x558472118c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66f536b640 1 --2- 172.21.15.98:0/1443434002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d526b0 0x558472118c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.749+0000 7f66f536b640 1 -- 172.21.15.98:0/1443434002 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55847203ec30 con 0x558471d73390 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.750+0000 7f66f536b640 1 --2- 172.21.15.98:0/1443434002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472117a20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f66e8010370 tx=0x7f66e8010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6134322bcefbd32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.750+0000 7f66daffd640 1 -- 172.21.15.98:0/1443434002 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8004d20 con 0x558471d73390 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.750+0000 7f66daffd640 1 -- 172.21.15.98:0/1443434002 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66e8015d40 con 0x558471d73390 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.750+0000 7f66daffd640 1 -- 172.21.15.98:0/1443434002 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80057f0 con 0x558471d73390 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.750+0000 7f66edffb640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f66e00afaf0 0x7f66e00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.750+0000 7f66edffb640 1 -- 172.21.15.98:0/972126427 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f66e40ff740 con 0x7f66f01029e0 2024-04-25T16:33:44.750 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.750+0000 7f66f4b6a640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f66e00afaf0 0x7f66e00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.751+0000 7f66f4b6a640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f66e00afaf0 0x7f66e00b2000 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f66e40c78b0 tx=0x7f66e402a410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.751+0000 7f66fdc81740 1 -- 172.21.15.98:0/1443434002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 msgr2=0x558472117a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.751+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1443434002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472117a20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f66e8010370 tx=0x7f66e8010830 comp rx=0 tx=0).stop 2024-04-25T16:33:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.751+0000 7f66fdc81740 1 -- 172.21.15.98:0/1443434002 shutdown_connections 2024-04-25T16:33:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.751+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1443434002 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d526b0 0x558472118c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.751+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1443434002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 0x558472117fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.751+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1443434002 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472117a20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.752 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.751+0000 7f66fdc81740 1 -- 172.21.15.98:0/1443434002 >> 172.21.15.98:0/1443434002 conn(0x558471d674e0 msgr2=0x558472111020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.752+0000 7f66fdc81740 1 -- 172.21.15.98:0/1443434002 shutdown_connections 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.752+0000 7f66fdc81740 1 -- 172.21.15.98:0/1443434002 wait complete. 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.752+0000 7f66fdc81740 1 -- start start 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.752+0000 7f66fdc81740 1 Processor -- start 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d4a4d0 0x55847211f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 0x55847211fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472126ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66fdc81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x558472119740 con 0x558471d73390 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66fdc81740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x558472000dc0 con 0x558471d4a4d0 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66fdc81740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5584721198c0 con 0x558471d526b0 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66fdc81740 10 client.0 _ll_register_callbacks cb 0x55847201c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472126ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472126ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42482/0 (socket says 172.21.15.98:42482) 2024-04-25T16:33:44.753 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 -- 172.21.15.98:0/1217187437 learned_addr learned my addr 172.21.15.98:0/1217187437 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d4a4d0 0x55847211f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.754 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66effff640 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 0x55847211fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 -- 172.21.15.98:0/1217187437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 msgr2=0x55847211fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 0x55847211fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 -- 172.21.15.98:0/1217187437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d4a4d0 msgr2=0x55847211f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d4a4d0 0x55847211f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.753+0000 7f66f536b640 1 -- 172.21.15.98:0/1217187437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558471d8b3f0 con 0x558471d73390 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66f536b640 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472126ab0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f66e8005f40 tx=0x7f66e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c2942dea15c5ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66cffff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66cffff640 1 -- 172.21.15.98:0/1217187437 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8015ac0 con 0x558471d73390 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66fdc81740 1 -- 172.21.15.98:0/1217187437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66e8015d40 con 0x558471d73390 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66cffff640 1 -- 172.21.15.98:0/1217187437 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f66e8004a40 con 0x558471d73390 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66cffff640 1 -- 172.21.15.98:0/1217187437 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f66e8005a70 con 0x558471d73390 2024-04-25T16:33:44.757 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66fdc81740 10 client.15954 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66fdc81740 1 -- 172.21.15.98:0/1217187437 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558471e557d0 con 0x558471d73390 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.754+0000 7f66cffff640 1 -- 172.21.15.98:0/1217187437 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66e8015d40 con 0x558471d73390 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66cffff640 1 -- 172.21.15.98:0/1217187437 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f66e800f040 con 0x558471d73390 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66fdc81740 10 client.15954 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66fdc81740 10 client.15954 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66fdc81740 20 client.15954 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x558471d31f60 0x55847212d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66fdc81740 4 client.15954 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66fdc81740 1 -- 172.21.15.98:0/1217187437 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558472087990 con 0x558471d31f60 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66effff640 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x558471d31f60 0x55847212d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.755+0000 7f66cffff640 10 client.15954 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.756+0000 7f66effff640 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x558471d31f60 0x55847212d5c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73ed9ce819fbd238 server_cookie=bae9bd60237bec50 in_seq=0 out_seq=0 2024-04-25T16:33:44.758 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.756+0000 7f66cffff640 1 -- 172.21.15.98:0/1217187437 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558472087990 con 0x558471d31f60 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.756+0000 7f66cffff640 10 client.15954 handle_command_reply: tid=0 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.756+0000 7f66fdc81740 1 client.15954 shutdown 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 -- 172.21.15.98:0/1217187437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 msgr2=0x558472126ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472126ab0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f66e8005f40 tx=0x7f66e8009bb0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 -- 172.21.15.98:0/1217187437 shutdown_connections 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x558471d31f60 0x55847212d5c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d73390 0x558472126ab0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66cffff640 0 client.15954 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 0x55847211fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 --2- 172.21.15.98:0/1217187437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d4a4d0 0x55847211f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 -- 172.21.15.98:0/1217187437 >> 172.21.15.98:0/1217187437 conn(0x558471d674e0 msgr2=0x558472110040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 -- 172.21.15.98:0/1217187437 shutdown_connections 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 1 -- 172.21.15.98:0/1217187437 wait complete. 2024-04-25T16:33:44.759 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.757+0000 7f66fdc81740 20 client.15954 trim_cache size 0 max 16384 2024-04-25T16:33:44.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.784+0000 7f66fdc81740 1 Processor -- start 2024-04-25T16:33:44.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.784+0000 7f66fdc81740 1 -- start start 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x55847211a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d4a4d0 0x55847211aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 0x55847211dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66fdc81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66f01aa710 con 0x558471d4a4d0 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66fdc81740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f66f01ac250 con 0x558471d31f60 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66fdc81740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f66f01aade0 con 0x558471d526b0 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66effff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d4a4d0 0x55847211aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66f536b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x55847211a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66effff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d4a4d0 0x55847211aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42484/0 (socket says 172.21.15.98:42484) 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66effff640 1 -- 172.21.15.98:0/3348898923 learned_addr learned my addr 172.21.15.98:0/3348898923 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66f536b640 1 --2- 172.21.15.98:0/3348898923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 0x55847211dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66f536b640 1 -- 172.21.15.98:0/3348898923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 msgr2=0x55847211dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66f536b640 1 --2- 172.21.15.98:0/3348898923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 0x55847211dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66f536b640 1 -- 172.21.15.98:0/3348898923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d4a4d0 msgr2=0x55847211aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.785+0000 7f66f536b640 1 --2- 172.21.15.98:0/3348898923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d4a4d0 0x55847211aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.786+0000 7f66f536b640 1 -- 172.21.15.98:0/3348898923 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558472120210 con 0x558471d31f60 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.786+0000 7f66f536b640 1 --2- 172.21.15.98:0/3348898923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x55847211a4f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66e80044c0 tx=0x7f66e8009600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ad4903d4741e9ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.786+0000 7f66da7fc640 1 -- 172.21.15.98:0/3348898923 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e80056d0 con 0x558471d31f60 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.786+0000 7f66da7fc640 1 -- 172.21.15.98:0/3348898923 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66e8004540 con 0x558471d31f60 2024-04-25T16:33:44.786 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.786+0000 7f66da7fc640 1 -- 172.21.15.98:0/3348898923 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e8015ac0 con 0x558471d31f60 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 -- 172.21.15.98:0/3348898923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 msgr2=0x55847211a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3348898923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x55847211a4f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f66e80044c0 tx=0x7f66e8009600 comp rx=0 tx=0).stop 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 -- 172.21.15.98:0/3348898923 shutdown_connections 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3348898923 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d526b0 0x55847211dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3348898923 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d4a4d0 0x55847211aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3348898923 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x55847211a4f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 -- 172.21.15.98:0/3348898923 >> 172.21.15.98:0/3348898923 conn(0x558471d674e0 msgr2=0x55847212d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 -- 172.21.15.98:0/3348898923 shutdown_connections 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.787+0000 7f66fdc81740 1 -- 172.21.15.98:0/3348898923 wait complete. 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.788+0000 7f66fdc81740 1 -- start start 2024-04-25T16:33:44.790 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.788+0000 7f66fdc81740 1 Processor -- start 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.788+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x558472125d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.788+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 0x558472122890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66fdc81740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d526b0 0x558472122dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66fdc81740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f66f01abb80 con 0x558471d526b0 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66fdc81740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f66f01ab4b0 con 0x558471d31f60 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66fdc81740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f66f01a86f0 con 0x558471d4a4d0 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66fdc81740 10 client.0 _ll_register_callbacks cb 0x7f66e8009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66f536b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d526b0 0x558472122dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66effff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 0x558472122890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66f536b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d526b0 0x558472122dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42496/0 (socket says 172.21.15.98:42496) 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66f536b640 1 -- 172.21.15.98:0/3691222032 learned_addr learned my addr 172.21.15.98:0/3691222032 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66f536b640 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x558472125d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66effff640 1 -- 172.21.15.98:0/3691222032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 msgr2=0x558472125d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66effff640 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x558472125d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.791 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66effff640 1 -- 172.21.15.98:0/3691222032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d526b0 msgr2=0x558472122dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66effff640 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d526b0 0x558472122dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66effff640 1 -- 172.21.15.98:0/3691222032 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558472123310 con 0x558471d4a4d0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66f536b640 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d526b0 0x558472122dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66effff640 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 0x558472122890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00b800 tx=0x7f66dc0045f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f56842f94a890bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.789+0000 7f66cffff640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66cffff640 1 -- 172.21.15.98:0/3691222032 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc005220 con 0x558471d4a4d0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66cffff640 1 -- 172.21.15.98:0/3691222032 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f66dc0053c0 con 0x558471d4a4d0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66fdc81740 1 -- 172.21.15.98:0/3691222032 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66e8004540 con 0x558471d4a4d0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66cffff640 1 -- 172.21.15.98:0/3691222032 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f66dc0055c0 con 0x558471d4a4d0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66fdc81740 10 client.25417 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66fdc81740 1 -- 172.21.15.98:0/3691222032 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558472194950 con 0x558471d4a4d0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66cffff640 1 -- 172.21.15.98:0/3691222032 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f66dc003c90 con 0x558471d4a4d0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66cffff640 1 -- 172.21.15.98:0/3691222032 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f66dc025070 con 0x558471d4a4d0 2024-04-25T16:33:44.792 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66fdc81740 10 client.25417 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66fdc81740 10 client.25417 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-25T16:33:44.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66fdc81740 20 client.25417 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.790+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x558471d73390 0x5584721973c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.791+0000 7f66fdc81740 4 client.25417 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:44.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.791+0000 7f66fdc81740 1 -- 172.21.15.98:0/3691222032 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f66e8004540 con 0x558471d73390 2024-04-25T16:33:44.793 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.791+0000 7f66f536b640 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x558471d73390 0x5584721973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.791+0000 7f66cffff640 10 client.25417 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:33:44.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.791+0000 7f66f536b640 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x558471d73390 0x5584721973c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e675006e7d17a55 server_cookie=9b7414cbd634d1f8 in_seq=0 out_seq=0 2024-04-25T16:33:44.794 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66cffff640 1 -- 172.21.15.98:0/3691222032 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f66e8004540 con 0x558471d73390 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66cffff640 10 client.25417 handle_command_reply: tid=0 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 client.25417 shutdown 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 -- 172.21.15.98:0/3691222032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 msgr2=0x558472122890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 0x558472122890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f66dc00b800 tx=0x7f66dc0045f0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 -- 172.21.15.98:0/3691222032 shutdown_connections 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x558471d73390 0x5584721973c0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x558471d526b0 0x558472122dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66cffff640 0 client.25417 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x558471d4a4d0 0x558472122890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 --2- 172.21.15.98:0/3691222032 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x558471d31f60 0x558472125d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 -- 172.21.15.98:0/3691222032 >> 172.21.15.98:0/3691222032 conn(0x558471d674e0 msgr2=0x55847212d570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 -- 172.21.15.98:0/3691222032 shutdown_connections 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 1 -- 172.21.15.98:0/3691222032 wait complete. 2024-04-25T16:33:44.795 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.792+0000 7f66fdc81740 20 client.25417 trim_cache size 0 max 16384 2024-04-25T16:33:44.796 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.793+0000 7f66f6dce640 1 -- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f66e00afaf0 msgr2=0x7f66e00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.793+0000 7f66f6dce640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f66e00afaf0 0x7f66e00b2000 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f66e40c78b0 tx=0x7f66e402a410 comp rx=0 tx=0).stop 2024-04-25T16:33:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.793+0000 7f66f6dce640 1 -- 172.21.15.98:0/972126427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66f01029e0 msgr2=0x7f66f0199340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.793+0000 7f66f6dce640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66f01029e0 0x7f66f0199340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f66e402aa00 tx=0x7f66e4003c10 comp rx=0 tx=0).stop 2024-04-25T16:33:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.793+0000 7f66f6dce640 1 -- 172.21.15.98:0/972126427 shutdown_connections 2024-04-25T16:33:44.797 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.794+0000 7f66f6dce640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f66e00afaf0 0x7f66e00b2000 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.794+0000 7f66f6dce640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f66f0108920 0x7f66f01a0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.794+0000 7f66f6dce640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f66f0103c20 0x7f66f0199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.794+0000 7f66f6dce640 1 --2- 172.21.15.98:0/972126427 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66f01029e0 0x7f66f0199340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.794+0000 7f66f6dce640 1 -- 172.21.15.98:0/972126427 >> 172.21.15.98:0/972126427 conn(0x7f66f00fdbd0 msgr2=0x7f66f0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.794+0000 7f66f6dce640 1 -- 172.21.15.98:0/972126427 shutdown_connections 2024-04-25T16:33:44.798 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.794+0000 7f66f6dce640 1 -- 172.21.15.98:0/972126427 wait complete. 2024-04-25T16:33:44.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:33:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 -- 172.21.15.98:0/107362848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664105090 msgr2=0x7fa6641054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 --2- 172.21.15.98:0/107362848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664105090 0x7fa6641054a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa654009900 tx=0x7fa65402a410 comp rx=0 tx=0).stop 2024-04-25T16:33:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 -- 172.21.15.98:0/107362848 shutdown_connections 2024-04-25T16:33:44.979 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 --2- 172.21.15.98:0/107362848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa664102240 0x7fa66410bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 --2- 172.21.15.98:0/107362848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664101740 0x7fa664101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 --2- 172.21.15.98:0/107362848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664105090 0x7fa6641054a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 -- 172.21.15.98:0/107362848 >> 172.21.15.98:0/107362848 conn(0x7fa6640fd340 msgr2=0x7fa6640ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 -- 172.21.15.98:0/107362848 shutdown_connections 2024-04-25T16:33:44.980 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.980+0000 7fa66aa42640 1 -- 172.21.15.98:0/107362848 wait complete. 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.981+0000 7fa66aa42640 1 Processor -- start 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.981+0000 7fa66aa42640 1 -- start start 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.981+0000 7fa66aa42640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664101740 0x7fa66419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.981+0000 7fa66aa42640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664102240 0x7fa66419a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.981+0000 7fa66aa42640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa664105090 0x7fa6641a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.981+0000 7fa66aa42640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410f4b0 con 0x7fa664102240 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.981+0000 7fa66aa42640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410f330 con 0x7fa664105090 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.981+0000 7fa66aa42640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa66410f630 con 0x7fa664101740 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa6637fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664102240 0x7fa66419a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.982 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa668fdf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa664105090 0x7fa6641a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa668fdf640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa664105090 0x7fa6641a1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40348/0 (socket says 172.21.15.98:40348) 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa668fdf640 1 -- 172.21.15.98:0/1715330819 learned_addr learned my addr 172.21.15.98:0/1715330819 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa663fff640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664101740 0x7fa66419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa6637fe640 1 -- 172.21.15.98:0/1715330819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664101740 msgr2=0x7fa66419a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa6637fe640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664101740 0x7fa66419a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa6637fe640 1 -- 172.21.15.98:0/1715330819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa664105090 msgr2=0x7fa6641a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa6637fe640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa664105090 0x7fa6641a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa6637fe640 1 -- 172.21.15.98:0/1715330819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6641a1ce0 con 0x7fa664102240 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa663fff640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664101740 0x7fa66419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.982+0000 7fa6637fe640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664102240 0x7fa66419a6e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa65800eb30 tx=0x7fa658009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.983+0000 7fa6617fa640 1 -- 172.21.15.98:0/1715330819 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa65800cc60 con 0x7fa664102240 2024-04-25T16:33:44.983 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.983+0000 7fa66aa42640 1 -- 172.21.15.98:0/1715330819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6641a1fd0 con 0x7fa664102240 2024-04-25T16:33:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.983+0000 7fa6617fa640 1 -- 172.21.15.98:0/1715330819 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa65800ce00 con 0x7fa664102240 2024-04-25T16:33:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.984+0000 7fa6617fa640 1 -- 172.21.15.98:0/1715330819 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa658004a00 con 0x7fa664102240 2024-04-25T16:33:44.984 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.984+0000 7fa66aa42640 1 -- 172.21.15.98:0/1715330819 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6641a2430 con 0x7fa664102240 2024-04-25T16:33:44.985 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.987+0000 7fa6617fa640 1 -- 172.21.15.98:0/1715330819 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fa6580050b0 con 0x7fa664102240 2024-04-25T16:33:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.987+0000 7fa6719c8740 1 Processor -- start 2024-04-25T16:33:44.986 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.987+0000 7fa6719c8740 1 -- start start 2024-04-25T16:33:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.987+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e905810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.987+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e53a4d0 0x55729e905d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e5426b0 0x55729e906a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa6719c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55729e869d80 con 0x55729e563390 2024-04-25T16:33:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa6719c8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55729e823a20 con 0x55729e5426b0 2024-04-25T16:33:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa6719c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55729e865d40 con 0x55729e53a4d0 2024-04-25T16:33:44.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa6617fa640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa64c0afb70 0x7fa64c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa668fdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e905810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa663fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e53a4d0 0x55729e905d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa668fdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e905810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42526/0 (socket says 172.21.15.98:42526) 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa668fdf640 1 -- 172.21.15.98:0/919500126 learned_addr learned my addr 172.21.15.98:0/919500126 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa668fdf640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa64c0afb70 0x7fa64c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa6637fe640 1 --2- 172.21.15.98:0/919500126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e5426b0 0x55729e906a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa6617fa640 1 -- 172.21.15.98:0/1715330819 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa6580def70 con 0x7fa664102240 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa663fff640 1 -- 172.21.15.98:0/919500126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e5426b0 msgr2=0x55729e906a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa663fff640 1 --2- 172.21.15.98:0/919500126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e5426b0 0x55729e906a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa663fff640 1 -- 172.21.15.98:0/919500126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 msgr2=0x55729e905810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa663fff640 1 --2- 172.21.15.98:0/919500126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e905810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.988+0000 7fa663fff640 1 -- 172.21.15.98:0/919500126 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55729e82ec30 con 0x55729e53a4d0 2024-04-25T16:33:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.989+0000 7fa668fdf640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa64c0afb70 0x7fa64c0b2080 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009970 tx=0x7fa65c009c50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.989+0000 7fa663fff640 1 --2- 172.21.15.98:0/919500126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e53a4d0 0x55729e905d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa65402a930 tx=0x7fa654005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d171da25cd8de850 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.990+0000 7fa64a7fc640 1 -- 172.21.15.98:0/919500126 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654035030 con 0x55729e53a4d0 2024-04-25T16:33:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.990+0000 7fa64a7fc640 1 -- 172.21.15.98:0/919500126 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa654040070 con 0x55729e53a4d0 2024-04-25T16:33:44.989 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.990+0000 7fa64a7fc640 1 -- 172.21.15.98:0/919500126 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654003a80 con 0x55729e53a4d0 2024-04-25T16:33:44.990 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 -- 172.21.15.98:0/919500126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e53a4d0 msgr2=0x55729e905d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 --2- 172.21.15.98:0/919500126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e53a4d0 0x55729e905d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa65402a930 tx=0x7fa654005e20 comp rx=0 tx=0).stop 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 -- 172.21.15.98:0/919500126 shutdown_connections 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 --2- 172.21.15.98:0/919500126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e5426b0 0x55729e906a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 --2- 172.21.15.98:0/919500126 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e53a4d0 0x55729e905d90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 --2- 172.21.15.98:0/919500126 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e905810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 -- 172.21.15.98:0/919500126 >> 172.21.15.98:0/919500126 conn(0x55729e5574e0 msgr2=0x55729e901020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 -- 172.21.15.98:0/919500126 shutdown_connections 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.991+0000 7fa6719c8740 1 -- 172.21.15.98:0/919500126 wait complete. 2024-04-25T16:33:44.991 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.992+0000 7fa6719c8740 1 -- start start 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.992+0000 7fa6719c8740 1 Processor -- start 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.992+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e917e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.992+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e918340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.992+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e91f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6719c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55729e9074d0 con 0x55729e563390 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6719c8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55729e7f0dc0 con 0x55729e53a4d0 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6719c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55729e907650 con 0x55729e5426b0 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6637fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e918340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.995 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6637fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e918340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:35650/0 (socket says 172.21.15.98:35650) 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa663fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e917e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6637fe640 1 -- 172.21.15.98:0/973654065 learned_addr learned my addr 172.21.15.98:0/973654065 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa668fdf640 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e91f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6719c8740 10 client.0 _ll_register_callbacks cb 0x55729e80c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa663fff640 1 -- 172.21.15.98:0/973654065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 msgr2=0x55729e918340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa663fff640 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e918340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa663fff640 1 -- 172.21.15.98:0/973654065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 msgr2=0x55729e91f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa663fff640 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e91f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa663fff640 1 -- 172.21.15.98:0/973654065 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55729e57b3f0 con 0x55729e53a4d0 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6637fe640 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e918340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa668fdf640 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e91f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa663fff640 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e917e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa6540057c0 tx=0x7fa654031d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93992ea08d25fed6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:44.996 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.993+0000 7fa6437fe640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6437fe640 1 -- 172.21.15.98:0/973654065 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654051060 con 0x55729e53a4d0 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6437fe640 1 -- 172.21.15.98:0/973654065 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa65403c850 con 0x55729e53a4d0 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6719c8740 1 -- 172.21.15.98:0/973654065 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa654040070 con 0x55729e53a4d0 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6437fe640 1 -- 172.21.15.98:0/973654065 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa654058050 con 0x55729e53a4d0 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6719c8740 10 client.25496 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6719c8740 1 -- 172.21.15.98:0/973654065 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55729e6457d0 con 0x55729e53a4d0 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6437fe640 1 -- 172.21.15.98:0/973654065 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa654044410 con 0x55729e53a4d0 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6437fe640 1 -- 172.21.15.98:0/973654065 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa65404c400 con 0x55729e53a4d0 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6719c8740 10 client.25496 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6719c8740 10 client.25496 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6719c8740 20 client.25496 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.994+0000 7fa6719c8740 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55729e521f60 0x55729e9256b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.995+0000 7fa6719c8740 4 client.25496 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:44.997 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.995+0000 7fa6719c8740 1 -- 172.21.15.98:0/973654065 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fa654040070 con 0x55729e521f60 2024-04-25T16:33:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.995+0000 7fa6637fe640 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55729e521f60 0x55729e9256b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.995+0000 7fa6437fe640 10 client.25496 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.995+0000 7fa6637fe640 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55729e521f60 0x55729e9256b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63b537320da238f9 server_cookie=9298f1c6a3f0d8cb in_seq=0 out_seq=0 2024-04-25T16:33:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.996+0000 7fa6437fe640 1 -- 172.21.15.98:0/973654065 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fa65808fe50 con 0x55729e521f60 2024-04-25T16:33:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.996+0000 7fa6437fe640 10 client.25496 handle_command_reply: tid=0 2024-04-25T16:33:44.998 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.996+0000 7fa6719c8740 1 client.25496 shutdown 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.996+0000 7fa6719c8740 1 -- 172.21.15.98:0/973654065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 msgr2=0x55729e917e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.996+0000 7fa6719c8740 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e917e00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa6540057c0 tx=0x7fa654031d20 comp rx=0 tx=0).stop 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 1 -- 172.21.15.98:0/973654065 shutdown_connections 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55729e521f60 0x55729e9256b0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e91f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e918340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 1 --2- 172.21.15.98:0/973654065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e917e00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 1 -- 172.21.15.98:0/973654065 >> 172.21.15.98:0/973654065 conn(0x55729e5574e0 msgr2=0x55729e903f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6437fe640 0 client.25496 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 1 -- 172.21.15.98:0/973654065 shutdown_connections 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 1 -- 172.21.15.98:0/973654065 wait complete. 2024-04-25T16:33:44.999 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:44.997+0000 7fa6719c8740 20 client.25496 trim_cache size 0 max 16384 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa6719c8740 1 Processor -- start 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa6719c8740 1 -- start start 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e916c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e91f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e903e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa6719c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641aa730 con 0x55729e563390 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa6719c8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ac270 con 0x55729e53a4d0 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa6719c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641aae00 con 0x55729e5426b0 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa663fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e916c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:45.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa668fdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e903e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa663fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e916c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:40358/0 (socket says 172.21.15.98:40358) 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa668fdf640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e903e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42538/0 (socket says 172.21.15.98:42538) 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.022+0000 7fa663fff640 1 -- 172.21.15.98:0/2103714036 learned_addr learned my addr 172.21.15.98:0/2103714036 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa663fff640 1 -- 172.21.15.98:0/2103714036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 msgr2=0x55729e91f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa6637fe640 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e91f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa663fff640 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e91f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa663fff640 1 -- 172.21.15.98:0/2103714036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 msgr2=0x55729e903e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa663fff640 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e903e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa663fff640 1 -- 172.21.15.98:0/2103714036 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55729e918a00 con 0x55729e53a4d0 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa6637fe640 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e91f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa668fdf640 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e903e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:45.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa663fff640 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e916c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa65404f040 tx=0x7fa6540097a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98711a15f11089c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:45.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.023+0000 7fa649ffb640 1 -- 172.21.15.98:0/2103714036 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654060020 con 0x55729e53a4d0 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.024+0000 7fa649ffb640 1 -- 172.21.15.98:0/2103714036 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa65404c400 con 0x55729e53a4d0 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.024+0000 7fa649ffb640 1 -- 172.21.15.98:0/2103714036 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa654051070 con 0x55729e53a4d0 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 -- 172.21.15.98:0/2103714036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 msgr2=0x55729e916c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e916c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa65404f040 tx=0x7fa6540097a0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 -- 172.21.15.98:0/2103714036 shutdown_connections 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e563390 0x55729e903e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e5426b0 0x55729e91f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 --2- 172.21.15.98:0/2103714036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e916c50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 -- 172.21.15.98:0/2103714036 >> 172.21.15.98:0/2103714036 conn(0x55729e5574e0 msgr2=0x55729e916540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 -- 172.21.15.98:0/2103714036 shutdown_connections 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 -- 172.21.15.98:0/2103714036 wait complete. 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.025+0000 7fa6719c8740 1 -- start start 2024-04-25T16:33:45.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6719c8740 1 Processor -- start 2024-04-25T16:33:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e91ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e5426b0 0x55729e91b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6719c8740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e563390 0x55729e91bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6719c8740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641abba0 con 0x55729e5426b0 2024-04-25T16:33:45.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6719c8740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641ab4d0 con 0x55729e53a4d0 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6719c8740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6641a8710 con 0x55729e563390 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6719c8740 10 client.0 _ll_register_callbacks cb 0x7fa65c013480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6637fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e5426b0 0x55729e91b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6637fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e5426b0 0x55729e91b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:42550/0 (socket says 172.21.15.98:42550) 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa6637fe640 1 -- 172.21.15.98:0/1696561533 learned_addr learned my addr 172.21.15.98:0/1696561533 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa663fff640 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e91ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.026+0000 7fa668fdf640 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e563390 0x55729e91bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6637fe640 1 -- 172.21.15.98:0/1696561533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e563390 msgr2=0x55729e91bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6637fe640 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e563390 0x55729e91bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6637fe640 1 -- 172.21.15.98:0/1696561533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 msgr2=0x55729e91ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6637fe640 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e91ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6637fe640 1 -- 172.21.15.98:0/1696561533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55729e91f3a0 con 0x55729e5426b0 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6637fe640 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e5426b0 0x55729e91b880 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa6580a6d40 tx=0x7fa65801bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a5d0fe1be2c877 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:45.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6437fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6437fe640 1 -- 172.21.15.98:0/1696561533 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6580a7040 con 0x55729e5426b0 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6437fe640 1 -- 172.21.15.98:0/1696561533 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa6580a32d0 con 0x55729e5426b0 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6719c8740 1 -- 172.21.15.98:0/1696561533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa65404c400 con 0x55729e5426b0 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6437fe640 1 -- 172.21.15.98:0/1696561533 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa6580ab3f0 con 0x55729e5426b0 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6719c8740 10 client.15978 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.027+0000 7fa6719c8740 1 -- 172.21.15.98:0/1696561533 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55729e98c730 con 0x55729e5426b0 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa6437fe640 1 -- 172.21.15.98:0/1696561533 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa65404c400 con 0x55729e5426b0 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa6437fe640 1 -- 172.21.15.98:0/1696561533 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa6580c9040 con 0x55729e5426b0 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa6719c8740 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa6719c8740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa6719c8740 20 client.15978 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa6719c8740 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55729e4f73a0 0x55729e98f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa6719c8740 4 client.15978 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:33:45.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa6719c8740 1 -- 172.21.15.98:0/1696561533 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55729e877990 con 0x55729e4f73a0 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.028+0000 7fa663fff640 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55729e4f73a0 0x55729e98f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6437fe640 10 client.15978 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa663fff640 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55729e4f73a0 0x55729e98f0b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a6bc9064b07d2ba server_cookie=c3019751aee7ca93 in_seq=0 out_seq=0 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6437fe640 1 -- 172.21.15.98:0/1696561533 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55729e877990 con 0x55729e4f73a0 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6437fe640 10 client.15978 handle_command_reply: tid=0 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 client.15978 shutdown 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 -- 172.21.15.98:0/1696561533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e5426b0 msgr2=0x55729e91b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e5426b0 0x55729e91b880 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa6580a6d40 tx=0x7fa65801bc70 comp rx=0 tx=0).stop 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 -- 172.21.15.98:0/1696561533 shutdown_connections 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55729e4f73a0 0x55729e98f0b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55729e563390 0x55729e91bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55729e5426b0 0x55729e91b880 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6437fe640 0 client.15978 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:45.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 --2- 172.21.15.98:0/1696561533 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55729e53a4d0 0x55729e91ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 -- 172.21.15.98:0/1696561533 >> 172.21.15.98:0/1696561533 conn(0x55729e5574e0 msgr2=0x55729e9045b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:45.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 -- 172.21.15.98:0/1696561533 shutdown_connections 2024-04-25T16:33:45.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.029+0000 7fa6719c8740 1 -- 172.21.15.98:0/1696561533 wait complete. 2024-04-25T16:33:45.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa6719c8740 20 client.15978 trim_cache size 0 max 16384 2024-04-25T16:33:45.033 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:33:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 -- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa64c0afb70 msgr2=0x7fa64c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa64c0afb70 0x7fa64c0b2080 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa65c009970 tx=0x7fa65c009c50 comp rx=0 tx=0).stop 2024-04-25T16:33:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 -- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664102240 msgr2=0x7fa66419a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664102240 0x7fa66419a6e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa65800eb30 tx=0x7fa658009f60 comp rx=0 tx=0).stop 2024-04-25T16:33:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 -- 172.21.15.98:0/1715330819 shutdown_connections 2024-04-25T16:33:45.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa64c0afb70 0x7fa64c0b2080 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa664105090 0x7fa6641a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa664102240 0x7fa66419a6e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 --2- 172.21.15.98:0/1715330819 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa664101740 0x7fa66419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 -- 172.21.15.98:0/1715330819 >> 172.21.15.98:0/1715330819 conn(0x7fa6640fd340 msgr2=0x7fa664105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 -- 172.21.15.98:0/1715330819 shutdown_connections 2024-04-25T16:33:45.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:45.030+0000 7fa66aa42640 1 -- 172.21.15.98:0/1715330819 wait complete. 2024-04-25T16:33:45.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:45 smithi203 ceph-mon[33115]: pgmap v393: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:33:45.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:45 smithi098 ceph-mon[29162]: pgmap v393: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:33:45.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:45 smithi173 ceph-mon[32556]: pgmap v393: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:33:46.041 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:33:46.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:33:46.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:33:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 -- 172.21.15.98:0/4043438952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0109a40 msgr2=0x7f5fd010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/4043438952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0109a40 0x7f5fd010be60 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009ca0 tx=0x7f5fc802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 -- 172.21.15.98:0/4043438952 shutdown_connections 2024-04-25T16:33:46.167 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/4043438952 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0109a40 0x7f5fd010be60 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/4043438952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fd0104d40 0x7f5fd0105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/4043438952 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0103b00 0x7f5fd0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 -- 172.21.15.98:0/4043438952 >> 172.21.15.98:0/4043438952 conn(0x7f5fd00fecf0 msgr2=0x7f5fd01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 -- 172.21.15.98:0/4043438952 shutdown_connections 2024-04-25T16:33:46.168 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.167+0000 7f5fd4eca640 1 -- 172.21.15.98:0/4043438952 wait complete. 2024-04-25T16:33:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.168+0000 7f5fd4eca640 1 Processor -- start 2024-04-25T16:33:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fd4eca640 1 -- start start 2024-04-25T16:33:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fd4eca640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fd0103b00 0x7f5fd019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fd4eca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0104d40 0x7f5fd019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fd4eca640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0109a40 0x7f5fd01a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fcdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0104d40 0x7f5fd019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.170 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fcdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0104d40 0x7f5fd019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41530/0 (socket says 172.21.15.98:41530) 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fd4eca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010f530 con 0x7f5fd0104d40 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fd4eca640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010f3b0 con 0x7f5fd0103b00 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fd4eca640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd010f6b0 con 0x7f5fd0109a40 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fcdd9b640 1 -- 172.21.15.98:0/122545971 learned_addr learned my addr 172.21.15.98:0/122545971 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fce59c640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fd0103b00 0x7f5fd019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fcdd9b640 1 -- 172.21.15.98:0/122545971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0109a40 msgr2=0x7f5fd01a15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0109a40 0x7f5fd01a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fcdd9b640 1 -- 172.21.15.98:0/122545971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fd0103b00 msgr2=0x7f5fd019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fd0103b00 0x7f5fd019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fcdd9b640 1 -- 172.21.15.98:0/122545971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd01a1c40 con 0x7f5fd0104d40 2024-04-25T16:33:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.169+0000 7f5fce59c640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fd0103b00 0x7f5fd019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.170+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0104d40 0x7f5fd019a640 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400d950 tx=0x7f5fc400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.170+0000 7f5fbf7fe640 1 -- 172.21.15.98:0/122545971 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4017070 con 0x7f5fd0104d40 2024-04-25T16:33:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.170+0000 7f5fbf7fe640 1 -- 172.21.15.98:0/122545971 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fc4009c30 con 0x7f5fd0104d40 2024-04-25T16:33:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.170+0000 7f5fd4eca640 1 -- 172.21.15.98:0/122545971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd01a1f30 con 0x7f5fd0104d40 2024-04-25T16:33:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.170+0000 7f5fbf7fe640 1 -- 172.21.15.98:0/122545971 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc400f620 con 0x7f5fd0104d40 2024-04-25T16:33:46.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.170+0000 7f5fd4eca640 1 -- 172.21.15.98:0/122545971 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd01a23f0 con 0x7f5fd0104d40 2024-04-25T16:33:46.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fbf7fe640 1 -- 172.21.15.98:0/122545971 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f5fc400f7c0 con 0x7f5fd0104d40 2024-04-25T16:33:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fdbd7a740 1 Processor -- start 2024-04-25T16:33:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fdbd7a740 1 -- start start 2024-04-25T16:33:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7d74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.181 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7d7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7de920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5612dc73cb00 con 0x5612dc436410 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5612dc6d8920 con 0x5612dc415730 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.173+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5612dc738ac0 con 0x5612dc40d550 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7d74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7d74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41534/0 (socket says 172.21.15.98:41534) 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 -- 172.21.15.98:0/1088802443 learned_addr learned my addr 172.21.15.98:0/1088802443 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fbf7fe640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5fb80afb70 0x7f5fb80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7de920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fce59c640 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7d7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 -- 172.21.15.98:0/1088802443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 msgr2=0x5612dc7d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7d7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.182 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 -- 172.21.15.98:0/1088802443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 msgr2=0x5612dc7de920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7de920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 -- 172.21.15.98:0/1088802443 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612dc5167e0 con 0x5612dc436410 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7de920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5fb80afb70 0x7f5fb80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fbf7fe640 1 -- 172.21.15.98:0/122545971 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f5fc40dec90 con 0x7f5fd0104d40 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.174+0000 7f5fced9d640 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7d74a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80319e0 tx=0x7f5fc8009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e3a7a108973536 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.175+0000 7f5fbcff9640 1 -- 172.21.15.98:0/1088802443 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc804d070 con 0x5612dc436410 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.175+0000 7f5fbcff9640 1 -- 172.21.15.98:0/1088802443 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fc8036040 con 0x5612dc436410 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.175+0000 7f5fbcff9640 1 -- 172.21.15.98:0/1088802443 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8030a50 con 0x5612dc436410 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1088802443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 msgr2=0x5612dc7d74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fced9d640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5fb80afb70 0x7f5fb80b2080 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5fc804a000 tx=0x7f5fc8047000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7d74a0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80319e0 tx=0x7f5fc8009960 comp rx=0 tx=0).stop 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1088802443 shutdown_connections 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7de920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.183 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7d7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1088802443 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7d74a0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1088802443 >> 172.21.15.98:0/1088802443 conn(0x5612dc42a560 msgr2=0x5612dc7d2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1088802443 shutdown_connections 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1088802443 wait complete. 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 -- start start 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.176+0000 7f5fdbd7a740 1 Processor -- start 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7e1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7e85f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5612dc7df1f0 con 0x5612dc436410 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5612dc6d65d0 con 0x5612dc415730 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5612dc7df370 con 0x5612dc40d550 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fdbd7a740 10 client.0 _ll_register_callbacks cb 0x5612dc6c4080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7e85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.184 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7e1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7e85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41536/0 (socket says 172.21.15.98:41536) 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7e1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:55480/0 (socket says 172.21.15.98:55480) 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fced9d640 1 -- 172.21.15.98:0/367820553 learned_addr learned my addr 172.21.15.98:0/367820553 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fce59c640 1 -- 172.21.15.98:0/367820553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 msgr2=0x5612dc7e1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fce59c640 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fce59c640 1 -- 172.21.15.98:0/367820553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 msgr2=0x5612dc7e85f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fce59c640 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7e85f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fce59c640 1 -- 172.21.15.98:0/367820553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612dc576f70 con 0x5612dc40d550 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fced9d640 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7e85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fce59c640 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7e1150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x5612dc7d8880 tx=0x7f5fc0013b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3669fc49fa4be3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.177+0000 7f5fb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/367820553 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0011da0 con 0x5612dc40d550 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/367820553 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f5fc001b3e0 con 0x5612dc40d550 2024-04-25T16:33:46.185 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/367820553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fc8036040 con 0x5612dc40d550 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/367820553 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5fc001b5e0 con 0x5612dc40d550 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fdbd7a740 10 client.25441 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/367820553 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5612dc40e170 con 0x5612dc40d550 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/367820553 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fc001bc10 con 0x5612dc40d550 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/367820553 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5fc001a070 con 0x5612dc40d550 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fdbd7a740 10 client.25441 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fdbd7a740 10 client.25441 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fdbd7a740 20 client.25441 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.178+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5612dc3f51f0 0x5612dc7eeff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.179+0000 7f5fdbd7a740 4 client.25441 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.179+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/367820553 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f5fc8036040 con 0x5612dc3f51f0 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.179+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5612dc3f51f0 0x5612dc7eeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.179+0000 7f5fb5ffb640 10 client.25441 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:46.186 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.179+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5612dc3f51f0 0x5612dc7eeff0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a351d00f9320c7b2 server_cookie=370c601b3ec461d in_seq=0 out_seq=0 2024-04-25T16:33:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/367820553 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f5fc8036040 con 0x5612dc3f51f0 2024-04-25T16:33:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fb5ffb640 10 client.25441 handle_command_reply: tid=0 2024-04-25T16:33:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 client.25441 shutdown 2024-04-25T16:33:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/367820553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 msgr2=0x5612dc7e1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.188 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7e1150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x5612dc7d8880 tx=0x7f5fc0013b70 comp rx=0 tx=0).stop 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/367820553 shutdown_connections 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5612dc3f51f0 0x5612dc7eeff0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc436410 0x5612dc7e85f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fb5ffb640 0 client.25441 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/367820553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc40d550 0x5612dc7e1150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/367820553 >> 172.21.15.98:0/367820553 conn(0x5612dc42a560 msgr2=0x5612dc7d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/367820553 shutdown_connections 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/367820553 wait complete. 2024-04-25T16:33:46.189 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.180+0000 7f5fdbd7a740 20 client.25441 trim_cache size 0 max 16384 2024-04-25T16:33:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fdbd7a740 1 Processor -- start 2024-04-25T16:33:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fdbd7a740 1 -- start start 2024-04-25T16:33:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 0x5612dc7dbd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc40d550 0x5612dc7e1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e8510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd01aa7c0 con 0x5612dc40d550 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd01ac300 con 0x5612dc415730 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd01aae90 con 0x5612dc3f51f0 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fcdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc40d550 0x5612dc7e1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fcdd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc40d550 0x5612dc7e1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41544/0 (socket says 172.21.15.98:41544) 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fcdd9b640 1 -- 172.21.15.98:0/4135813779 learned_addr learned my addr 172.21.15.98:0/4135813779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fce59c640 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 0x5612dc7dbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fced9d640 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fcdd9b640 1 -- 172.21.15.98:0/4135813779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 msgr2=0x5612dc7dbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 0x5612dc7dbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fcdd9b640 1 -- 172.21.15.98:0/4135813779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 msgr2=0x5612dc7e8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.199+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.200+0000 7f5fcdd9b640 1 -- 172.21.15.98:0/4135813779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612dc7deab0 con 0x5612dc40d550 2024-04-25T16:33:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.200+0000 7f5fced9d640 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:33:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.200+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc40d550 0x5612dc7e1430 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40dc3e0 tx=0x7f5fc4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2952cebbd1ba6262 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.200+0000 7f5fb7fff640 1 -- 172.21.15.98:0/4135813779 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc4020070 con 0x5612dc40d550 2024-04-25T16:33:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.200+0000 7f5fb7fff640 1 -- 172.21.15.98:0/4135813779 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fc40ab170 con 0x5612dc40d550 2024-04-25T16:33:46.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.200+0000 7f5fb7fff640 1 -- 172.21.15.98:0/4135813779 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc40a2a20 con 0x5612dc40d550 2024-04-25T16:33:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/4135813779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc40d550 msgr2=0x5612dc7e1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc40d550 0x5612dc7e1430 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40dc3e0 tx=0x7f5fc4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/4135813779 shutdown_connections 2024-04-25T16:33:46.206 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc415730 0x5612dc7e8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc40d550 0x5612dc7e1430 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/4135813779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 0x5612dc7dbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/4135813779 >> 172.21.15.98:0/4135813779 conn(0x5612dc42a560 msgr2=0x5612dc7e0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/4135813779 shutdown_connections 2024-04-25T16:33:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/4135813779 wait complete. 2024-04-25T16:33:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 -- start start 2024-04-25T16:33:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.201+0000 7f5fdbd7a740 1 Processor -- start 2024-04-25T16:33:46.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 0x5612dc7e8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc40d550 0x5612dc7e5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fdbd7a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc415730 0x5612dc7e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd01abc30 con 0x5612dc415730 2024-04-25T16:33:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fdbd7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd01ab560 con 0x5612dc40d550 2024-04-25T16:33:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc415730 0x5612dc7e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 0x5612dc7e8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc415730 0x5612dc7e5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41556/0 (socket says 172.21.15.98:41556) 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fced9d640 1 -- 172.21.15.98:0/1068384294 learned_addr learned my addr 172.21.15.98:0/1068384294 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fcdd9b640 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc40d550 0x5612dc7e5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd01a87a0 con 0x5612dc3f51f0 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fdbd7a740 10 client.0 _ll_register_callbacks cb 0x7f5fc0009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fced9d640 1 -- 172.21.15.98:0/1068384294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 msgr2=0x5612dc7e8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fced9d640 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 0x5612dc7e8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fced9d640 1 -- 172.21.15.98:0/1068384294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc40d550 msgr2=0x5612dc7e5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fced9d640 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc40d550 0x5612dc7e5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.202+0000 7f5fced9d640 1 -- 172.21.15.98:0/1068384294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612dc8553c0 con 0x5612dc415730 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fced9d640 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc415730 0x5612dc7e5f70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80319e0 tx=0x7f5fc80041f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debf08cc24bca596 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/1068384294 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc804d040 con 0x5612dc415730 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/1068384294 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f5fc8004670 con 0x5612dc415730 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fc40ab170 con 0x5612dc415730 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/1068384294 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5fc8064050 con 0x5612dc415730 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fdbd7a740 10 client.15996 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5612dc856600 con 0x5612dc415730 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/1068384294 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5fc40ab170 con 0x5612dc415730 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/1068384294 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f5fc8044040 con 0x5612dc415730 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.203+0000 7f5fdbd7a740 10 client.15996 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.204+0000 7f5fdbd7a740 10 client.15996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:46.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.204+0000 7f5fdbd7a740 20 client.15996 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.204+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5612dc436410 0x5612dc858c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.204+0000 7f5fdbd7a740 4 client.15996 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.204+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5612dc74a710 con 0x5612dc436410 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.204+0000 7f5fce59c640 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5612dc436410 0x5612dc858c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.204+0000 7f5fb5ffb640 10 client.15996 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.204+0000 7f5fce59c640 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5612dc436410 0x5612dc858c80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf40d7fe77487c6f server_cookie=fda8b40c26ea4771 in_seq=0 out_seq=0 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fb5ffb640 1 -- 172.21.15.98:0/1068384294 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5612dc74a710 con 0x5612dc436410 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fb5ffb640 10 client.15996 handle_command_reply: tid=0 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 client.15996 shutdown 2024-04-25T16:33:46.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc415730 msgr2=0x5612dc7e5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc415730 0x5612dc7e5f70 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5fc80319e0 tx=0x7f5fc80041f0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 shutdown_connections 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5612dc436410 0x5612dc858c80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5612dc415730 0x5612dc7e5f70 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fb5ffb640 0 client.15996 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5612dc40d550 0x5612dc7e5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 --2- 172.21.15.98:0/1068384294 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5612dc3f51f0 0x5612dc7e8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 >> 172.21.15.98:0/1068384294 conn(0x5612dc42a560 msgr2=0x5612dc7d5c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 shutdown_connections 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 1 -- 172.21.15.98:0/1068384294 wait complete. 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.206+0000 7f5fdbd7a740 20 client.15996 trim_cache size 0 max 16384 2024-04-25T16:33:46.212 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:33:46.213 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-04-25T16:33:46.213 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "5fdb7f66-c7df-4628-9e9b-3c80356ef362", 2024-04-25T16:33:46.213 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-04-25T16:33:46.213 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:33:46.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 -- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5fb80afb70 msgr2=0x7f5fb80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5fb80afb70 0x7f5fb80b2080 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5fc804a000 tx=0x7f5fc8047000 comp rx=0 tx=0).stop 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 -- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0104d40 msgr2=0x7f5fd019a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0104d40 0x7f5fd019a640 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f5fc400d950 tx=0x7f5fc400de10 comp rx=0 tx=0).stop 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 -- 172.21.15.98:0/122545971 shutdown_connections 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f5fb80afb70 0x7f5fb80b2080 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5fd0109a40 0x7f5fd01a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5fd0104d40 0x7f5fd019a640 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 --2- 172.21.15.98:0/122545971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5fd0103b00 0x7f5fd019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 -- 172.21.15.98:0/122545971 >> 172.21.15.98:0/122545971 conn(0x7f5fd00fecf0 msgr2=0x7f5fd0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 -- 172.21.15.98:0/122545971 shutdown_connections 2024-04-25T16:33:46.214 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.207+0000 7f5fd4eca640 1 -- 172.21.15.98:0/122545971 wait complete. 2024-04-25T16:33:46.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:33:46.217 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:33:46.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/427389027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4101740 msgr2=0x7f6df4101bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.333 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/427389027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4101740 0x7f6df4101bb0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009900 tx=0x7f6de402a410 comp rx=0 tx=0).stop 2024-04-25T16:33:46.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/427389027 shutdown_connections 2024-04-25T16:33:46.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/427389027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df4102240 0x7f6df410bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/427389027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4101740 0x7f6df4101bb0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/427389027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6df4105090 0x7f6df41054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.334 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/427389027 >> 172.21.15.98:0/427389027 conn(0x7f6df40fd340 msgr2=0x7f6df40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/427389027 shutdown_connections 2024-04-25T16:33:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/427389027 wait complete. 2024-04-25T16:33:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.330+0000 7f6dfc7a8640 1 Processor -- start 2024-04-25T16:33:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfc7a8640 1 -- start start 2024-04-25T16:33:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfc7a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6df4101740 0x7f6df419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.335 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfc7a8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df4102240 0x7f6df419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfc7a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4105090 0x7f6df41a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfc7a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6df410f500 con 0x7f6df4105090 2024-04-25T16:33:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfc7a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6df410f380 con 0x7f6df4101740 2024-04-25T16:33:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfc7a8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6df410f680 con 0x7f6df4102240 2024-04-25T16:33:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfa544640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6df4101740 0x7f6df419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.336 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfad45640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4105090 0x7f6df41a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfa544640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6df4101740 0x7f6df419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:56810/0 (socket says 172.21.15.98:56810) 2024-04-25T16:33:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfa544640 1 -- 172.21.15.98:0/4161278093 learned_addr learned my addr 172.21.15.98:0/4161278093 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6df9d43640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df4102240 0x7f6df419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfad45640 1 -- 172.21.15.98:0/4161278093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df4102240 msgr2=0x7f6df419a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.337 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfad45640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df4102240 0x7f6df419a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfad45640 1 -- 172.21.15.98:0/4161278093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6df4101740 msgr2=0x7f6df419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfad45640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6df4101740 0x7f6df419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfad45640 1 -- 172.21.15.98:0/4161278093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df41a1ca0 con 0x7f6df4105090 2024-04-25T16:33:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.331+0000 7f6dfa544640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6df4101740 0x7f6df419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:33:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.332+0000 7f6dfad45640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4105090 0x7f6df41a1630 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00ec50 tx=0x7f6dec00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.332+0000 7f6deb7fe640 1 -- 172.21.15.98:0/4161278093 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec003b70 con 0x7f6df4105090 2024-04-25T16:33:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.332+0000 7f6deb7fe640 1 -- 172.21.15.98:0/4161278093 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6dec003d10 con 0x7f6df4105090 2024-04-25T16:33:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.332+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/4161278093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df41a1f90 con 0x7f6df4105090 2024-04-25T16:33:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.332+0000 7f6deb7fe640 1 -- 172.21.15.98:0/4161278093 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec0107a0 con 0x7f6df4105090 2024-04-25T16:33:46.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.332+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/4161278093 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df41a23f0 con 0x7f6df4105090 2024-04-25T16:33:46.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.335+0000 7f6e03656740 1 Processor -- start 2024-04-25T16:33:46.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.336+0000 7f6deb7fe640 1 -- 172.21.15.98:0/4161278093 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f6dec003fc0 con 0x7f6df4105090 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.336+0000 7f6e03656740 1 -- start start 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6e03656740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 0x55d5afa97e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6e03656740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 0x55d5afa983f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6e03656740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6d16b0 0x55d5afa99080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6e03656740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d5af9f8d80 con 0x55d5af6d16b0 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6e03656740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d5af9b28b0 con 0x55d5af6c94d0 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6e03656740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d5af9f4d40 con 0x55d5af6f2390 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6d16b0 0x55d5afa99080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 0x55d5afa97e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6d16b0 0x55d5afa99080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41586/0 (socket says 172.21.15.98:41586) 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6df9d43640 1 -- 172.21.15.98:0/4119876410 learned_addr learned my addr 172.21.15.98:0/4119876410 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6dfa544640 1 --2- 172.21.15.98:0/4119876410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 0x55d5afa983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.337+0000 7f6deb7fe640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6de00afb70 0x7f6de00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6df9d43640 1 -- 172.21.15.98:0/4119876410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 msgr2=0x55d5afa97e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6df9d43640 1 --2- 172.21.15.98:0/4119876410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 0x55d5afa97e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6df9d43640 1 -- 172.21.15.98:0/4119876410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 msgr2=0x55d5afa983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6df9d43640 1 --2- 172.21.15.98:0/4119876410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 0x55d5afa983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6df9d43640 1 -- 172.21.15.98:0/4119876410 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5af983a40 con 0x55d5af6d16b0 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6deb7fe640 1 -- 172.21.15.98:0/4161278093 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f6dec0df4a0 con 0x7f6df4105090 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6dfad45640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6de00afb70 0x7f6de00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6df9d43640 1 --2- 172.21.15.98:0/4119876410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6d16b0 0x55d5afa99080 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x55d5afa98ec0 tx=0x7f6de4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883bdc8783296cf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6de8ff9640 1 -- 172.21.15.98:0/4119876410 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4035040 con 0x55d5af6d16b0 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6de8ff9640 1 -- 172.21.15.98:0/4119876410 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6de4003d10 con 0x55d5af6d16b0 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.338+0000 7f6de8ff9640 1 -- 172.21.15.98:0/4119876410 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4031be0 con 0x55d5af6d16b0 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 -- 172.21.15.98:0/4119876410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6d16b0 msgr2=0x55d5afa99080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6dfad45640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6de00afb70 0x7f6de00b2080 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0049d0 tx=0x7f6dec004960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 --2- 172.21.15.98:0/4119876410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6d16b0 0x55d5afa99080 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x55d5afa98ec0 tx=0x7f6de4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 -- 172.21.15.98:0/4119876410 shutdown_connections 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 --2- 172.21.15.98:0/4119876410 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6d16b0 0x55d5afa99080 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 --2- 172.21.15.98:0/4119876410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 0x55d5afa983f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 --2- 172.21.15.98:0/4119876410 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 0x55d5afa97e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 -- 172.21.15.98:0/4119876410 >> 172.21.15.98:0/4119876410 conn(0x55d5af6e64e0 msgr2=0x55d5afa8f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 -- 172.21.15.98:0/4119876410 shutdown_connections 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.339+0000 7f6e03656740 1 -- 172.21.15.98:0/4119876410 wait complete. 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 1 -- start start 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 1 Processor -- start 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 0x55d5afa96d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6d16b0 0x55d5afa97240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa93880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d5afa99b30 con 0x55d5af6f2390 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d5af97fd10 con 0x55d5af6c94d0 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d5afa99cb0 con 0x55d5af6d16b0 2024-04-25T16:33:46.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6e03656740 10 client.0 _ll_register_callbacks cb 0x55d5af9ac210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa93880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa93880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41600/0 (socket says 172.21.15.98:41600) 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.340+0000 7f6df9d43640 1 -- 172.21.15.98:0/1956335315 learned_addr learned my addr 172.21.15.98:0/1956335315 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6dfa544640 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6d16b0 0x55d5afa97240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6df9d43640 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 0x55d5afa96d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6df9d43640 1 -- 172.21.15.98:0/1956335315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6d16b0 msgr2=0x55d5afa97240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6df9d43640 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6d16b0 0x55d5afa97240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6df9d43640 1 -- 172.21.15.98:0/1956335315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 msgr2=0x55d5afa96d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6df9d43640 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 0x55d5afa96d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6df9d43640 1 -- 172.21.15.98:0/1956335315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5af7d47d0 con 0x55d5af6f2390 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6df9d43640 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa93880 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f6de402a930 tx=0x7f6de4041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a90e7ae5499606 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6dddffb640 1 -- 172.21.15.98:0/1956335315 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de404f030 con 0x55d5af6f2390 2024-04-25T16:33:46.350 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6dddffb640 1 -- 172.21.15.98:0/1956335315 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f6de4003680 con 0x55d5af6f2390 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6e03656740 1 -- 172.21.15.98:0/1956335315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de4003d10 con 0x55d5af6f2390 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6dddffb640 1 -- 172.21.15.98:0/1956335315 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6de4058050 con 0x55d5af6f2390 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6e03656740 10 client.16008 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6e03656740 1 -- 172.21.15.98:0/1956335315 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5af70a3f0 con 0x55d5af6f2390 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.341+0000 7f6dddffb640 1 -- 172.21.15.98:0/1956335315 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6de4003d10 con 0x55d5af6f2390 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6dddffb640 1 -- 172.21.15.98:0/1956335315 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6de403a040 con 0x55d5af6f2390 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6e03656740 10 client.16008 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6e03656740 10 client.16008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6e03656740 20 client.16008 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6e03656740 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55d5af6b0f60 0x55d5afab36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6e03656740 4 client.16008 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6e03656740 1 -- 172.21.15.98:0/1956335315 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5afa06990 con 0x55d5af6b0f60 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6dfa544640 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55d5af6b0f60 0x55d5afab36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6dddffb640 10 client.16008 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:46.351 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.342+0000 7f6dfa544640 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55d5af6b0f60 0x55d5afab36a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2277d8aaca848770 server_cookie=55854008684a3a47 in_seq=0 out_seq=0 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6dddffb640 1 -- 172.21.15.98:0/1956335315 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d5afa06990 con 0x55d5af6b0f60 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6dddffb640 10 client.16008 handle_command_reply: tid=0 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 client.16008 shutdown 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 -- 172.21.15.98:0/1956335315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 msgr2=0x55d5afa93880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa93880 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f6de402a930 tx=0x7f6de4041040 comp rx=0 tx=0).stop 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 -- 172.21.15.98:0/1956335315 shutdown_connections 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55d5af6b0f60 0x55d5afab36a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa93880 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6d16b0 0x55d5afa97240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6dddffb640 0 client.16008 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 --2- 172.21.15.98:0/1956335315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6c94d0 0x55d5afa96d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.343+0000 7f6e03656740 1 -- 172.21.15.98:0/1956335315 >> 172.21.15.98:0/1956335315 conn(0x55d5af6e64e0 msgr2=0x55d5afa92180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.344+0000 7f6e03656740 1 -- 172.21.15.98:0/1956335315 shutdown_connections 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.344+0000 7f6e03656740 1 -- 172.21.15.98:0/1956335315 wait complete. 2024-04-25T16:33:46.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.344+0000 7f6e03656740 20 client.16008 trim_cache size 0 max 16384 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6e03656740 1 Processor -- start 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6e03656740 1 -- start start 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6e03656740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6c94d0 0x55d5afaa0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6e03656740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afaa0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6e03656740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa8eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6e03656740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6df41aa720 con 0x55d5af6f2390 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6e03656740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6df41ac260 con 0x55d5af6d16b0 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6e03656740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6df41aadf0 con 0x55d5af6c94d0 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa8eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa8eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41602/0 (socket says 172.21.15.98:41602) 2024-04-25T16:33:46.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6df9d43640 1 -- 172.21.15.98:0/1600478451 learned_addr learned my addr 172.21.15.98:0/1600478451 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.379+0000 7f6df9d43640 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6c94d0 0x55d5afaa0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6dfa544640 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afaa0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6df9d43640 1 -- 172.21.15.98:0/1600478451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6c94d0 msgr2=0x55d5afaa0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6df9d43640 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6c94d0 0x55d5afaa0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6df9d43640 1 -- 172.21.15.98:0/1600478451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 msgr2=0x55d5afaa0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6df9d43640 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afaa0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6df9d43640 1 -- 172.21.15.98:0/1600478451 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5af6ca0f0 con 0x55d5af6f2390 2024-04-25T16:33:46.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6dfa544640 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afaa0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:33:46.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6df9d43640 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa8eca0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009900 tx=0x7f6de4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e2ad2d6a2821e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6ddffff640 1 -- 172.21.15.98:0/1600478451 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de4035070 con 0x55d5af6f2390 2024-04-25T16:33:46.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6ddffff640 1 -- 172.21.15.98:0/1600478451 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6de4031700 con 0x55d5af6f2390 2024-04-25T16:33:46.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.380+0000 7f6ddffff640 1 -- 172.21.15.98:0/1600478451 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de404f040 con 0x55d5af6f2390 2024-04-25T16:33:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 -- 172.21.15.98:0/1600478451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 msgr2=0x55d5afa8eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa8eca0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009900 tx=0x7f6de4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 -- 172.21.15.98:0/1600478451 shutdown_connections 2024-04-25T16:33:46.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6f2390 0x55d5afa8eca0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afaa0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 --2- 172.21.15.98:0/1600478451 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6c94d0 0x55d5afaa0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 -- 172.21.15.98:0/1600478451 >> 172.21.15.98:0/1600478451 conn(0x55d5af6e64e0 msgr2=0x55d5afa97df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 -- 172.21.15.98:0/1600478451 shutdown_connections 2024-04-25T16:33:46.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.381+0000 7f6e03656740 1 -- 172.21.15.98:0/1600478451 wait complete. 2024-04-25T16:33:46.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 1 -- start start 2024-04-25T16:33:46.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 1 Processor -- start 2024-04-25T16:33:46.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6c94d0 0x55d5afac1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afac22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 0x55d5afb19480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6df41abb90 con 0x55d5af6c94d0 2024-04-25T16:33:46.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6df41ab4c0 con 0x55d5af6d16b0 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6df41a8700 con 0x55d5af6f2390 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6e03656740 10 client.0 _ll_register_callbacks cb 0x7f6dec0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6c94d0 0x55d5afac1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6df9d43640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6c94d0 0x55d5afac1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:41614/0 (socket says 172.21.15.98:41614) 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6df9d43640 1 -- 172.21.15.98:0/215673438 learned_addr learned my addr 172.21.15.98:0/215673438 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6dfa544640 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afac22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.382+0000 7f6df9d43640 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 0x55d5afb19480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6df9d43640 1 -- 172.21.15.98:0/215673438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 msgr2=0x55d5afb19480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6df9d43640 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 0x55d5afb19480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6df9d43640 1 -- 172.21.15.98:0/215673438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 msgr2=0x55d5afac22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6df9d43640 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afac22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.397 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6df9d43640 1 -- 172.21.15.98:0/215673438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5af9b94b0 con 0x55d5af6c94d0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6df9d43640 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6c94d0 0x55d5afac1da0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6de40059d0 tx=0x7f6de404a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bd47c90273fcb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6dddffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6dddffb640 1 -- 172.21.15.98:0/215673438 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de404f040 con 0x55d5af6c94d0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6dddffb640 1 -- 172.21.15.98:0/215673438 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f6de404b630 con 0x55d5af6c94d0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6e03656740 1 -- 172.21.15.98:0/215673438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6de4031700 con 0x55d5af6c94d0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6dddffb640 1 -- 172.21.15.98:0/215673438 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6de406e050 con 0x55d5af6c94d0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6e03656740 10 client.16014 fetch_fsmap learned FSMap version 29 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6e03656740 1 -- 172.21.15.98:0/215673438 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5afb1ab00 con 0x55d5af6c94d0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.383+0000 7f6dddffb640 1 -- 172.21.15.98:0/215673438 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6de4041070 con 0x55d5af6c94d0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6dddffb640 1 -- 172.21.15.98:0/215673438 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6de404a850 con 0x55d5af6c94d0 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6e03656740 10 client.16014 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6e03656740 10 client.16014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:33:46.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6e03656740 20 client.16014 populate_metadata read hostname 'smithi098' 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6e03656740 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55d5af6863a0 0x55d5afb1d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6e03656740 4 client.16014 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6e03656740 1 -- 172.21.15.98:0/215673438 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f6de4031700 con 0x55d5af6863a0 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6dfa544640 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55d5af6863a0 0x55d5afb1d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6dddffb640 10 client.16014 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.384+0000 7f6dfa544640 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55d5af6863a0 0x55d5afb1d2b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85ef2df91bac09e7 server_cookie=fd1a7ee4b89bb2f4 in_seq=0 out_seq=0 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6dddffb640 1 -- 172.21.15.98:0/215673438 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7f6de4031700 con 0x55d5af6863a0 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6dddffb640 10 client.16014 handle_command_reply: tid=0 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 client.16014 shutdown 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 -- 172.21.15.98:0/215673438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6c94d0 msgr2=0x55d5afac1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6c94d0 0x55d5afac1da0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6de40059d0 tx=0x7f6de404a410 comp rx=0 tx=0).stop 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 -- 172.21.15.98:0/215673438 shutdown_connections 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55d5af6863a0 0x55d5afb1d2b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d5af6f2390 0x55d5afb19480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6dddffb640 0 client.16014 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d5af6d16b0 0x55d5afac22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 --2- 172.21.15.98:0/215673438 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d5af6c94d0 0x55d5afac1da0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 -- 172.21.15.98:0/215673438 >> 172.21.15.98:0/215673438 conn(0x55d5af6e64e0 msgr2=0x55d5afa8ec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 -- 172.21.15.98:0/215673438 shutdown_connections 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 1 -- 172.21.15.98:0/215673438 wait complete. 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.385+0000 7f6e03656740 20 client.16014 trim_cache size 0 max 16384 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:33:46.400 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:33:46.401 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-04-25T16:33:46.401 INFO:teuthology.orchestra.run.smithi098.stdout: "5fdb7f66-c7df-4628-9e9b-3c80356ef362": { 2024-04-25T16:33:46.401 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-04-25T16:33:46.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "5fdb7f66-c7df-4628-9e9b-3c80356ef362", 2024-04-25T16:33:46.401 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-04-25T16:33:46.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:33:46.401 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:33:46.401 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:33:46.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6de00afb70 msgr2=0x7f6de00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6de00afb70 0x7f6de00b2080 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6dec0049d0 tx=0x7f6dec004960 comp rx=0 tx=0).stop 2024-04-25T16:33:46.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4105090 msgr2=0x7f6df41a1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4105090 0x7f6df41a1630 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f6dec00ec50 tx=0x7f6dec00c6a0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/4161278093 shutdown_connections 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6de00afb70 0x7f6de00b2080 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6df4105090 0x7f6df41a1630 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6df4102240 0x7f6df419a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 --2- 172.21.15.98:0/4161278093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6df4101740 0x7f6df419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/4161278093 >> 172.21.15.98:0/4161278093 conn(0x7f6df40fd340 msgr2=0x7f6df4105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/4161278093 shutdown_connections 2024-04-25T16:33:46.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:33:46.386+0000 7f6dfc7a8640 1 -- 172.21.15.98:0/4161278093 wait complete. 2024-04-25T16:33:46.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:46 smithi098 ceph-mon[29162]: pgmap v394: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:33:46.404 INFO:tasks.cephfs.filesystem:scrub status for tag:5fdb7f66-c7df-4628-9e9b-3c80356ef362 - {'path': '/', 'tag': '5fdb7f66-c7df-4628-9e9b-3c80356ef362', 'options': 'recursive,force'} 2024-04-25T16:33:46.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:46 smithi203 ceph-mon[33115]: pgmap v394: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:33:46.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:46 smithi173 ceph-mon[32556]: pgmap v394: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:33:47.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:47 smithi203 ceph-mon[33115]: scrub queued for path: / 2024-04-25T16:33:47.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:47 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:47.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:47 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:33:47.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:47 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:47.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:47 smithi098 ceph-mon[29162]: scrub queued for path: / 2024-04-25T16:33:47.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:47 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:47.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:47 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:33:47.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:47 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:47.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:47 smithi173 ceph-mon[32556]: scrub queued for path: / 2024-04-25T16:33:47.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:47 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:47.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:47 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:33:47.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:47 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:33:48.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:48 smithi203 ceph-mon[33115]: pgmap v395: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:33:48.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:48 smithi098 ceph-mon[29162]: pgmap v395: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:33:48.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:48 smithi173 ceph-mon[32556]: pgmap v395: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:33:49.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:49 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:33:49.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:49 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:33:49.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:49 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:33:50.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: scrub summary: idle 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: pgmap v396: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:33:50.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: scrub summary: idle 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: pgmap v396: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T16:33:50.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:33:50.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:33:50.753 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: scrub summary: idle 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: pgmap v396: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:33:50.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:33:52.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:52 smithi203 ceph-mon[33115]: pgmap v397: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:33:52.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:52 smithi098 ceph-mon[29162]: pgmap v397: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:33:52.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:52 smithi173 ceph-mon[32556]: pgmap v397: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-25T16:33:54.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:54 smithi203 ceph-mon[33115]: pgmap v398: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:33:54.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:54 smithi098 ceph-mon[29162]: pgmap v398: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:33:54.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:54 smithi173 ceph-mon[32556]: pgmap v398: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 85 B/s wr, 10 op/s 2024-04-25T16:33:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:56 smithi203 ceph-mon[33115]: pgmap v399: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:56.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:56 smithi098 ceph-mon[29162]: pgmap v399: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:56.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:56 smithi173 ceph-mon[32556]: pgmap v399: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-04-25T16:33:58.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:33:58 smithi203 ceph-mon[33115]: pgmap v400: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:33:58.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:33:58 smithi098 ceph-mon[29162]: pgmap v400: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:33:58.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:33:58 smithi173 ceph-mon[32556]: pgmap v400: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 10 op/s 2024-04-25T16:34:00.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:00 smithi203 ceph-mon[33115]: pgmap v401: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T16:34:00.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:00 smithi098 ceph-mon[29162]: pgmap v401: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T16:34:00.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:00 smithi173 ceph-mon[32556]: pgmap v401: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-25T16:34:02.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:02 smithi203 ceph-mon[33115]: pgmap v402: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T16:34:02.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:02 smithi098 ceph-mon[29162]: pgmap v402: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T16:34:02.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:02 smithi173 ceph-mon[32556]: pgmap v402: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-04-25T16:34:04.008 INFO:tasks.workunit.client.0.smithi098.stderr:Note: switching to '994e2f20102a20fa9a27966c14b625cc01884df8'. 2024-04-25T16:34:04.008 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:04.008 INFO:tasks.workunit.client.0.smithi098.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr: git switch -c 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr:Or undo this operation with: 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr: git switch - 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:04.009 INFO:tasks.workunit.client.0.smithi098.stderr:HEAD is now at 994e2f20102 Merge PR #56810 into wip-pdonnell-testing-20240425.015853-debug 2024-04-25T16:34:04.015 DEBUG:teuthology.orchestra.run.smithi098:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T16:34:04.051 INFO:tasks.workunit.client.0.smithi098.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T16:34:04.053 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T16:34:04.053 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T16:34:04.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:04 smithi203 ceph-mon[33115]: pgmap v403: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:34:04.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:04 smithi098 ceph-mon[29162]: pgmap v403: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:34:04.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:04 smithi173 ceph-mon[32556]: pgmap v403: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-04-25T16:34:04.699 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T16:34:04.754 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T16:34:04.793 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T16:34:04.794 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T16:34:04.794 INFO:tasks.workunit.client.0.smithi098.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T16:34:04.832 INFO:tasks.workunit.client.0.smithi098.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T16:34:04.836 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:34:04.836 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T16:34:04.891 INFO:tasks.workunit:Running workunits matching kernel_untar_build.sh on client.0... 2024-04-25T16:34:04.892 INFO:tasks.workunit:Running workunit kernel_untar_build.sh... 2024-04-25T16:34:04.893 DEBUG:teuthology.orchestra.run.smithi098:workunit test kernel_untar_build.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh 2024-04-25T16:34:04.956 INFO:tasks.workunit.client.0.smithi098.stderr:+ wget -O linux.tar.xz http://download.ceph.com/qa/linux-6.5.11.tar.xz 2024-04-25T16:34:04.980 INFO:tasks.workunit.client.0.smithi098.stderr:--2024-04-25 16:34:04-- http://download.ceph.com/qa/linux-6.5.11.tar.xz 2024-04-25T16:34:04.981 INFO:tasks.workunit.client.0.smithi098.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-25T16:34:05.002 INFO:tasks.workunit.client.0.smithi098.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-25T16:34:05.023 INFO:tasks.workunit.client.0.smithi098.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-25T16:34:05.023 INFO:tasks.workunit.client.0.smithi098.stderr:Length: 139010916 (133M) [application/octet-stream] 2024-04-25T16:34:05.023 INFO:tasks.workunit.client.0.smithi098.stderr:Saving to: ‘linux.tar.xz’ 2024-04-25T16:34:05.026 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:05.064 INFO:tasks.workunit.client.0.smithi098.stderr: 0K .......... .......... .......... .......... .......... 0% 1.19M 1m51s 2024-04-25T16:34:05.085 INFO:tasks.workunit.client.0.smithi098.stderr: 50K .......... .......... .......... .......... .......... 0% 2.40M 83s 2024-04-25T16:34:05.086 INFO:tasks.workunit.client.0.smithi098.stderr: 100K .......... .......... .......... .......... .......... 0% 34.2M 57s 2024-04-25T16:34:05.087 INFO:tasks.workunit.client.0.smithi098.stderr: 150K .......... .......... .......... .......... .......... 0% 34.7M 43s 2024-04-25T16:34:05.107 INFO:tasks.workunit.client.0.smithi098.stderr: 200K .......... .......... .......... .......... .......... 0% 2.52M 45s 2024-04-25T16:34:05.108 INFO:tasks.workunit.client.0.smithi098.stderr: 250K .......... .......... .......... .......... .......... 0% 33.0M 38s 2024-04-25T16:34:05.110 INFO:tasks.workunit.client.0.smithi098.stderr: 300K .......... .......... .......... .......... .......... 0% 39.7M 33s 2024-04-25T16:34:05.111 INFO:tasks.workunit.client.0.smithi098.stderr: 350K .......... .......... .......... .......... .......... 0% 38.7M 30s 2024-04-25T16:34:05.127 INFO:tasks.workunit.client.0.smithi098.stderr: 400K .......... .......... .......... .......... .......... 0% 3.04M 31s 2024-04-25T16:34:05.130 INFO:tasks.workunit.client.0.smithi098.stderr: 450K .......... .......... .......... .......... .......... 0% 30.6M 28s 2024-04-25T16:34:05.130 INFO:tasks.workunit.client.0.smithi098.stderr: 500K .......... .......... .......... .......... .......... 0% 44.6M 26s 2024-04-25T16:34:05.131 INFO:tasks.workunit.client.0.smithi098.stderr: 550K .......... .......... .......... .......... .......... 0% 44.4M 24s 2024-04-25T16:34:05.132 INFO:tasks.workunit.client.0.smithi098.stderr: 600K .......... .......... .......... .......... .......... 0% 42.6M 23s 2024-04-25T16:34:05.133 INFO:tasks.workunit.client.0.smithi098.stderr: 650K .......... .......... .......... .......... .......... 0% 36.8M 21s 2024-04-25T16:34:05.134 INFO:tasks.workunit.client.0.smithi098.stderr: 700K .......... .......... .......... .......... .......... 0% 39.7M 20s 2024-04-25T16:34:05.135 INFO:tasks.workunit.client.0.smithi098.stderr: 750K .......... .......... .......... .......... .......... 0% 39.2M 19s 2024-04-25T16:34:05.137 INFO:tasks.workunit.client.0.smithi098.stderr: 800K .......... .......... .......... .......... .......... 0% 38.9M 18s 2024-04-25T16:34:05.148 INFO:tasks.workunit.client.0.smithi098.stderr: 850K .......... .......... .......... .......... .......... 0% 4.47M 19s 2024-04-25T16:34:05.149 INFO:tasks.workunit.client.0.smithi098.stderr: 900K .......... .......... .......... .......... .......... 0% 33.7M 18s 2024-04-25T16:34:05.150 INFO:tasks.workunit.client.0.smithi098.stderr: 950K .......... .......... .......... .......... .......... 0% 38.3M 17s 2024-04-25T16:34:05.152 INFO:tasks.workunit.client.0.smithi098.stderr: 1000K .......... .......... .......... .......... .......... 0% 37.1M 16s 2024-04-25T16:34:05.153 INFO:tasks.workunit.client.0.smithi098.stderr: 1050K .......... .......... .......... .......... .......... 0% 39.3M 16s 2024-04-25T16:34:05.154 INFO:tasks.workunit.client.0.smithi098.stderr: 1100K .......... .......... .......... .......... .......... 0% 32.0M 15s 2024-04-25T16:34:05.156 INFO:tasks.workunit.client.0.smithi098.stderr: 1150K .......... .......... .......... .......... .......... 0% 38.1M 15s 2024-04-25T16:34:05.157 INFO:tasks.workunit.client.0.smithi098.stderr: 1200K .......... .......... .......... .......... .......... 0% 38.2M 14s 2024-04-25T16:34:05.158 INFO:tasks.workunit.client.0.smithi098.stderr: 1250K .......... .......... .......... .......... .......... 0% 37.3M 14s 2024-04-25T16:34:05.161 INFO:tasks.workunit.client.0.smithi098.stderr: 1300K .......... .......... .......... .......... .......... 0% 32.0M 14s 2024-04-25T16:34:05.162 INFO:tasks.workunit.client.0.smithi098.stderr: 1350K .......... .......... .......... .......... .......... 1% 36.9M 13s 2024-04-25T16:34:05.163 INFO:tasks.workunit.client.0.smithi098.stderr: 1400K .......... .......... .......... .......... .......... 1% 31.2M 13s 2024-04-25T16:34:05.164 INFO:tasks.workunit.client.0.smithi098.stderr: 1450K .......... .......... .......... .......... .......... 1% 36.7M 13s 2024-04-25T16:34:05.166 INFO:tasks.workunit.client.0.smithi098.stderr: 1500K .......... .......... .......... .......... .......... 1% 31.0M 12s 2024-04-25T16:34:05.167 INFO:tasks.workunit.client.0.smithi098.stderr: 1550K .......... .......... .......... .......... .......... 1% 36.0M 12s 2024-04-25T16:34:05.168 INFO:tasks.workunit.client.0.smithi098.stderr: 1600K .......... .......... .......... .......... .......... 1% 35.1M 12s 2024-04-25T16:34:05.170 INFO:tasks.workunit.client.0.smithi098.stderr: 1650K .......... .......... .......... .......... .......... 1% 36.1M 12s 2024-04-25T16:34:05.171 INFO:tasks.workunit.client.0.smithi098.stderr: 1700K .......... .......... .......... .......... .......... 1% 31.3M 11s 2024-04-25T16:34:05.173 INFO:tasks.workunit.client.0.smithi098.stderr: 1750K .......... .......... .......... .......... .......... 1% 35.5M 11s 2024-04-25T16:34:05.174 INFO:tasks.workunit.client.0.smithi098.stderr: 1800K .......... .......... .......... .......... .......... 1% 35.5M 11s 2024-04-25T16:34:05.175 INFO:tasks.workunit.client.0.smithi098.stderr: 1850K .......... .......... .......... .......... .......... 1% 36.0M 11s 2024-04-25T16:34:05.177 INFO:tasks.workunit.client.0.smithi098.stderr: 1900K .......... .......... .......... .......... .......... 1% 30.5M 11s 2024-04-25T16:34:05.178 INFO:tasks.workunit.client.0.smithi098.stderr: 1950K .......... .......... .......... .......... .......... 1% 35.8M 10s 2024-04-25T16:34:05.180 INFO:tasks.workunit.client.0.smithi098.stderr: 2000K .......... .......... .......... .......... .......... 1% 35.3M 10s 2024-04-25T16:34:05.182 INFO:tasks.workunit.client.0.smithi098.stderr: 2050K .......... .......... .......... .......... .......... 1% 32.8M 10s 2024-04-25T16:34:05.183 INFO:tasks.workunit.client.0.smithi098.stderr: 2100K .......... .......... .......... .......... .......... 1% 29.9M 10s 2024-04-25T16:34:05.185 INFO:tasks.workunit.client.0.smithi098.stderr: 2150K .......... .......... .......... .......... .......... 1% 30.3M 10s 2024-04-25T16:34:05.186 INFO:tasks.workunit.client.0.smithi098.stderr: 2200K .......... .......... .......... .......... .......... 1% 30.0M 10s 2024-04-25T16:34:05.188 INFO:tasks.workunit.client.0.smithi098.stderr: 2250K .......... .......... .......... .......... .......... 1% 33.7M 10s 2024-04-25T16:34:05.189 INFO:tasks.workunit.client.0.smithi098.stderr: 2300K .......... .......... .......... .......... .......... 1% 30.6M 9s 2024-04-25T16:34:05.191 INFO:tasks.workunit.client.0.smithi098.stderr: 2350K .......... .......... .......... .......... .......... 1% 34.7M 9s 2024-04-25T16:34:05.192 INFO:tasks.workunit.client.0.smithi098.stderr: 2400K .......... .......... .......... .......... .......... 1% 31.2M 9s 2024-04-25T16:34:05.194 INFO:tasks.workunit.client.0.smithi098.stderr: 2450K .......... .......... .......... .......... .......... 1% 33.4M 9s 2024-04-25T16:34:05.195 INFO:tasks.workunit.client.0.smithi098.stderr: 2500K .......... .......... .......... .......... .......... 1% 29.3M 9s 2024-04-25T16:34:05.197 INFO:tasks.workunit.client.0.smithi098.stderr: 2550K .......... .......... .......... .......... .......... 1% 33.3M 9s 2024-04-25T16:34:05.198 INFO:tasks.workunit.client.0.smithi098.stderr: 2600K .......... .......... .......... .......... .......... 1% 29.2M 9s 2024-04-25T16:34:05.200 INFO:tasks.workunit.client.0.smithi098.stderr: 2650K .......... .......... .......... .......... .......... 1% 25.6M 9s 2024-04-25T16:34:05.202 INFO:tasks.workunit.client.0.smithi098.stderr: 2700K .......... .......... .......... .......... .......... 2% 33.7M 9s 2024-04-25T16:34:05.204 INFO:tasks.workunit.client.0.smithi098.stderr: 2750K .......... .......... .......... .......... .......... 2% 28.7M 9s 2024-04-25T16:34:05.205 INFO:tasks.workunit.client.0.smithi098.stderr: 2800K .......... .......... .......... .......... .......... 2% 27.7M 8s 2024-04-25T16:34:05.207 INFO:tasks.workunit.client.0.smithi098.stderr: 2850K .......... .......... .......... .......... .......... 2% 29.4M 8s 2024-04-25T16:34:05.208 INFO:tasks.workunit.client.0.smithi098.stderr: 2900K .......... .......... .......... .......... .......... 2% 34.0M 8s 2024-04-25T16:34:05.211 INFO:tasks.workunit.client.0.smithi098.stderr: 2950K .......... .......... .......... .......... .......... 2% 22.6M 8s 2024-04-25T16:34:05.212 INFO:tasks.workunit.client.0.smithi098.stderr: 3000K .......... .......... .......... .......... .......... 2% 33.7M 8s 2024-04-25T16:34:05.214 INFO:tasks.workunit.client.0.smithi098.stderr: 3050K .......... .......... .......... .......... .......... 2% 32.8M 8s 2024-04-25T16:34:05.215 INFO:tasks.workunit.client.0.smithi098.stderr: 3100K .......... .......... .......... .......... .......... 2% 28.3M 8s 2024-04-25T16:34:05.217 INFO:tasks.workunit.client.0.smithi098.stderr: 3150K .......... .......... .......... .......... .......... 2% 28.1M 8s 2024-04-25T16:34:05.218 INFO:tasks.workunit.client.0.smithi098.stderr: 3200K .......... .......... .......... .......... .......... 2% 33.2M 8s 2024-04-25T16:34:05.220 INFO:tasks.workunit.client.0.smithi098.stderr: 3250K .......... .......... .......... .......... .......... 2% 28.1M 8s 2024-04-25T16:34:05.222 INFO:tasks.workunit.client.0.smithi098.stderr: 3300K .......... .......... .......... .......... .......... 2% 32.8M 8s 2024-04-25T16:34:05.223 INFO:tasks.workunit.client.0.smithi098.stderr: 3350K .......... .......... .......... .......... .......... 2% 32.7M 8s 2024-04-25T16:34:05.225 INFO:tasks.workunit.client.0.smithi098.stderr: 3400K .......... .......... .......... .......... .......... 2% 27.3M 8s 2024-04-25T16:34:05.226 INFO:tasks.workunit.client.0.smithi098.stderr: 3450K .......... .......... .......... .......... .......... 2% 32.9M 8s 2024-04-25T16:34:05.228 INFO:tasks.workunit.client.0.smithi098.stderr: 3500K .......... .......... .......... .......... .......... 2% 32.5M 8s 2024-04-25T16:34:05.230 INFO:tasks.workunit.client.0.smithi098.stderr: 3550K .......... .......... .......... .......... .......... 2% 26.8M 8s 2024-04-25T16:34:05.232 INFO:tasks.workunit.client.0.smithi098.stderr: 3600K .......... .......... .......... .......... .......... 2% 22.2M 8s 2024-04-25T16:34:05.234 INFO:tasks.workunit.client.0.smithi098.stderr: 3650K .......... .......... .......... .......... .......... 2% 30.3M 8s 2024-04-25T16:34:05.235 INFO:tasks.workunit.client.0.smithi098.stderr: 3700K .......... .......... .......... .......... .......... 2% 30.6M 7s 2024-04-25T16:34:05.237 INFO:tasks.workunit.client.0.smithi098.stderr: 3750K .......... .......... .......... .......... .......... 2% 27.0M 7s 2024-04-25T16:34:05.239 INFO:tasks.workunit.client.0.smithi098.stderr: 3800K .......... .......... .......... .......... .......... 2% 31.6M 7s 2024-04-25T16:34:05.240 INFO:tasks.workunit.client.0.smithi098.stderr: 3850K .......... .......... .......... .......... .......... 2% 31.1M 7s 2024-04-25T16:34:05.242 INFO:tasks.workunit.client.0.smithi098.stderr: 3900K .......... .......... .......... .......... .......... 2% 26.9M 7s 2024-04-25T16:34:05.244 INFO:tasks.workunit.client.0.smithi098.stderr: 3950K .......... .......... .......... .......... .......... 2% 27.0M 7s 2024-04-25T16:34:05.245 INFO:tasks.workunit.client.0.smithi098.stderr: 4000K .......... .......... .......... .......... .......... 2% 30.5M 7s 2024-04-25T16:34:05.247 INFO:tasks.workunit.client.0.smithi098.stderr: 4050K .......... .......... .......... .......... .......... 3% 30.7M 7s 2024-04-25T16:34:05.248 INFO:tasks.workunit.client.0.smithi098.stderr: 4100K .......... .......... .......... .......... .......... 3% 34.6M 7s 2024-04-25T16:34:05.250 INFO:tasks.workunit.client.0.smithi098.stderr: 4150K .......... .......... .......... .......... .......... 3% 25.1M 7s 2024-04-25T16:34:05.252 INFO:tasks.workunit.client.0.smithi098.stderr: 4200K .......... .......... .......... .......... .......... 3% 29.1M 7s 2024-04-25T16:34:05.254 INFO:tasks.workunit.client.0.smithi098.stderr: 4250K .......... .......... .......... .......... .......... 3% 31.4M 7s 2024-04-25T16:34:05.255 INFO:tasks.workunit.client.0.smithi098.stderr: 4300K .......... .......... .......... .......... .......... 3% 34.8M 7s 2024-04-25T16:34:05.257 INFO:tasks.workunit.client.0.smithi098.stderr: 4350K .......... .......... .......... .......... .......... 3% 30.0M 7s 2024-04-25T16:34:05.258 INFO:tasks.workunit.client.0.smithi098.stderr: 4400K .......... .......... .......... .......... .......... 3% 25.6M 7s 2024-04-25T16:34:05.260 INFO:tasks.workunit.client.0.smithi098.stderr: 4450K .......... .......... .......... .......... .......... 3% 25.4M 7s 2024-04-25T16:34:05.262 INFO:tasks.workunit.client.0.smithi098.stderr: 4500K .......... .......... .......... .......... .......... 3% 31.7M 7s 2024-04-25T16:34:05.263 INFO:tasks.workunit.client.0.smithi098.stderr: 4550K .......... .......... .......... .......... .......... 3% 36.6M 7s 2024-04-25T16:34:05.265 INFO:tasks.workunit.client.0.smithi098.stderr: 4600K .......... .......... .......... .......... .......... 3% 26.7M 7s 2024-04-25T16:34:05.267 INFO:tasks.workunit.client.0.smithi098.stderr: 4650K .......... .......... .......... .......... .......... 3% 25.0M 7s 2024-04-25T16:34:05.269 INFO:tasks.workunit.client.0.smithi098.stderr: 4700K .......... .......... .......... .......... .......... 3% 30.6M 7s 2024-04-25T16:34:05.270 INFO:tasks.workunit.client.0.smithi098.stderr: 4750K .......... .......... .......... .......... .......... 3% 29.9M 7s 2024-04-25T16:34:05.271 INFO:tasks.workunit.client.0.smithi098.stderr: 4800K .......... .......... .......... .......... .......... 3% 34.7M 7s 2024-04-25T16:34:05.273 INFO:tasks.workunit.client.0.smithi098.stderr: 4850K .......... .......... .......... .......... .......... 3% 27.4M 7s 2024-04-25T16:34:05.275 INFO:tasks.workunit.client.0.smithi098.stderr: 4900K .......... .......... .......... .......... .......... 3% 26.1M 7s 2024-04-25T16:34:05.277 INFO:tasks.workunit.client.0.smithi098.stderr: 4950K .......... .......... .......... .......... .......... 3% 29.5M 7s 2024-04-25T16:34:05.279 INFO:tasks.workunit.client.0.smithi098.stderr: 5000K .......... .......... .......... .......... .......... 3% 21.0M 7s 2024-04-25T16:34:05.280 INFO:tasks.workunit.client.0.smithi098.stderr: 5050K .......... .......... .......... .......... .......... 3% 43.7M 7s 2024-04-25T16:34:05.282 INFO:tasks.workunit.client.0.smithi098.stderr: 5100K .......... .......... .......... .......... .......... 3% 35.8M 7s 2024-04-25T16:34:05.284 INFO:tasks.workunit.client.0.smithi098.stderr: 5150K .......... .......... .......... .......... .......... 3% 28.1M 7s 2024-04-25T16:34:05.285 INFO:tasks.workunit.client.0.smithi098.stderr: 5200K .......... .......... .......... .......... .......... 3% 26.7M 7s 2024-04-25T16:34:05.287 INFO:tasks.workunit.client.0.smithi098.stderr: 5250K .......... .......... .......... .......... .......... 3% 27.3M 6s 2024-04-25T16:34:05.288 INFO:tasks.workunit.client.0.smithi098.stderr: 5300K .......... .......... .......... .......... .......... 3% 39.5M 6s 2024-04-25T16:34:05.290 INFO:tasks.workunit.client.0.smithi098.stderr: 5350K .......... .......... .......... .......... .......... 3% 27.2M 6s 2024-04-25T16:34:05.292 INFO:tasks.workunit.client.0.smithi098.stderr: 5400K .......... .......... .......... .......... .......... 4% 25.8M 6s 2024-04-25T16:34:05.294 INFO:tasks.workunit.client.0.smithi098.stderr: 5450K .......... .......... .......... .......... .......... 4% 29.4M 6s 2024-04-25T16:34:05.295 INFO:tasks.workunit.client.0.smithi098.stderr: 5500K .......... .......... .......... .......... .......... 4% 28.8M 6s 2024-04-25T16:34:05.297 INFO:tasks.workunit.client.0.smithi098.stderr: 5550K .......... .......... .......... .......... .......... 4% 32.8M 6s 2024-04-25T16:34:05.299 INFO:tasks.workunit.client.0.smithi098.stderr: 5600K .......... .......... .......... .......... .......... 4% 28.1M 6s 2024-04-25T16:34:05.301 INFO:tasks.workunit.client.0.smithi098.stderr: 5650K .......... .......... .......... .......... .......... 4% 28.0M 6s 2024-04-25T16:34:05.302 INFO:tasks.workunit.client.0.smithi098.stderr: 5700K .......... .......... .......... .......... .......... 4% 26.1M 6s 2024-04-25T16:34:05.303 INFO:tasks.workunit.client.0.smithi098.stderr: 5750K .......... .......... .......... .......... .......... 4% 37.2M 6s 2024-04-25T16:34:05.305 INFO:tasks.workunit.client.0.smithi098.stderr: 5800K .......... .......... .......... .......... .......... 4% 24.9M 6s 2024-04-25T16:34:05.307 INFO:tasks.workunit.client.0.smithi098.stderr: 5850K .......... .......... .......... .......... .......... 4% 31.6M 6s 2024-04-25T16:34:05.309 INFO:tasks.workunit.client.0.smithi098.stderr: 5900K .......... .......... .......... .......... .......... 4% 27.4M 6s 2024-04-25T16:34:05.310 INFO:tasks.workunit.client.0.smithi098.stderr: 5950K .......... .......... .......... .......... .......... 4% 30.4M 6s 2024-04-25T16:34:05.312 INFO:tasks.workunit.client.0.smithi098.stderr: 6000K .......... .......... .......... .......... .......... 4% 28.5M 6s 2024-04-25T16:34:05.314 INFO:tasks.workunit.client.0.smithi098.stderr: 6050K .......... .......... .......... .......... .......... 4% 30.9M 6s 2024-04-25T16:34:05.316 INFO:tasks.workunit.client.0.smithi098.stderr: 6100K .......... .......... .......... .......... .......... 4% 27.6M 6s 2024-04-25T16:34:05.317 INFO:tasks.workunit.client.0.smithi098.stderr: 6150K .......... .......... .......... .......... .......... 4% 24.5M 6s 2024-04-25T16:34:05.319 INFO:tasks.workunit.client.0.smithi098.stderr: 6200K .......... .......... .......... .......... .......... 4% 29.8M 6s 2024-04-25T16:34:05.321 INFO:tasks.workunit.client.0.smithi098.stderr: 6250K .......... .......... .......... .......... .......... 4% 24.2M 6s 2024-04-25T16:34:05.323 INFO:tasks.workunit.client.0.smithi098.stderr: 6300K .......... .......... .......... .......... .......... 4% 27.8M 6s 2024-04-25T16:34:05.324 INFO:tasks.workunit.client.0.smithi098.stderr: 6350K .......... .......... .......... .......... .......... 4% 33.9M 6s 2024-04-25T16:34:05.326 INFO:tasks.workunit.client.0.smithi098.stderr: 6400K .......... .......... .......... .......... .......... 4% 26.2M 6s 2024-04-25T16:34:05.328 INFO:tasks.workunit.client.0.smithi098.stderr: 6450K .......... .......... .......... .......... .......... 4% 28.0M 6s 2024-04-25T16:34:05.329 INFO:tasks.workunit.client.0.smithi098.stderr: 6500K .......... .......... .......... .......... .......... 4% 31.8M 6s 2024-04-25T16:34:05.331 INFO:tasks.workunit.client.0.smithi098.stderr: 6550K .......... .......... .......... .......... .......... 4% 26.7M 6s 2024-04-25T16:34:05.333 INFO:tasks.workunit.client.0.smithi098.stderr: 6600K .......... .......... .......... .......... .......... 4% 32.3M 6s 2024-04-25T16:34:05.335 INFO:tasks.workunit.client.0.smithi098.stderr: 6650K .......... .......... .......... .......... .......... 4% 26.4M 6s 2024-04-25T16:34:05.336 INFO:tasks.workunit.client.0.smithi098.stderr: 6700K .......... .......... .......... .......... .......... 4% 31.4M 6s 2024-04-25T16:34:05.338 INFO:tasks.workunit.client.0.smithi098.stderr: 6750K .......... .......... .......... .......... .......... 5% 28.8M 6s 2024-04-25T16:34:05.340 INFO:tasks.workunit.client.0.smithi098.stderr: 6800K .......... .......... .......... .......... .......... 5% 27.0M 6s 2024-04-25T16:34:05.341 INFO:tasks.workunit.client.0.smithi098.stderr: 6850K .......... .......... .......... .......... .......... 5% 26.7M 6s 2024-04-25T16:34:05.343 INFO:tasks.workunit.client.0.smithi098.stderr: 6900K .......... .......... .......... .......... .......... 5% 29.9M 6s 2024-04-25T16:34:05.345 INFO:tasks.workunit.client.0.smithi098.stderr: 6950K .......... .......... .......... .......... .......... 5% 26.5M 6s 2024-04-25T16:34:05.347 INFO:tasks.workunit.client.0.smithi098.stderr: 7000K .......... .......... .......... .......... .......... 5% 30.4M 6s 2024-04-25T16:34:05.348 INFO:tasks.workunit.client.0.smithi098.stderr: 7050K .......... .......... .......... .......... .......... 5% 31.5M 6s 2024-04-25T16:34:05.350 INFO:tasks.workunit.client.0.smithi098.stderr: 7100K .......... .......... .......... .......... .......... 5% 27.0M 6s 2024-04-25T16:34:05.352 INFO:tasks.workunit.client.0.smithi098.stderr: 7150K .......... .......... .......... .......... .......... 5% 29.9M 6s 2024-04-25T16:34:05.354 INFO:tasks.workunit.client.0.smithi098.stderr: 7200K .......... .......... .......... .......... .......... 5% 30.1M 6s 2024-04-25T16:34:05.355 INFO:tasks.workunit.client.0.smithi098.stderr: 7250K .......... .......... .......... .......... .......... 5% 25.0M 6s 2024-04-25T16:34:05.357 INFO:tasks.workunit.client.0.smithi098.stderr: 7300K .......... .......... .......... .......... .......... 5% 29.7M 6s 2024-04-25T16:34:05.358 INFO:tasks.workunit.client.0.smithi098.stderr: 7350K .......... .......... .......... .......... .......... 5% 30.0M 6s 2024-04-25T16:34:05.360 INFO:tasks.workunit.client.0.smithi098.stderr: 7400K .......... .......... .......... .......... .......... 5% 30.7M 6s 2024-04-25T16:34:05.362 INFO:tasks.workunit.client.0.smithi098.stderr: 7450K .......... .......... .......... .......... .......... 5% 25.9M 6s 2024-04-25T16:34:05.363 INFO:tasks.workunit.client.0.smithi098.stderr: 7500K .......... .......... .......... .......... .......... 5% 29.6M 6s 2024-04-25T16:34:05.365 INFO:tasks.workunit.client.0.smithi098.stderr: 7550K .......... .......... .......... .......... .......... 5% 30.2M 6s 2024-04-25T16:34:05.367 INFO:tasks.workunit.client.0.smithi098.stderr: 7600K .......... .......... .......... .......... .......... 5% 29.0M 6s 2024-04-25T16:34:05.369 INFO:tasks.workunit.client.0.smithi098.stderr: 7650K .......... .......... .......... .......... .......... 5% 24.3M 6s 2024-04-25T16:34:05.371 INFO:tasks.workunit.client.0.smithi098.stderr: 7700K .......... .......... .......... .......... .......... 5% 26.4M 6s 2024-04-25T16:34:05.372 INFO:tasks.workunit.client.0.smithi098.stderr: 7750K .......... .......... .......... .......... .......... 5% 29.3M 6s 2024-04-25T16:34:05.374 INFO:tasks.workunit.client.0.smithi098.stderr: 7800K .......... .......... .......... .......... .......... 5% 24.9M 6s 2024-04-25T16:34:05.376 INFO:tasks.workunit.client.0.smithi098.stderr: 7850K .......... .......... .......... .......... .......... 5% 26.7M 6s 2024-04-25T16:34:05.378 INFO:tasks.workunit.client.0.smithi098.stderr: 7900K .......... .......... .......... .......... .......... 5% 31.3M 6s 2024-04-25T16:34:05.380 INFO:tasks.workunit.client.0.smithi098.stderr: 7950K .......... .......... .......... .......... .......... 5% 31.1M 6s 2024-04-25T16:34:05.381 INFO:tasks.workunit.client.0.smithi098.stderr: 8000K .......... .......... .......... .......... .......... 5% 30.5M 6s 2024-04-25T16:34:05.383 INFO:tasks.workunit.client.0.smithi098.stderr: 8050K .......... .......... .......... .......... .......... 5% 22.9M 6s 2024-04-25T16:34:05.385 INFO:tasks.workunit.client.0.smithi098.stderr: 8100K .......... .......... .......... .......... .......... 6% 25.4M 6s 2024-04-25T16:34:05.387 INFO:tasks.workunit.client.0.smithi098.stderr: 8150K .......... .......... .......... .......... .......... 6% 26.8M 6s 2024-04-25T16:34:05.388 INFO:tasks.workunit.client.0.smithi098.stderr: 8200K .......... .......... .......... .......... .......... 6% 29.7M 6s 2024-04-25T16:34:05.390 INFO:tasks.workunit.client.0.smithi098.stderr: 8250K .......... .......... .......... .......... .......... 6% 30.7M 6s 2024-04-25T16:34:05.391 INFO:tasks.workunit.client.0.smithi098.stderr: 8300K .......... .......... .......... .......... .......... 6% 36.9M 6s 2024-04-25T16:34:05.393 INFO:tasks.workunit.client.0.smithi098.stderr: 8350K .......... .......... .......... .......... .......... 6% 39.9M 6s 2024-04-25T16:34:05.394 INFO:tasks.workunit.client.0.smithi098.stderr: 8400K .......... .......... .......... .......... .......... 6% 40.0M 6s 2024-04-25T16:34:05.395 INFO:tasks.workunit.client.0.smithi098.stderr: 8450K .......... .......... .......... .......... .......... 6% 30.6M 6s 2024-04-25T16:34:05.397 INFO:tasks.workunit.client.0.smithi098.stderr: 8500K .......... .......... .......... .......... .......... 6% 33.6M 6s 2024-04-25T16:34:05.398 INFO:tasks.workunit.client.0.smithi098.stderr: 8550K .......... .......... .......... .......... .......... 6% 35.8M 6s 2024-04-25T16:34:05.400 INFO:tasks.workunit.client.0.smithi098.stderr: 8600K .......... .......... .......... .......... .......... 6% 26.2M 6s 2024-04-25T16:34:05.402 INFO:tasks.workunit.client.0.smithi098.stderr: 8650K .......... .......... .......... .......... .......... 6% 29.5M 6s 2024-04-25T16:34:05.403 INFO:tasks.workunit.client.0.smithi098.stderr: 8700K .......... .......... .......... .......... .......... 6% 28.7M 6s 2024-04-25T16:34:05.405 INFO:tasks.workunit.client.0.smithi098.stderr: 8750K .......... .......... .......... .......... .......... 6% 26.9M 6s 2024-04-25T16:34:05.407 INFO:tasks.workunit.client.0.smithi098.stderr: 8800K .......... .......... .......... .......... .......... 6% 29.4M 6s 2024-04-25T16:34:05.408 INFO:tasks.workunit.client.0.smithi098.stderr: 8850K .......... .......... .......... .......... .......... 6% 39.9M 5s 2024-04-25T16:34:05.410 INFO:tasks.workunit.client.0.smithi098.stderr: 8900K .......... .......... .......... .......... .......... 6% 24.2M 5s 2024-04-25T16:34:05.412 INFO:tasks.workunit.client.0.smithi098.stderr: 8950K .......... .......... .......... .......... .......... 6% 30.0M 5s 2024-04-25T16:34:05.414 INFO:tasks.workunit.client.0.smithi098.stderr: 9000K .......... .......... .......... .......... .......... 6% 26.9M 5s 2024-04-25T16:34:05.415 INFO:tasks.workunit.client.0.smithi098.stderr: 9050K .......... .......... .......... .......... .......... 6% 34.7M 5s 2024-04-25T16:34:05.417 INFO:tasks.workunit.client.0.smithi098.stderr: 9100K .......... .......... .......... .......... .......... 6% 27.0M 5s 2024-04-25T16:34:05.418 INFO:tasks.workunit.client.0.smithi098.stderr: 9150K .......... .......... .......... .......... .......... 6% 28.9M 5s 2024-04-25T16:34:05.420 INFO:tasks.workunit.client.0.smithi098.stderr: 9200K .......... .......... .......... .......... .......... 6% 28.3M 5s 2024-04-25T16:34:05.422 INFO:tasks.workunit.client.0.smithi098.stderr: 9250K .......... .......... .......... .......... .......... 6% 29.7M 5s 2024-04-25T16:34:05.424 INFO:tasks.workunit.client.0.smithi098.stderr: 9300K .......... .......... .......... .......... .......... 6% 25.9M 5s 2024-04-25T16:34:05.425 INFO:tasks.workunit.client.0.smithi098.stderr: 9350K .......... .......... .......... .......... .......... 6% 37.8M 5s 2024-04-25T16:34:05.427 INFO:tasks.workunit.client.0.smithi098.stderr: 9400K .......... .......... .......... .......... .......... 6% 28.0M 5s 2024-04-25T16:34:05.428 INFO:tasks.workunit.client.0.smithi098.stderr: 9450K .......... .......... .......... .......... .......... 6% 29.5M 5s 2024-04-25T16:34:05.430 INFO:tasks.workunit.client.0.smithi098.stderr: 9500K .......... .......... .......... .......... .......... 7% 30.3M 5s 2024-04-25T16:34:05.432 INFO:tasks.workunit.client.0.smithi098.stderr: 9550K .......... .......... .......... .......... .......... 7% 24.6M 5s 2024-04-25T16:34:05.434 INFO:tasks.workunit.client.0.smithi098.stderr: 9600K .......... .......... .......... .......... .......... 7% 31.3M 5s 2024-04-25T16:34:05.435 INFO:tasks.workunit.client.0.smithi098.stderr: 9650K .......... .......... .......... .......... .......... 7% 31.9M 5s 2024-04-25T16:34:05.437 INFO:tasks.workunit.client.0.smithi098.stderr: 9700K .......... .......... .......... .......... .......... 7% 27.5M 5s 2024-04-25T16:34:05.439 INFO:tasks.workunit.client.0.smithi098.stderr: 9750K .......... .......... .......... .......... .......... 7% 26.2M 5s 2024-04-25T16:34:05.440 INFO:tasks.workunit.client.0.smithi098.stderr: 9800K .......... .......... .......... .......... .......... 7% 34.9M 5s 2024-04-25T16:34:05.442 INFO:tasks.workunit.client.0.smithi098.stderr: 9850K .......... .......... .......... .......... .......... 7% 28.3M 5s 2024-04-25T16:34:05.444 INFO:tasks.workunit.client.0.smithi098.stderr: 9900K .......... .......... .......... .......... .......... 7% 27.5M 5s 2024-04-25T16:34:05.445 INFO:tasks.workunit.client.0.smithi098.stderr: 9950K .......... .......... .......... .......... .......... 7% 27.1M 5s 2024-04-25T16:34:05.447 INFO:tasks.workunit.client.0.smithi098.stderr: 10000K .......... .......... .......... .......... .......... 7% 32.5M 5s 2024-04-25T16:34:05.449 INFO:tasks.workunit.client.0.smithi098.stderr: 10050K .......... .......... .......... .......... .......... 7% 26.7M 5s 2024-04-25T16:34:05.450 INFO:tasks.workunit.client.0.smithi098.stderr: 10100K .......... .......... .......... .......... .......... 7% 35.0M 5s 2024-04-25T16:34:05.452 INFO:tasks.workunit.client.0.smithi098.stderr: 10150K .......... .......... .......... .......... .......... 7% 29.2M 5s 2024-04-25T16:34:05.454 INFO:tasks.workunit.client.0.smithi098.stderr: 10200K .......... .......... .......... .......... .......... 7% 24.1M 5s 2024-04-25T16:34:05.458 INFO:tasks.workunit.client.0.smithi098.stderr: 10250K .......... .......... .......... .......... .......... 7% 32.3M 5s 2024-04-25T16:34:05.458 INFO:tasks.workunit.client.0.smithi098.stderr: 10300K .......... .......... .......... .......... .......... 7% 25.0M 5s 2024-04-25T16:34:05.459 INFO:tasks.workunit.client.0.smithi098.stderr: 10350K .......... .......... .......... .......... .......... 7% 34.7M 5s 2024-04-25T16:34:05.460 INFO:tasks.workunit.client.0.smithi098.stderr: 10400K .......... .......... .......... .......... .......... 7% 27.6M 5s 2024-04-25T16:34:05.462 INFO:tasks.workunit.client.0.smithi098.stderr: 10450K .......... .......... .......... .......... .......... 7% 24.1M 5s 2024-04-25T16:34:05.464 INFO:tasks.workunit.client.0.smithi098.stderr: 10500K .......... .......... .......... .......... .......... 7% 29.9M 5s 2024-04-25T16:34:05.466 INFO:tasks.workunit.client.0.smithi098.stderr: 10550K .......... .......... .......... .......... .......... 7% 25.9M 5s 2024-04-25T16:34:05.468 INFO:tasks.workunit.client.0.smithi098.stderr: 10600K .......... .......... .......... .......... .......... 7% 29.9M 5s 2024-04-25T16:34:05.469 INFO:tasks.workunit.client.0.smithi098.stderr: 10650K .......... .......... .......... .......... .......... 7% 29.6M 5s 2024-04-25T16:34:05.471 INFO:tasks.workunit.client.0.smithi098.stderr: 10700K .......... .......... .......... .......... .......... 7% 24.9M 5s 2024-04-25T16:34:05.473 INFO:tasks.workunit.client.0.smithi098.stderr: 10750K .......... .......... .......... .......... .......... 7% 28.8M 5s 2024-04-25T16:34:05.475 INFO:tasks.workunit.client.0.smithi098.stderr: 10800K .......... .......... .......... .......... .......... 7% 28.8M 5s 2024-04-25T16:34:05.477 INFO:tasks.workunit.client.0.smithi098.stderr: 10850K .......... .......... .......... .......... .......... 8% 29.0M 5s 2024-04-25T16:34:05.478 INFO:tasks.workunit.client.0.smithi098.stderr: 10900K .......... .......... .......... .......... .......... 8% 23.8M 5s 2024-04-25T16:34:05.480 INFO:tasks.workunit.client.0.smithi098.stderr: 10950K .......... .......... .......... .......... .......... 8% 29.3M 5s 2024-04-25T16:34:05.482 INFO:tasks.workunit.client.0.smithi098.stderr: 11000K .......... .......... .......... .......... .......... 8% 29.1M 5s 2024-04-25T16:34:05.484 INFO:tasks.workunit.client.0.smithi098.stderr: 11050K .......... .......... .......... .......... .......... 8% 29.7M 5s 2024-04-25T16:34:05.485 INFO:tasks.workunit.client.0.smithi098.stderr: 11100K .......... .......... .......... .......... .......... 8% 24.7M 5s 2024-04-25T16:34:05.487 INFO:tasks.workunit.client.0.smithi098.stderr: 11150K .......... .......... .......... .......... .......... 8% 28.9M 5s 2024-04-25T16:34:05.489 INFO:tasks.workunit.client.0.smithi098.stderr: 11200K .......... .......... .......... .......... .......... 8% 28.7M 5s 2024-04-25T16:34:05.491 INFO:tasks.workunit.client.0.smithi098.stderr: 11250K .......... .......... .......... .......... .......... 8% 28.5M 5s 2024-04-25T16:34:05.493 INFO:tasks.workunit.client.0.smithi098.stderr: 11300K .......... .......... .......... .......... .......... 8% 24.9M 5s 2024-04-25T16:34:05.494 INFO:tasks.workunit.client.0.smithi098.stderr: 11350K .......... .......... .......... .......... .......... 8% 29.0M 5s 2024-04-25T16:34:05.496 INFO:tasks.workunit.client.0.smithi098.stderr: 11400K .......... .......... .......... .......... .......... 8% 28.5M 5s 2024-04-25T16:34:05.498 INFO:tasks.workunit.client.0.smithi098.stderr: 11450K .......... .......... .......... .......... .......... 8% 28.4M 5s 2024-04-25T16:34:05.500 INFO:tasks.workunit.client.0.smithi098.stderr: 11500K .......... .......... .......... .......... .......... 8% 24.2M 5s 2024-04-25T16:34:05.501 INFO:tasks.workunit.client.0.smithi098.stderr: 11550K .......... .......... .......... .......... .......... 8% 27.4M 5s 2024-04-25T16:34:05.503 INFO:tasks.workunit.client.0.smithi098.stderr: 11600K .......... .......... .......... .......... .......... 8% 27.4M 5s 2024-04-25T16:34:05.505 INFO:tasks.workunit.client.0.smithi098.stderr: 11650K .......... .......... .......... .......... .......... 8% 26.8M 5s 2024-04-25T16:34:05.507 INFO:tasks.workunit.client.0.smithi098.stderr: 11700K .......... .......... .......... .......... .......... 8% 22.9M 5s 2024-04-25T16:34:05.509 INFO:tasks.workunit.client.0.smithi098.stderr: 11750K .......... .......... .......... .......... .......... 8% 26.9M 5s 2024-04-25T16:34:05.511 INFO:tasks.workunit.client.0.smithi098.stderr: 11800K .......... .......... .......... .......... .......... 8% 27.0M 5s 2024-04-25T16:34:05.513 INFO:tasks.workunit.client.0.smithi098.stderr: 11850K .......... .......... .......... .......... .......... 8% 27.0M 5s 2024-04-25T16:34:05.515 INFO:tasks.workunit.client.0.smithi098.stderr: 11900K .......... .......... .......... .......... .......... 8% 23.3M 5s 2024-04-25T16:34:05.517 INFO:tasks.workunit.client.0.smithi098.stderr: 11950K .......... .......... .......... .......... .......... 8% 27.0M 5s 2024-04-25T16:34:05.518 INFO:tasks.workunit.client.0.smithi098.stderr: 12000K .......... .......... .......... .......... .......... 8% 26.5M 5s 2024-04-25T16:34:05.520 INFO:tasks.workunit.client.0.smithi098.stderr: 12050K .......... .......... .......... .......... .......... 8% 25.8M 5s 2024-04-25T16:34:05.522 INFO:tasks.workunit.client.0.smithi098.stderr: 12100K .......... .......... .......... .......... .......... 8% 22.3M 5s 2024-04-25T16:34:05.524 INFO:tasks.workunit.client.0.smithi098.stderr: 12150K .......... .......... .......... .......... .......... 8% 25.2M 5s 2024-04-25T16:34:05.526 INFO:tasks.workunit.client.0.smithi098.stderr: 12200K .......... .......... .......... .......... .......... 9% 26.5M 5s 2024-04-25T16:34:05.528 INFO:tasks.workunit.client.0.smithi098.stderr: 12250K .......... .......... .......... .......... .......... 9% 26.4M 5s 2024-04-25T16:34:05.530 INFO:tasks.workunit.client.0.smithi098.stderr: 12300K .......... .......... .......... .......... .......... 9% 25.4M 5s 2024-04-25T16:34:05.531 INFO:tasks.workunit.client.0.smithi098.stderr: 12350K .......... .......... .......... .......... .......... 9% 34.5M 5s 2024-04-25T16:34:05.533 INFO:tasks.workunit.client.0.smithi098.stderr: 12400K .......... .......... .......... .......... .......... 9% 35.2M 5s 2024-04-25T16:34:05.534 INFO:tasks.workunit.client.0.smithi098.stderr: 12450K .......... .......... .......... .......... .......... 9% 35.0M 5s 2024-04-25T16:34:05.536 INFO:tasks.workunit.client.0.smithi098.stderr: 12500K .......... .......... .......... .......... .......... 9% 30.6M 5s 2024-04-25T16:34:05.537 INFO:tasks.workunit.client.0.smithi098.stderr: 12550K .......... .......... .......... .......... .......... 9% 33.7M 5s 2024-04-25T16:34:05.538 INFO:tasks.workunit.client.0.smithi098.stderr: 12600K .......... .......... .......... .......... .......... 9% 33.9M 5s 2024-04-25T16:34:05.540 INFO:tasks.workunit.client.0.smithi098.stderr: 12650K .......... .......... .......... .......... .......... 9% 34.7M 5s 2024-04-25T16:34:05.542 INFO:tasks.workunit.client.0.smithi098.stderr: 12700K .......... .......... .......... .......... .......... 9% 30.0M 5s 2024-04-25T16:34:05.543 INFO:tasks.workunit.client.0.smithi098.stderr: 12750K .......... .......... .......... .......... .......... 9% 30.9M 5s 2024-04-25T16:34:05.545 INFO:tasks.workunit.client.0.smithi098.stderr: 12800K .......... .......... .......... .......... .......... 9% 27.7M 5s 2024-04-25T16:34:05.547 INFO:tasks.workunit.client.0.smithi098.stderr: 12850K .......... .......... .......... .......... .......... 9% 29.5M 5s 2024-04-25T16:34:05.548 INFO:tasks.workunit.client.0.smithi098.stderr: 12900K .......... .......... .......... .......... .......... 9% 29.4M 5s 2024-04-25T16:34:05.550 INFO:tasks.workunit.client.0.smithi098.stderr: 12950K .......... .......... .......... .......... .......... 9% 31.8M 5s 2024-04-25T16:34:05.552 INFO:tasks.workunit.client.0.smithi098.stderr: 13000K .......... .......... .......... .......... .......... 9% 27.9M 5s 2024-04-25T16:34:05.553 INFO:tasks.workunit.client.0.smithi098.stderr: 13050K .......... .......... .......... .......... .......... 9% 32.3M 5s 2024-04-25T16:34:05.555 INFO:tasks.workunit.client.0.smithi098.stderr: 13100K .......... .......... .......... .......... .......... 9% 29.0M 5s 2024-04-25T16:34:05.556 INFO:tasks.workunit.client.0.smithi098.stderr: 13150K .......... .......... .......... .......... .......... 9% 33.8M 5s 2024-04-25T16:34:05.558 INFO:tasks.workunit.client.0.smithi098.stderr: 13200K .......... .......... .......... .......... .......... 9% 34.2M 5s 2024-04-25T16:34:05.559 INFO:tasks.workunit.client.0.smithi098.stderr: 13250K .......... .......... .......... .......... .......... 9% 33.9M 5s 2024-04-25T16:34:05.561 INFO:tasks.workunit.client.0.smithi098.stderr: 13300K .......... .......... .......... .......... .......... 9% 29.0M 5s 2024-04-25T16:34:05.562 INFO:tasks.workunit.client.0.smithi098.stderr: 13350K .......... .......... .......... .......... .......... 9% 32.5M 5s 2024-04-25T16:34:05.564 INFO:tasks.workunit.client.0.smithi098.stderr: 13400K .......... .......... .......... .......... .......... 9% 31.9M 5s 2024-04-25T16:34:05.565 INFO:tasks.workunit.client.0.smithi098.stderr: 13450K .......... .......... .......... .......... .......... 9% 32.7M 5s 2024-04-25T16:34:05.567 INFO:tasks.workunit.client.0.smithi098.stderr: 13500K .......... .......... .......... .......... .......... 9% 28.2M 5s 2024-04-25T16:34:05.569 INFO:tasks.workunit.client.0.smithi098.stderr: 13550K .......... .......... .......... .......... .......... 10% 32.5M 5s 2024-04-25T16:34:05.570 INFO:tasks.workunit.client.0.smithi098.stderr: 13600K .......... .......... .......... .......... .......... 10% 32.1M 5s 2024-04-25T16:34:05.572 INFO:tasks.workunit.client.0.smithi098.stderr: 13650K .......... .......... .......... .......... .......... 10% 30.8M 5s 2024-04-25T16:34:05.573 INFO:tasks.workunit.client.0.smithi098.stderr: 13700K .......... .......... .......... .......... .......... 10% 27.2M 5s 2024-04-25T16:34:05.575 INFO:tasks.workunit.client.0.smithi098.stderr: 13750K .......... .......... .......... .......... .......... 10% 31.2M 5s 2024-04-25T16:34:05.577 INFO:tasks.workunit.client.0.smithi098.stderr: 13800K .......... .......... .......... .......... .......... 10% 31.7M 5s 2024-04-25T16:34:05.578 INFO:tasks.workunit.client.0.smithi098.stderr: 13850K .......... .......... .......... .......... .......... 10% 30.8M 5s 2024-04-25T16:34:05.580 INFO:tasks.workunit.client.0.smithi098.stderr: 13900K .......... .......... .......... .......... .......... 10% 26.7M 5s 2024-04-25T16:34:05.582 INFO:tasks.workunit.client.0.smithi098.stderr: 13950K .......... .......... .......... .......... .......... 10% 30.8M 5s 2024-04-25T16:34:05.583 INFO:tasks.workunit.client.0.smithi098.stderr: 14000K .......... .......... .......... .......... .......... 10% 31.4M 5s 2024-04-25T16:34:05.585 INFO:tasks.workunit.client.0.smithi098.stderr: 14050K .......... .......... .......... .......... .......... 10% 30.9M 5s 2024-04-25T16:34:05.587 INFO:tasks.workunit.client.0.smithi098.stderr: 14100K .......... .......... .......... .......... .......... 10% 26.4M 5s 2024-04-25T16:34:05.588 INFO:tasks.workunit.client.0.smithi098.stderr: 14150K .......... .......... .......... .......... .......... 10% 31.3M 5s 2024-04-25T16:34:05.590 INFO:tasks.workunit.client.0.smithi098.stderr: 14200K .......... .......... .......... .......... .......... 10% 31.8M 5s 2024-04-25T16:34:05.591 INFO:tasks.workunit.client.0.smithi098.stderr: 14250K .......... .......... .......... .......... .......... 10% 31.5M 5s 2024-04-25T16:34:05.593 INFO:tasks.workunit.client.0.smithi098.stderr: 14300K .......... .......... .......... .......... .......... 10% 26.4M 5s 2024-04-25T16:34:05.595 INFO:tasks.workunit.client.0.smithi098.stderr: 14350K .......... .......... .......... .......... .......... 10% 31.2M 5s 2024-04-25T16:34:05.596 INFO:tasks.workunit.client.0.smithi098.stderr: 14400K .......... .......... .......... .......... .......... 10% 29.7M 5s 2024-04-25T16:34:05.598 INFO:tasks.workunit.client.0.smithi098.stderr: 14450K .......... .......... .......... .......... .......... 10% 29.5M 5s 2024-04-25T16:34:05.600 INFO:tasks.workunit.client.0.smithi098.stderr: 14500K .......... .......... .......... .......... .......... 10% 25.7M 5s 2024-04-25T16:34:05.602 INFO:tasks.workunit.client.0.smithi098.stderr: 14550K .......... .......... .......... .......... .......... 10% 29.0M 5s 2024-04-25T16:34:05.603 INFO:tasks.workunit.client.0.smithi098.stderr: 14600K .......... .......... .......... .......... .......... 10% 29.8M 5s 2024-04-25T16:34:05.605 INFO:tasks.workunit.client.0.smithi098.stderr: 14650K .......... .......... .......... .......... .......... 10% 30.5M 5s 2024-04-25T16:34:05.607 INFO:tasks.workunit.client.0.smithi098.stderr: 14700K .......... .......... .......... .......... .......... 10% 22.8M 5s 2024-04-25T16:34:05.609 INFO:tasks.workunit.client.0.smithi098.stderr: 14750K .......... .......... .......... .......... .......... 10% 30.4M 5s 2024-04-25T16:34:05.610 INFO:tasks.workunit.client.0.smithi098.stderr: 14800K .......... .......... .......... .......... .......... 10% 30.6M 5s 2024-04-25T16:34:05.612 INFO:tasks.workunit.client.0.smithi098.stderr: 14850K .......... .......... .......... .......... .......... 10% 30.5M 5s 2024-04-25T16:34:05.614 INFO:tasks.workunit.client.0.smithi098.stderr: 14900K .......... .......... .......... .......... .......... 11% 26.3M 5s 2024-04-25T16:34:05.615 INFO:tasks.workunit.client.0.smithi098.stderr: 14950K .......... .......... .......... .......... .......... 11% 30.2M 5s 2024-04-25T16:34:05.617 INFO:tasks.workunit.client.0.smithi098.stderr: 15000K .......... .......... .......... .......... .......... 11% 31.5M 5s 2024-04-25T16:34:05.618 INFO:tasks.workunit.client.0.smithi098.stderr: 15050K .......... .......... .......... .......... .......... 11% 31.5M 5s 2024-04-25T16:34:05.620 INFO:tasks.workunit.client.0.smithi098.stderr: 15100K .......... .......... .......... .......... .......... 11% 26.8M 5s 2024-04-25T16:34:05.622 INFO:tasks.workunit.client.0.smithi098.stderr: 15150K .......... .......... .......... .......... .......... 11% 30.9M 5s 2024-04-25T16:34:05.623 INFO:tasks.workunit.client.0.smithi098.stderr: 15200K .......... .......... .......... .......... .......... 11% 30.1M 5s 2024-04-25T16:34:05.625 INFO:tasks.workunit.client.0.smithi098.stderr: 15250K .......... .......... .......... .......... .......... 11% 28.4M 5s 2024-04-25T16:34:05.627 INFO:tasks.workunit.client.0.smithi098.stderr: 15300K .......... .......... .......... .......... .......... 11% 24.8M 5s 2024-04-25T16:34:05.629 INFO:tasks.workunit.client.0.smithi098.stderr: 15350K .......... .......... .......... .......... .......... 11% 25.3M 5s 2024-04-25T16:34:05.631 INFO:tasks.workunit.client.0.smithi098.stderr: 15400K .......... .......... .......... .......... .......... 11% 25.8M 5s 2024-04-25T16:34:05.633 INFO:tasks.workunit.client.0.smithi098.stderr: 15450K .......... .......... .......... .......... .......... 11% 26.0M 5s 2024-04-25T16:34:05.635 INFO:tasks.workunit.client.0.smithi098.stderr: 15500K .......... .......... .......... .......... .......... 11% 22.1M 5s 2024-04-25T16:34:05.637 INFO:tasks.workunit.client.0.smithi098.stderr: 15550K .......... .......... .......... .......... .......... 11% 25.7M 5s 2024-04-25T16:34:05.638 INFO:tasks.workunit.client.0.smithi098.stderr: 15600K .......... .......... .......... .......... .......... 11% 26.5M 5s 2024-04-25T16:34:05.641 INFO:tasks.workunit.client.0.smithi098.stderr: 15650K .......... .......... .......... .......... .......... 11% 25.7M 5s 2024-04-25T16:34:05.643 INFO:tasks.workunit.client.0.smithi098.stderr: 15700K .......... .......... .......... .......... .......... 11% 20.8M 5s 2024-04-25T16:34:05.645 INFO:tasks.workunit.client.0.smithi098.stderr: 15750K .......... .......... .......... .......... .......... 11% 25.3M 5s 2024-04-25T16:34:05.647 INFO:tasks.workunit.client.0.smithi098.stderr: 15800K .......... .......... .......... .......... .......... 11% 25.4M 5s 2024-04-25T16:34:05.649 INFO:tasks.workunit.client.0.smithi098.stderr: 15850K .......... .......... .......... .......... .......... 11% 24.6M 5s 2024-04-25T16:34:05.651 INFO:tasks.workunit.client.0.smithi098.stderr: 15900K .......... .......... .......... .......... .......... 11% 21.5M 5s 2024-04-25T16:34:05.653 INFO:tasks.workunit.client.0.smithi098.stderr: 15950K .......... .......... .......... .......... .......... 11% 25.6M 5s 2024-04-25T16:34:05.655 INFO:tasks.workunit.client.0.smithi098.stderr: 16000K .......... .......... .......... .......... .......... 11% 25.0M 5s 2024-04-25T16:34:05.657 INFO:tasks.workunit.client.0.smithi098.stderr: 16050K .......... .......... .......... .......... .......... 11% 25.3M 5s 2024-04-25T16:34:05.659 INFO:tasks.workunit.client.0.smithi098.stderr: 16100K .......... .......... .......... .......... .......... 11% 22.2M 5s 2024-04-25T16:34:05.661 INFO:tasks.workunit.client.0.smithi098.stderr: 16150K .......... .......... .......... .......... .......... 11% 26.4M 5s 2024-04-25T16:34:05.663 INFO:tasks.workunit.client.0.smithi098.stderr: 16200K .......... .......... .......... .......... .......... 11% 23.9M 5s 2024-04-25T16:34:05.665 INFO:tasks.workunit.client.0.smithi098.stderr: 16250K .......... .......... .......... .......... .......... 12% 24.3M 5s 2024-04-25T16:34:05.667 INFO:tasks.workunit.client.0.smithi098.stderr: 16300K .......... .......... .......... .......... .......... 12% 22.0M 5s 2024-04-25T16:34:05.669 INFO:tasks.workunit.client.0.smithi098.stderr: 16350K .......... .......... .......... .......... .......... 12% 23.5M 5s 2024-04-25T16:34:05.671 INFO:tasks.workunit.client.0.smithi098.stderr: 16400K .......... .......... .......... .......... .......... 12% 28.0M 5s 2024-04-25T16:34:05.672 INFO:tasks.workunit.client.0.smithi098.stderr: 16450K .......... .......... .......... .......... .......... 12% 32.9M 5s 2024-04-25T16:34:05.674 INFO:tasks.workunit.client.0.smithi098.stderr: 16500K .......... .......... .......... .......... .......... 12% 29.5M 5s 2024-04-25T16:34:05.676 INFO:tasks.workunit.client.0.smithi098.stderr: 16550K .......... .......... .......... .......... .......... 12% 34.3M 5s 2024-04-25T16:34:05.677 INFO:tasks.workunit.client.0.smithi098.stderr: 16600K .......... .......... .......... .......... .......... 12% 32.8M 5s 2024-04-25T16:34:05.679 INFO:tasks.workunit.client.0.smithi098.stderr: 16650K .......... .......... .......... .......... .......... 12% 31.7M 5s 2024-04-25T16:34:05.680 INFO:tasks.workunit.client.0.smithi098.stderr: 16700K .......... .......... .......... .......... .......... 12% 25.0M 5s 2024-04-25T16:34:05.682 INFO:tasks.workunit.client.0.smithi098.stderr: 16750K .......... .......... .......... .......... .......... 12% 34.5M 5s 2024-04-25T16:34:05.683 INFO:tasks.workunit.client.0.smithi098.stderr: 16800K .......... .......... .......... .......... .......... 12% 35.5M 5s 2024-04-25T16:34:05.685 INFO:tasks.workunit.client.0.smithi098.stderr: 16850K .......... .......... .......... .......... .......... 12% 36.1M 5s 2024-04-25T16:34:05.686 INFO:tasks.workunit.client.0.smithi098.stderr: 16900K .......... .......... .......... .......... .......... 12% 30.5M 5s 2024-04-25T16:34:05.688 INFO:tasks.workunit.client.0.smithi098.stderr: 16950K .......... .......... .......... .......... .......... 12% 35.9M 5s 2024-04-25T16:34:05.689 INFO:tasks.workunit.client.0.smithi098.stderr: 17000K .......... .......... .......... .......... .......... 12% 37.9M 5s 2024-04-25T16:34:05.690 INFO:tasks.workunit.client.0.smithi098.stderr: 17050K .......... .......... .......... .......... .......... 12% 36.2M 5s 2024-04-25T16:34:05.692 INFO:tasks.workunit.client.0.smithi098.stderr: 17100K .......... .......... .......... .......... .......... 12% 32.1M 5s 2024-04-25T16:34:05.693 INFO:tasks.workunit.client.0.smithi098.stderr: 17150K .......... .......... .......... .......... .......... 12% 37.9M 5s 2024-04-25T16:34:05.694 INFO:tasks.workunit.client.0.smithi098.stderr: 17200K .......... .......... .......... .......... .......... 12% 37.6M 5s 2024-04-25T16:34:05.696 INFO:tasks.workunit.client.0.smithi098.stderr: 17250K .......... .......... .......... .......... .......... 12% 36.5M 5s 2024-04-25T16:34:05.697 INFO:tasks.workunit.client.0.smithi098.stderr: 17300K .......... .......... .......... .......... .......... 12% 31.7M 5s 2024-04-25T16:34:05.698 INFO:tasks.workunit.client.0.smithi098.stderr: 17350K .......... .......... .......... .......... .......... 12% 36.9M 5s 2024-04-25T16:34:05.700 INFO:tasks.workunit.client.0.smithi098.stderr: 17400K .......... .......... .......... .......... .......... 12% 36.8M 5s 2024-04-25T16:34:05.701 INFO:tasks.workunit.client.0.smithi098.stderr: 17450K .......... .......... .......... .......... .......... 12% 36.1M 5s 2024-04-25T16:34:05.703 INFO:tasks.workunit.client.0.smithi098.stderr: 17500K .......... .......... .......... .......... .......... 12% 28.0M 5s 2024-04-25T16:34:05.704 INFO:tasks.workunit.client.0.smithi098.stderr: 17550K .......... .......... .......... .......... .......... 12% 32.2M 5s 2024-04-25T16:34:05.706 INFO:tasks.workunit.client.0.smithi098.stderr: 17600K .......... .......... .......... .......... .......... 13% 32.2M 5s 2024-04-25T16:34:05.707 INFO:tasks.workunit.client.0.smithi098.stderr: 17650K .......... .......... .......... .......... .......... 13% 32.6M 5s 2024-04-25T16:34:05.709 INFO:tasks.workunit.client.0.smithi098.stderr: 17700K .......... .......... .......... .......... .......... 13% 26.8M 5s 2024-04-25T16:34:05.711 INFO:tasks.workunit.client.0.smithi098.stderr: 17750K .......... .......... .......... .......... .......... 13% 32.0M 5s 2024-04-25T16:34:05.712 INFO:tasks.workunit.client.0.smithi098.stderr: 17800K .......... .......... .......... .......... .......... 13% 34.0M 5s 2024-04-25T16:34:05.714 INFO:tasks.workunit.client.0.smithi098.stderr: 17850K .......... .......... .......... .......... .......... 13% 32.5M 5s 2024-04-25T16:34:05.715 INFO:tasks.workunit.client.0.smithi098.stderr: 17900K .......... .......... .......... .......... .......... 13% 28.6M 5s 2024-04-25T16:34:05.717 INFO:tasks.workunit.client.0.smithi098.stderr: 17950K .......... .......... .......... .......... .......... 13% 32.7M 5s 2024-04-25T16:34:05.718 INFO:tasks.workunit.client.0.smithi098.stderr: 18000K .......... .......... .......... .......... .......... 13% 33.3M 5s 2024-04-25T16:34:05.720 INFO:tasks.workunit.client.0.smithi098.stderr: 18050K .......... .......... .......... .......... .......... 13% 32.8M 5s 2024-04-25T16:34:05.721 INFO:tasks.workunit.client.0.smithi098.stderr: 18100K .......... .......... .......... .......... .......... 13% 27.4M 5s 2024-04-25T16:34:05.723 INFO:tasks.workunit.client.0.smithi098.stderr: 18150K .......... .......... .......... .......... .......... 13% 33.3M 5s 2024-04-25T16:34:05.724 INFO:tasks.workunit.client.0.smithi098.stderr: 18200K .......... .......... .......... .......... .......... 13% 31.9M 5s 2024-04-25T16:34:05.726 INFO:tasks.workunit.client.0.smithi098.stderr: 18250K .......... .......... .......... .......... .......... 13% 31.7M 5s 2024-04-25T16:34:05.728 INFO:tasks.workunit.client.0.smithi098.stderr: 18300K .......... .......... .......... .......... .......... 13% 26.9M 5s 2024-04-25T16:34:05.729 INFO:tasks.workunit.client.0.smithi098.stderr: 18350K .......... .......... .......... .......... .......... 13% 31.5M 5s 2024-04-25T16:34:05.731 INFO:tasks.workunit.client.0.smithi098.stderr: 18400K .......... .......... .......... .......... .......... 13% 31.6M 4s 2024-04-25T16:34:05.733 INFO:tasks.workunit.client.0.smithi098.stderr: 18450K .......... .......... .......... .......... .......... 13% 31.8M 4s 2024-04-25T16:34:05.734 INFO:tasks.workunit.client.0.smithi098.stderr: 18500K .......... .......... .......... .......... .......... 13% 26.5M 4s 2024-04-25T16:34:05.736 INFO:tasks.workunit.client.0.smithi098.stderr: 18550K .......... .......... .......... .......... .......... 13% 30.5M 4s 2024-04-25T16:34:05.738 INFO:tasks.workunit.client.0.smithi098.stderr: 18600K .......... .......... .......... .......... .......... 13% 29.8M 4s 2024-04-25T16:34:05.739 INFO:tasks.workunit.client.0.smithi098.stderr: 18650K .......... .......... .......... .......... .......... 13% 30.7M 4s 2024-04-25T16:34:05.741 INFO:tasks.workunit.client.0.smithi098.stderr: 18700K .......... .......... .......... .......... .......... 13% 25.4M 4s 2024-04-25T16:34:05.743 INFO:tasks.workunit.client.0.smithi098.stderr: 18750K .......... .......... .......... .......... .......... 13% 32.0M 4s 2024-04-25T16:34:05.744 INFO:tasks.workunit.client.0.smithi098.stderr: 18800K .......... .......... .......... .......... .......... 13% 31.8M 4s 2024-04-25T16:34:05.746 INFO:tasks.workunit.client.0.smithi098.stderr: 18850K .......... .......... .......... .......... .......... 13% 32.1M 4s 2024-04-25T16:34:05.747 INFO:tasks.workunit.client.0.smithi098.stderr: 18900K .......... .......... .......... .......... .......... 13% 27.6M 4s 2024-04-25T16:34:05.749 INFO:tasks.workunit.client.0.smithi098.stderr: 18950K .......... .......... .......... .......... .......... 13% 31.6M 4s 2024-04-25T16:34:05.751 INFO:tasks.workunit.client.0.smithi098.stderr: 19000K .......... .......... .......... .......... .......... 14% 31.1M 4s 2024-04-25T16:34:05.753 INFO:tasks.workunit.client.0.smithi098.stderr: 19050K .......... .......... .......... .......... .......... 14% 30.8M 4s 2024-04-25T16:34:05.754 INFO:tasks.workunit.client.0.smithi098.stderr: 19100K .......... .......... .......... .......... .......... 14% 26.0M 4s 2024-04-25T16:34:05.756 INFO:tasks.workunit.client.0.smithi098.stderr: 19150K .......... .......... .......... .......... .......... 14% 30.3M 4s 2024-04-25T16:34:05.757 INFO:tasks.workunit.client.0.smithi098.stderr: 19200K .......... .......... .......... .......... .......... 14% 30.0M 4s 2024-04-25T16:34:05.759 INFO:tasks.workunit.client.0.smithi098.stderr: 19250K .......... .......... .......... .......... .......... 14% 29.0M 4s 2024-04-25T16:34:05.761 INFO:tasks.workunit.client.0.smithi098.stderr: 19300K .......... .......... .......... .......... .......... 14% 26.0M 4s 2024-04-25T16:34:05.762 INFO:tasks.workunit.client.0.smithi098.stderr: 19350K .......... .......... .......... .......... .......... 14% 30.6M 4s 2024-04-25T16:34:05.764 INFO:tasks.workunit.client.0.smithi098.stderr: 19400K .......... .......... .......... .......... .......... 14% 28.2M 4s 2024-04-25T16:34:05.766 INFO:tasks.workunit.client.0.smithi098.stderr: 19450K .......... .......... .......... .......... .......... 14% 30.3M 4s 2024-04-25T16:34:05.770 INFO:tasks.workunit.client.0.smithi098.stderr: 19500K .......... .......... .......... .......... .......... 14% 13.1M 4s 2024-04-25T16:34:05.771 INFO:tasks.workunit.client.0.smithi098.stderr: 19550K .......... .......... .......... .......... .......... 14% 29.4M 4s 2024-04-25T16:34:05.773 INFO:tasks.workunit.client.0.smithi098.stderr: 19600K .......... .......... .......... .......... .......... 14% 28.4M 4s 2024-04-25T16:34:05.775 INFO:tasks.workunit.client.0.smithi098.stderr: 19650K .......... .......... .......... .......... .......... 14% 29.0M 4s 2024-04-25T16:34:05.777 INFO:tasks.workunit.client.0.smithi098.stderr: 19700K .......... .......... .......... .......... .......... 14% 26.2M 4s 2024-04-25T16:34:05.778 INFO:tasks.workunit.client.0.smithi098.stderr: 19750K .......... .......... .......... .......... .......... 14% 29.8M 4s 2024-04-25T16:34:05.780 INFO:tasks.workunit.client.0.smithi098.stderr: 19800K .......... .......... .......... .......... .......... 14% 29.8M 4s 2024-04-25T16:34:05.781 INFO:tasks.workunit.client.0.smithi098.stderr: 19850K .......... .......... .......... .......... .......... 14% 30.8M 4s 2024-04-25T16:34:05.783 INFO:tasks.workunit.client.0.smithi098.stderr: 19900K .......... .......... .......... .......... .......... 14% 24.4M 4s 2024-04-25T16:34:05.785 INFO:tasks.workunit.client.0.smithi098.stderr: 19950K .......... .......... .......... .......... .......... 14% 28.9M 4s 2024-04-25T16:34:05.787 INFO:tasks.workunit.client.0.smithi098.stderr: 20000K .......... .......... .......... .......... .......... 14% 28.0M 4s 2024-04-25T16:34:05.789 INFO:tasks.workunit.client.0.smithi098.stderr: 20050K .......... .......... .......... .......... .......... 14% 27.8M 4s 2024-04-25T16:34:05.791 INFO:tasks.workunit.client.0.smithi098.stderr: 20100K .......... .......... .......... .......... .......... 14% 24.0M 4s 2024-04-25T16:34:05.792 INFO:tasks.workunit.client.0.smithi098.stderr: 20150K .......... .......... .......... .......... .......... 14% 28.0M 4s 2024-04-25T16:34:05.794 INFO:tasks.workunit.client.0.smithi098.stderr: 20200K .......... .......... .......... .......... .......... 14% 27.8M 4s 2024-04-25T16:34:05.796 INFO:tasks.workunit.client.0.smithi098.stderr: 20250K .......... .......... .......... .......... .......... 14% 27.5M 4s 2024-04-25T16:34:05.798 INFO:tasks.workunit.client.0.smithi098.stderr: 20300K .......... .......... .......... .......... .......... 14% 23.8M 4s 2024-04-25T16:34:05.800 INFO:tasks.workunit.client.0.smithi098.stderr: 20350K .......... .......... .......... .......... .......... 15% 26.0M 4s 2024-04-25T16:34:05.802 INFO:tasks.workunit.client.0.smithi098.stderr: 20400K .......... .......... .......... .......... .......... 15% 25.3M 4s 2024-04-25T16:34:05.804 INFO:tasks.workunit.client.0.smithi098.stderr: 20450K .......... .......... .......... .......... .......... 15% 24.3M 4s 2024-04-25T16:34:05.806 INFO:tasks.workunit.client.0.smithi098.stderr: 20500K .......... .......... .......... .......... .......... 15% 28.1M 4s 2024-04-25T16:34:05.807 INFO:tasks.workunit.client.0.smithi098.stderr: 20550K .......... .......... .......... .......... .......... 15% 30.8M 4s 2024-04-25T16:34:05.810 INFO:tasks.workunit.client.0.smithi098.stderr: 20600K .......... .......... .......... .......... .......... 15% 34.7M 4s 2024-04-25T16:34:05.810 INFO:tasks.workunit.client.0.smithi098.stderr: 20650K .......... .......... .......... .......... .......... 15% 35.7M 4s 2024-04-25T16:34:05.812 INFO:tasks.workunit.client.0.smithi098.stderr: 20700K .......... .......... .......... .......... .......... 15% 30.5M 4s 2024-04-25T16:34:05.814 INFO:tasks.workunit.client.0.smithi098.stderr: 20750K .......... .......... .......... .......... .......... 15% 22.3M 4s 2024-04-25T16:34:05.816 INFO:tasks.workunit.client.0.smithi098.stderr: 20800K .......... .......... .......... .......... .......... 15% 22.1M 4s 2024-04-25T16:34:05.817 INFO:tasks.workunit.client.0.smithi098.stderr: 20850K .......... .......... .......... .......... .......... 15% 36.0M 4s 2024-04-25T16:34:05.819 INFO:tasks.workunit.client.0.smithi098.stderr: 20900K .......... .......... .......... .......... .......... 15% 28.5M 4s 2024-04-25T16:34:05.821 INFO:tasks.workunit.client.0.smithi098.stderr: 20950K .......... .......... .......... .......... .......... 15% 25.9M 4s 2024-04-25T16:34:05.822 INFO:tasks.workunit.client.0.smithi098.stderr: 21000K .......... .......... .......... .......... .......... 15% 36.8M 4s 2024-04-25T16:34:05.824 INFO:tasks.workunit.client.0.smithi098.stderr: 21050K .......... .......... .......... .......... .......... 15% 36.7M 4s 2024-04-25T16:34:05.825 INFO:tasks.workunit.client.0.smithi098.stderr: 21100K .......... .......... .......... .......... .......... 15% 32.2M 4s 2024-04-25T16:34:05.826 INFO:tasks.workunit.client.0.smithi098.stderr: 21150K .......... .......... .......... .......... .......... 15% 37.8M 4s 2024-04-25T16:34:05.828 INFO:tasks.workunit.client.0.smithi098.stderr: 21200K .......... .......... .......... .......... .......... 15% 36.3M 4s 2024-04-25T16:34:05.829 INFO:tasks.workunit.client.0.smithi098.stderr: 21250K .......... .......... .......... .......... .......... 15% 37.9M 4s 2024-04-25T16:34:05.830 INFO:tasks.workunit.client.0.smithi098.stderr: 21300K .......... .......... .......... .......... .......... 15% 31.9M 4s 2024-04-25T16:34:05.832 INFO:tasks.workunit.client.0.smithi098.stderr: 21350K .......... .......... .......... .......... .......... 15% 37.8M 4s 2024-04-25T16:34:05.833 INFO:tasks.workunit.client.0.smithi098.stderr: 21400K .......... .......... .......... .......... .......... 15% 36.0M 4s 2024-04-25T16:34:05.834 INFO:tasks.workunit.client.0.smithi098.stderr: 21450K .......... .......... .......... .......... .......... 15% 36.8M 4s 2024-04-25T16:34:05.836 INFO:tasks.workunit.client.0.smithi098.stderr: 21500K .......... .......... .......... .......... .......... 15% 32.0M 4s 2024-04-25T16:34:05.837 INFO:tasks.workunit.client.0.smithi098.stderr: 21550K .......... .......... .......... .......... .......... 15% 35.5M 4s 2024-04-25T16:34:05.839 INFO:tasks.workunit.client.0.smithi098.stderr: 21600K .......... .......... .......... .......... .......... 15% 34.8M 4s 2024-04-25T16:34:05.840 INFO:tasks.workunit.client.0.smithi098.stderr: 21650K .......... .......... .......... .......... .......... 15% 35.7M 4s 2024-04-25T16:34:05.842 INFO:tasks.workunit.client.0.smithi098.stderr: 21700K .......... .......... .......... .......... .......... 16% 30.0M 4s 2024-04-25T16:34:05.843 INFO:tasks.workunit.client.0.smithi098.stderr: 21750K .......... .......... .......... .......... .......... 16% 35.3M 4s 2024-04-25T16:34:05.845 INFO:tasks.workunit.client.0.smithi098.stderr: 21800K .......... .......... .......... .......... .......... 16% 31.6M 4s 2024-04-25T16:34:05.846 INFO:tasks.workunit.client.0.smithi098.stderr: 21850K .......... .......... .......... .......... .......... 16% 30.3M 4s 2024-04-25T16:34:05.848 INFO:tasks.workunit.client.0.smithi098.stderr: 21900K .......... .......... .......... .......... .......... 16% 26.3M 4s 2024-04-25T16:34:05.850 INFO:tasks.workunit.client.0.smithi098.stderr: 21950K .......... .......... .......... .......... .......... 16% 27.2M 4s 2024-04-25T16:34:05.852 INFO:tasks.workunit.client.0.smithi098.stderr: 22000K .......... .......... .......... .......... .......... 16% 30.3M 4s 2024-04-25T16:34:05.853 INFO:tasks.workunit.client.0.smithi098.stderr: 22050K .......... .......... .......... .......... .......... 16% 31.0M 4s 2024-04-25T16:34:05.855 INFO:tasks.workunit.client.0.smithi098.stderr: 22100K .......... .......... .......... .......... .......... 16% 26.6M 4s 2024-04-25T16:34:05.856 INFO:tasks.workunit.client.0.smithi098.stderr: 22150K .......... .......... .......... .......... .......... 16% 31.6M 4s 2024-04-25T16:34:05.858 INFO:tasks.workunit.client.0.smithi098.stderr: 22200K .......... .......... .......... .......... .......... 16% 26.7M 4s 2024-04-25T16:34:05.860 INFO:tasks.workunit.client.0.smithi098.stderr: 22250K .......... .......... .......... .......... .......... 16% 30.1M 4s 2024-04-25T16:34:05.862 INFO:tasks.workunit.client.0.smithi098.stderr: 22300K .......... .......... .......... .......... .......... 16% 30.3M 4s 2024-04-25T16:34:05.863 INFO:tasks.workunit.client.0.smithi098.stderr: 22350K .......... .......... .......... .......... .......... 16% 26.1M 4s 2024-04-25T16:34:05.865 INFO:tasks.workunit.client.0.smithi098.stderr: 22400K .......... .......... .......... .......... .......... 16% 34.7M 4s 2024-04-25T16:34:05.867 INFO:tasks.workunit.client.0.smithi098.stderr: 22450K .......... .......... .......... .......... .......... 16% 29.0M 4s 2024-04-25T16:34:05.868 INFO:tasks.workunit.client.0.smithi098.stderr: 22500K .......... .......... .......... .......... .......... 16% 33.4M 4s 2024-04-25T16:34:05.870 INFO:tasks.workunit.client.0.smithi098.stderr: 22550K .......... .......... .......... .......... .......... 16% 25.1M 4s 2024-04-25T16:34:05.871 INFO:tasks.workunit.client.0.smithi098.stderr: 22600K .......... .......... .......... .......... .......... 16% 29.1M 4s 2024-04-25T16:34:05.873 INFO:tasks.workunit.client.0.smithi098.stderr: 22650K .......... .......... .......... .......... .......... 16% 33.0M 4s 2024-04-25T16:34:05.875 INFO:tasks.workunit.client.0.smithi098.stderr: 22700K .......... .......... .......... .......... .......... 16% 25.7M 4s 2024-04-25T16:34:05.877 INFO:tasks.workunit.client.0.smithi098.stderr: 22750K .......... .......... .......... .......... .......... 16% 30.0M 4s 2024-04-25T16:34:05.879 INFO:tasks.workunit.client.0.smithi098.stderr: 22800K .......... .......... .......... .......... .......... 16% 24.1M 4s 2024-04-25T16:34:05.881 INFO:tasks.workunit.client.0.smithi098.stderr: 22850K .......... .......... .......... .......... .......... 16% 28.5M 4s 2024-04-25T16:34:05.882 INFO:tasks.workunit.client.0.smithi098.stderr: 22900K .......... .......... .......... .......... .......... 16% 28.3M 4s 2024-04-25T16:34:05.884 INFO:tasks.workunit.client.0.smithi098.stderr: 22950K .......... .......... .......... .......... .......... 16% 28.1M 4s 2024-04-25T16:34:05.886 INFO:tasks.workunit.client.0.smithi098.stderr: 23000K .......... .......... .......... .......... .......... 16% 24.6M 4s 2024-04-25T16:34:05.887 INFO:tasks.workunit.client.0.smithi098.stderr: 23050K .......... .......... .......... .......... .......... 17% 29.8M 4s 2024-04-25T16:34:05.889 INFO:tasks.workunit.client.0.smithi098.stderr: 23100K .......... .......... .......... .......... .......... 17% 32.9M 4s 2024-04-25T16:34:05.891 INFO:tasks.workunit.client.0.smithi098.stderr: 23150K .......... .......... .......... .......... .......... 17% 32.7M 4s 2024-04-25T16:34:05.976 INFO:tasks.workunit.client.0.smithi098.stderr: 23200K .......... .......... .......... .......... .......... 17% 585K 5s 2024-04-25T16:34:05.978 INFO:tasks.workunit.client.0.smithi098.stderr: 23250K .......... .......... .......... .......... .......... 17% 31.5M 5s 2024-04-25T16:34:05.979 INFO:tasks.workunit.client.0.smithi098.stderr: 23300K .......... .......... .......... .......... .......... 17% 26.3M 5s 2024-04-25T16:34:05.981 INFO:tasks.workunit.client.0.smithi098.stderr: 23350K .......... .......... .......... .......... .......... 17% 30.8M 5s 2024-04-25T16:34:05.983 INFO:tasks.workunit.client.0.smithi098.stderr: 23400K .......... .......... .......... .......... .......... 17% 30.9M 5s 2024-04-25T16:34:05.984 INFO:tasks.workunit.client.0.smithi098.stderr: 23450K .......... .......... .......... .......... .......... 17% 31.4M 5s 2024-04-25T16:34:05.986 INFO:tasks.workunit.client.0.smithi098.stderr: 23500K .......... .......... .......... .......... .......... 17% 23.7M 5s 2024-04-25T16:34:05.988 INFO:tasks.workunit.client.0.smithi098.stderr: 23550K .......... .......... .......... .......... .......... 17% 27.7M 5s 2024-04-25T16:34:05.990 INFO:tasks.workunit.client.0.smithi098.stderr: 23600K .......... .......... .......... .......... .......... 17% 29.3M 5s 2024-04-25T16:34:05.991 INFO:tasks.workunit.client.0.smithi098.stderr: 23650K .......... .......... .......... .......... .......... 17% 27.9M 5s 2024-04-25T16:34:05.993 INFO:tasks.workunit.client.0.smithi098.stderr: 23700K .......... .......... .......... .......... .......... 17% 23.5M 5s 2024-04-25T16:34:05.995 INFO:tasks.workunit.client.0.smithi098.stderr: 23750K .......... .......... .......... .......... .......... 17% 27.9M 5s 2024-04-25T16:34:05.997 INFO:tasks.workunit.client.0.smithi098.stderr: 23800K .......... .......... .......... .......... .......... 17% 24.9M 5s 2024-04-25T16:34:05.999 INFO:tasks.workunit.client.0.smithi098.stderr: 23850K .......... .......... .......... .......... .......... 17% 26.8M 5s 2024-04-25T16:34:06.001 INFO:tasks.workunit.client.0.smithi098.stderr: 23900K .......... .......... .......... .......... .......... 17% 21.8M 5s 2024-04-25T16:34:06.004 INFO:tasks.workunit.client.0.smithi098.stderr: 23950K .......... .......... .......... .......... .......... 17% 21.6M 5s 2024-04-25T16:34:06.006 INFO:tasks.workunit.client.0.smithi098.stderr: 24000K .......... .......... .......... .......... .......... 17% 22.2M 5s 2024-04-25T16:34:06.008 INFO:tasks.workunit.client.0.smithi098.stderr: 24050K .......... .......... .......... .......... .......... 17% 22.7M 5s 2024-04-25T16:34:06.011 INFO:tasks.workunit.client.0.smithi098.stderr: 24100K .......... .......... .......... .......... .......... 17% 15.9M 5s 2024-04-25T16:34:06.013 INFO:tasks.workunit.client.0.smithi098.stderr: 24150K .......... .......... .......... .......... .......... 17% 24.6M 5s 2024-04-25T16:34:06.015 INFO:tasks.workunit.client.0.smithi098.stderr: 24200K .......... .......... .......... .......... .......... 17% 26.4M 5s 2024-04-25T16:34:06.017 INFO:tasks.workunit.client.0.smithi098.stderr: 24250K .......... .......... .......... .......... .......... 17% 26.7M 5s 2024-04-25T16:34:06.019 INFO:tasks.workunit.client.0.smithi098.stderr: 24300K .......... .......... .......... .......... .......... 17% 22.5M 5s 2024-04-25T16:34:06.021 INFO:tasks.workunit.client.0.smithi098.stderr: 24350K .......... .......... .......... .......... .......... 17% 24.0M 5s 2024-04-25T16:34:06.023 INFO:tasks.workunit.client.0.smithi098.stderr: 24400K .......... .......... .......... .......... .......... 18% 24.5M 5s 2024-04-25T16:34:06.025 INFO:tasks.workunit.client.0.smithi098.stderr: 24450K .......... .......... .......... .......... .......... 18% 24.8M 5s 2024-04-25T16:34:06.027 INFO:tasks.workunit.client.0.smithi098.stderr: 24500K .......... .......... .......... .......... .......... 18% 21.5M 5s 2024-04-25T16:34:06.029 INFO:tasks.workunit.client.0.smithi098.stderr: 24550K .......... .......... .......... .......... .......... 18% 24.7M 5s 2024-04-25T16:34:06.031 INFO:tasks.workunit.client.0.smithi098.stderr: 24600K .......... .......... .......... .......... .......... 18% 24.6M 5s 2024-04-25T16:34:06.033 INFO:tasks.workunit.client.0.smithi098.stderr: 24650K .......... .......... .......... .......... .......... 18% 31.0M 5s 2024-04-25T16:34:06.034 INFO:tasks.workunit.client.0.smithi098.stderr: 24700K .......... .......... .......... .......... .......... 18% 26.8M 5s 2024-04-25T16:34:06.036 INFO:tasks.workunit.client.0.smithi098.stderr: 24750K .......... .......... .......... .......... .......... 18% 35.8M 5s 2024-04-25T16:34:06.037 INFO:tasks.workunit.client.0.smithi098.stderr: 24800K .......... .......... .......... .......... .......... 18% 37.3M 5s 2024-04-25T16:34:06.038 INFO:tasks.workunit.client.0.smithi098.stderr: 24850K .......... .......... .......... .......... .......... 18% 32.1M 5s 2024-04-25T16:34:06.040 INFO:tasks.workunit.client.0.smithi098.stderr: 24900K .......... .......... .......... .......... .......... 18% 27.2M 5s 2024-04-25T16:34:06.042 INFO:tasks.workunit.client.0.smithi098.stderr: 24950K .......... .......... .......... .......... .......... 18% 27.0M 5s 2024-04-25T16:34:06.044 INFO:tasks.workunit.client.0.smithi098.stderr: 25000K .......... .......... .......... .......... .......... 18% 24.6M 5s 2024-04-25T16:34:06.046 INFO:tasks.workunit.client.0.smithi098.stderr: 25050K .......... .......... .......... .......... .......... 18% 27.9M 5s 2024-04-25T16:34:06.048 INFO:tasks.workunit.client.0.smithi098.stderr: 25100K .......... .......... .......... .......... .......... 18% 24.9M 5s 2024-04-25T16:34:06.050 INFO:tasks.workunit.client.0.smithi098.stderr: 25150K .......... .......... .......... .......... .......... 18% 23.7M 5s 2024-04-25T16:34:06.052 INFO:tasks.workunit.client.0.smithi098.stderr: 25200K .......... .......... .......... .......... .......... 18% 28.9M 4s 2024-04-25T16:34:06.053 INFO:tasks.workunit.client.0.smithi098.stderr: 25250K .......... .......... .......... .......... .......... 18% 31.3M 4s 2024-04-25T16:34:06.055 INFO:tasks.workunit.client.0.smithi098.stderr: 25300K .......... .......... .......... .......... .......... 18% 27.5M 4s 2024-04-25T16:34:06.056 INFO:tasks.workunit.client.0.smithi098.stderr: 25350K .......... .......... .......... .......... .......... 18% 32.9M 4s 2024-04-25T16:34:06.059 INFO:tasks.workunit.client.0.smithi098.stderr: 25400K .......... .......... .......... .......... .......... 18% 32.8M 4s 2024-04-25T16:34:06.059 INFO:tasks.workunit.client.0.smithi098.stderr: 25450K .......... .......... .......... .......... .......... 18% 33.3M 4s 2024-04-25T16:34:06.061 INFO:tasks.workunit.client.0.smithi098.stderr: 25500K .......... .......... .......... .......... .......... 18% 27.9M 4s 2024-04-25T16:34:06.063 INFO:tasks.workunit.client.0.smithi098.stderr: 25550K .......... .......... .......... .......... .......... 18% 31.9M 4s 2024-04-25T16:34:06.064 INFO:tasks.workunit.client.0.smithi098.stderr: 25600K .......... .......... .......... .......... .......... 18% 31.2M 4s 2024-04-25T16:34:06.066 INFO:tasks.workunit.client.0.smithi098.stderr: 25650K .......... .......... .......... .......... .......... 18% 32.8M 4s 2024-04-25T16:34:06.067 INFO:tasks.workunit.client.0.smithi098.stderr: 25700K .......... .......... .......... .......... .......... 18% 27.7M 4s 2024-04-25T16:34:06.069 INFO:tasks.workunit.client.0.smithi098.stderr: 25750K .......... .......... .......... .......... .......... 19% 32.0M 4s 2024-04-25T16:34:06.071 INFO:tasks.workunit.client.0.smithi098.stderr: 25800K .......... .......... .......... .......... .......... 19% 29.0M 4s 2024-04-25T16:34:06.072 INFO:tasks.workunit.client.0.smithi098.stderr: 25850K .......... .......... .......... .......... .......... 19% 30.8M 4s 2024-04-25T16:34:06.074 INFO:tasks.workunit.client.0.smithi098.stderr: 25900K .......... .......... .......... .......... .......... 19% 23.2M 4s 2024-04-25T16:34:06.076 INFO:tasks.workunit.client.0.smithi098.stderr: 25950K .......... .......... .......... .......... .......... 19% 30.8M 4s 2024-04-25T16:34:06.078 INFO:tasks.workunit.client.0.smithi098.stderr: 26000K .......... .......... .......... .......... .......... 19% 32.4M 4s 2024-04-25T16:34:06.079 INFO:tasks.workunit.client.0.smithi098.stderr: 26050K .......... .......... .......... .......... .......... 19% 31.0M 4s 2024-04-25T16:34:06.081 INFO:tasks.workunit.client.0.smithi098.stderr: 26100K .......... .......... .......... .......... .......... 19% 27.1M 4s 2024-04-25T16:34:06.082 INFO:tasks.workunit.client.0.smithi098.stderr: 26150K .......... .......... .......... .......... .......... 19% 32.6M 4s 2024-04-25T16:34:06.084 INFO:tasks.workunit.client.0.smithi098.stderr: 26200K .......... .......... .......... .......... .......... 19% 29.7M 4s 2024-04-25T16:34:06.086 INFO:tasks.workunit.client.0.smithi098.stderr: 26250K .......... .......... .......... .......... .......... 19% 30.2M 4s 2024-04-25T16:34:06.087 INFO:tasks.workunit.client.0.smithi098.stderr: 26300K .......... .......... .......... .......... .......... 19% 26.8M 4s 2024-04-25T16:34:06.089 INFO:tasks.workunit.client.0.smithi098.stderr: 26350K .......... .......... .......... .......... .......... 19% 31.6M 4s 2024-04-25T16:34:06.090 INFO:tasks.workunit.client.0.smithi098.stderr: 26400K .......... .......... .......... .......... .......... 19% 32.2M 4s 2024-04-25T16:34:06.092 INFO:tasks.workunit.client.0.smithi098.stderr: 26450K .......... .......... .......... .......... .......... 19% 32.3M 4s 2024-04-25T16:34:06.094 INFO:tasks.workunit.client.0.smithi098.stderr: 26500K .......... .......... .......... .......... .......... 19% 27.3M 4s 2024-04-25T16:34:06.095 INFO:tasks.workunit.client.0.smithi098.stderr: 26550K .......... .......... .......... .......... .......... 19% 32.2M 4s 2024-04-25T16:34:06.097 INFO:tasks.workunit.client.0.smithi098.stderr: 26600K .......... .......... .......... .......... .......... 19% 31.8M 4s 2024-04-25T16:34:06.098 INFO:tasks.workunit.client.0.smithi098.stderr: 26650K .......... .......... .......... .......... .......... 19% 32.8M 4s 2024-04-25T16:34:06.100 INFO:tasks.workunit.client.0.smithi098.stderr: 26700K .......... .......... .......... .......... .......... 19% 27.6M 4s 2024-04-25T16:34:06.102 INFO:tasks.workunit.client.0.smithi098.stderr: 26750K .......... .......... .......... .......... .......... 19% 30.6M 4s 2024-04-25T16:34:06.103 INFO:tasks.workunit.client.0.smithi098.stderr: 26800K .......... .......... .......... .......... .......... 19% 32.3M 4s 2024-04-25T16:34:06.105 INFO:tasks.workunit.client.0.smithi098.stderr: 26850K .......... .......... .......... .......... .......... 19% 32.0M 4s 2024-04-25T16:34:06.106 INFO:tasks.workunit.client.0.smithi098.stderr: 26900K .......... .......... .......... .......... .......... 19% 27.9M 4s 2024-04-25T16:34:06.108 INFO:tasks.workunit.client.0.smithi098.stderr: 26950K .......... .......... .......... .......... .......... 19% 32.4M 4s 2024-04-25T16:34:06.109 INFO:tasks.workunit.client.0.smithi098.stderr: 27000K .......... .......... .......... .......... .......... 19% 32.3M 4s 2024-04-25T16:34:06.111 INFO:tasks.workunit.client.0.smithi098.stderr: 27050K .......... .......... .......... .......... .......... 19% 31.9M 4s 2024-04-25T16:34:06.113 INFO:tasks.workunit.client.0.smithi098.stderr: 27100K .......... .......... .......... .......... .......... 19% 27.4M 4s 2024-04-25T16:34:06.114 INFO:tasks.workunit.client.0.smithi098.stderr: 27150K .......... .......... .......... .......... .......... 20% 31.7M 4s 2024-04-25T16:34:06.116 INFO:tasks.workunit.client.0.smithi098.stderr: 27200K .......... .......... .......... .......... .......... 20% 31.3M 4s 2024-04-25T16:34:06.118 INFO:tasks.workunit.client.0.smithi098.stderr: 27250K .......... .......... .......... .......... .......... 20% 31.6M 4s 2024-04-25T16:34:06.120 INFO:tasks.workunit.client.0.smithi098.stderr: 27300K .......... .......... .......... .......... .......... 20% 26.1M 4s 2024-04-25T16:34:06.121 INFO:tasks.workunit.client.0.smithi098.stderr: 27350K .......... .......... .......... .......... .......... 20% 30.9M 4s 2024-04-25T16:34:06.122 INFO:tasks.workunit.client.0.smithi098.stderr: 27400K .......... .......... .......... .......... .......... 20% 30.9M 4s 2024-04-25T16:34:06.124 INFO:tasks.workunit.client.0.smithi098.stderr: 27450K .......... .......... .......... .......... .......... 20% 30.5M 4s 2024-04-25T16:34:06.126 INFO:tasks.workunit.client.0.smithi098.stderr: 27500K .......... .......... .......... .......... .......... 20% 25.0M 4s 2024-04-25T16:34:06.128 INFO:tasks.workunit.client.0.smithi098.stderr: 27550K .......... .......... .......... .......... .......... 20% 29.8M 4s 2024-04-25T16:34:06.129 INFO:tasks.workunit.client.0.smithi098.stderr: 27600K .......... .......... .......... .......... .......... 20% 25.5M 4s 2024-04-25T16:34:06.131 INFO:tasks.workunit.client.0.smithi098.stderr: 27650K .......... .......... .......... .......... .......... 20% 28.4M 4s 2024-04-25T16:34:06.133 INFO:tasks.workunit.client.0.smithi098.stderr: 27700K .......... .......... .......... .......... .......... 20% 24.9M 4s 2024-04-25T16:34:06.135 INFO:tasks.workunit.client.0.smithi098.stderr: 27750K .......... .......... .......... .......... .......... 20% 28.6M 4s 2024-04-25T16:34:06.137 INFO:tasks.workunit.client.0.smithi098.stderr: 27800K .......... .......... .......... .......... .......... 20% 30.0M 4s 2024-04-25T16:34:06.138 INFO:tasks.workunit.client.0.smithi098.stderr: 27850K .......... .......... .......... .......... .......... 20% 30.0M 4s 2024-04-25T16:34:06.140 INFO:tasks.workunit.client.0.smithi098.stderr: 27900K .......... .......... .......... .......... .......... 20% 25.8M 4s 2024-04-25T16:34:06.142 INFO:tasks.workunit.client.0.smithi098.stderr: 27950K .......... .......... .......... .......... .......... 20% 29.8M 4s 2024-04-25T16:34:06.143 INFO:tasks.workunit.client.0.smithi098.stderr: 28000K .......... .......... .......... .......... .......... 20% 30.0M 4s 2024-04-25T16:34:06.145 INFO:tasks.workunit.client.0.smithi098.stderr: 28050K .......... .......... .......... .......... .......... 20% 28.0M 4s 2024-04-25T16:34:06.147 INFO:tasks.workunit.client.0.smithi098.stderr: 28100K .......... .......... .......... .......... .......... 20% 24.7M 4s 2024-04-25T16:34:06.149 INFO:tasks.workunit.client.0.smithi098.stderr: 28150K .......... .......... .......... .......... .......... 20% 29.2M 4s 2024-04-25T16:34:06.150 INFO:tasks.workunit.client.0.smithi098.stderr: 28200K .......... .......... .......... .......... .......... 20% 29.3M 4s 2024-04-25T16:34:06.152 INFO:tasks.workunit.client.0.smithi098.stderr: 28250K .......... .......... .......... .......... .......... 20% 28.0M 4s 2024-04-25T16:34:06.155 INFO:tasks.workunit.client.0.smithi098.stderr: 28300K .......... .......... .......... .......... .......... 20% 18.6M 4s 2024-04-25T16:34:06.156 INFO:tasks.workunit.client.0.smithi098.stderr: 28350K .......... .......... .......... .......... .......... 20% 27.6M 4s 2024-04-25T16:34:06.158 INFO:tasks.workunit.client.0.smithi098.stderr: 28400K .......... .......... .......... .......... .......... 20% 28.5M 4s 2024-04-25T16:34:06.160 INFO:tasks.workunit.client.0.smithi098.stderr: 28450K .......... .......... .......... .......... .......... 20% 27.7M 4s 2024-04-25T16:34:06.162 INFO:tasks.workunit.client.0.smithi098.stderr: 28500K .......... .......... .......... .......... .......... 21% 23.3M 4s 2024-04-25T16:34:06.164 INFO:tasks.workunit.client.0.smithi098.stderr: 28550K .......... .......... .......... .......... .......... 21% 25.3M 4s 2024-04-25T16:34:06.166 INFO:tasks.workunit.client.0.smithi098.stderr: 28600K .......... .......... .......... .......... .......... 21% 28.6M 4s 2024-04-25T16:34:06.167 INFO:tasks.workunit.client.0.smithi098.stderr: 28650K .......... .......... .......... .......... .......... 21% 27.2M 4s 2024-04-25T16:34:06.169 INFO:tasks.workunit.client.0.smithi098.stderr: 28700K .......... .......... .......... .......... .......... 21% 31.2M 4s 2024-04-25T16:34:06.171 INFO:tasks.workunit.client.0.smithi098.stderr: 28750K .......... .......... .......... .......... .......... 21% 36.0M 4s 2024-04-25T16:34:06.172 INFO:tasks.workunit.client.0.smithi098.stderr: 28800K .......... .......... .......... .......... .......... 21% 37.1M 4s 2024-04-25T16:34:06.173 INFO:tasks.workunit.client.0.smithi098.stderr: 28850K .......... .......... .......... .......... .......... 21% 38.9M 4s 2024-04-25T16:34:06.175 INFO:tasks.workunit.client.0.smithi098.stderr: 28900K .......... .......... .......... .......... .......... 21% 32.0M 4s 2024-04-25T16:34:06.176 INFO:tasks.workunit.client.0.smithi098.stderr: 28950K .......... .......... .......... .......... .......... 21% 37.4M 4s 2024-04-25T16:34:06.177 INFO:tasks.workunit.client.0.smithi098.stderr: 29000K .......... .......... .......... .......... .......... 21% 38.6M 4s 2024-04-25T16:34:06.179 INFO:tasks.workunit.client.0.smithi098.stderr: 29050K .......... .......... .......... .......... .......... 21% 37.5M 4s 2024-04-25T16:34:06.180 INFO:tasks.workunit.client.0.smithi098.stderr: 29100K .......... .......... .......... .......... .......... 21% 27.6M 4s 2024-04-25T16:34:06.182 INFO:tasks.workunit.client.0.smithi098.stderr: 29150K .......... .......... .......... .......... .......... 21% 28.0M 4s 2024-04-25T16:34:06.183 INFO:tasks.workunit.client.0.smithi098.stderr: 29200K .......... .......... .......... .......... .......... 21% 37.4M 4s 2024-04-25T16:34:06.185 INFO:tasks.workunit.client.0.smithi098.stderr: 29250K .......... .......... .......... .......... .......... 21% 33.4M 4s 2024-04-25T16:34:06.187 INFO:tasks.workunit.client.0.smithi098.stderr: 29300K .......... .......... .......... .......... .......... 21% 25.4M 4s 2024-04-25T16:34:06.188 INFO:tasks.workunit.client.0.smithi098.stderr: 29350K .......... .......... .......... .......... .......... 21% 37.0M 4s 2024-04-25T16:34:06.189 INFO:tasks.workunit.client.0.smithi098.stderr: 29400K .......... .......... .......... .......... .......... 21% 37.5M 4s 2024-04-25T16:34:06.190 INFO:tasks.workunit.client.0.smithi098.stderr: 29450K .......... .......... .......... .......... .......... 21% 37.6M 4s 2024-04-25T16:34:06.192 INFO:tasks.workunit.client.0.smithi098.stderr: 29500K .......... .......... .......... .......... .......... 21% 32.0M 4s 2024-04-25T16:34:06.193 INFO:tasks.workunit.client.0.smithi098.stderr: 29550K .......... .......... .......... .......... .......... 21% 37.2M 4s 2024-04-25T16:34:06.195 INFO:tasks.workunit.client.0.smithi098.stderr: 29600K .......... .......... .......... .......... .......... 21% 36.5M 4s 2024-04-25T16:34:06.196 INFO:tasks.workunit.client.0.smithi098.stderr: 29650K .......... .......... .......... .......... .......... 21% 36.7M 4s 2024-04-25T16:34:06.198 INFO:tasks.workunit.client.0.smithi098.stderr: 29700K .......... .......... .......... .......... .......... 21% 31.6M 4s 2024-04-25T16:34:06.199 INFO:tasks.workunit.client.0.smithi098.stderr: 29750K .......... .......... .......... .......... .......... 21% 36.4M 4s 2024-04-25T16:34:06.200 INFO:tasks.workunit.client.0.smithi098.stderr: 29800K .......... .......... .......... .......... .......... 21% 36.3M 4s 2024-04-25T16:34:06.202 INFO:tasks.workunit.client.0.smithi098.stderr: 29850K .......... .......... .......... .......... .......... 22% 36.0M 4s 2024-04-25T16:34:06.203 INFO:tasks.workunit.client.0.smithi098.stderr: 29900K .......... .......... .......... .......... .......... 22% 32.1M 4s 2024-04-25T16:34:06.204 INFO:tasks.workunit.client.0.smithi098.stderr: 29950K .......... .......... .......... .......... .......... 22% 36.8M 4s 2024-04-25T16:34:06.206 INFO:tasks.workunit.client.0.smithi098.stderr: 30000K .......... .......... .......... .......... .......... 22% 36.8M 4s 2024-04-25T16:34:06.207 INFO:tasks.workunit.client.0.smithi098.stderr: 30050K .......... .......... .......... .......... .......... 22% 37.5M 4s 2024-04-25T16:34:06.209 INFO:tasks.workunit.client.0.smithi098.stderr: 30100K .......... .......... .......... .......... .......... 22% 31.6M 4s 2024-04-25T16:34:06.210 INFO:tasks.workunit.client.0.smithi098.stderr: 30150K .......... .......... .......... .......... .......... 22% 35.8M 4s 2024-04-25T16:34:06.213 INFO:tasks.workunit.client.0.smithi098.stderr: 30200K .......... .......... .......... .......... .......... 22% 34.5M 4s 2024-04-25T16:34:06.213 INFO:tasks.workunit.client.0.smithi098.stderr: 30250K .......... .......... .......... .......... .......... 22% 33.8M 4s 2024-04-25T16:34:06.215 INFO:tasks.workunit.client.0.smithi098.stderr: 30300K .......... .......... .......... .......... .......... 22% 27.1M 4s 2024-04-25T16:34:06.216 INFO:tasks.workunit.client.0.smithi098.stderr: 30350K .......... .......... .......... .......... .......... 22% 34.5M 4s 2024-04-25T16:34:06.218 INFO:tasks.workunit.client.0.smithi098.stderr: 30400K .......... .......... .......... .......... .......... 22% 34.4M 4s 2024-04-25T16:34:06.219 INFO:tasks.workunit.client.0.smithi098.stderr: 30450K .......... .......... .......... .......... .......... 22% 32.5M 4s 2024-04-25T16:34:06.221 INFO:tasks.workunit.client.0.smithi098.stderr: 30500K .......... .......... .......... .......... .......... 22% 28.8M 4s 2024-04-25T16:34:06.222 INFO:tasks.workunit.client.0.smithi098.stderr: 30550K .......... .......... .......... .......... .......... 22% 33.5M 4s 2024-04-25T16:34:06.224 INFO:tasks.workunit.client.0.smithi098.stderr: 30600K .......... .......... .......... .......... .......... 22% 33.1M 4s 2024-04-25T16:34:06.225 INFO:tasks.workunit.client.0.smithi098.stderr: 30650K .......... .......... .......... .......... .......... 22% 33.2M 4s 2024-04-25T16:34:06.227 INFO:tasks.workunit.client.0.smithi098.stderr: 30700K .......... .......... .......... .......... .......... 22% 26.5M 4s 2024-04-25T16:34:06.228 INFO:tasks.workunit.client.0.smithi098.stderr: 30750K .......... .......... .......... .......... .......... 22% 32.0M 4s 2024-04-25T16:34:06.230 INFO:tasks.workunit.client.0.smithi098.stderr: 30800K .......... .......... .......... .......... .......... 22% 32.2M 4s 2024-04-25T16:34:06.232 INFO:tasks.workunit.client.0.smithi098.stderr: 30850K .......... .......... .......... .......... .......... 22% 31.9M 4s 2024-04-25T16:34:06.233 INFO:tasks.workunit.client.0.smithi098.stderr: 30900K .......... .......... .......... .......... .......... 22% 27.6M 4s 2024-04-25T16:34:06.235 INFO:tasks.workunit.client.0.smithi098.stderr: 30950K .......... .......... .......... .......... .......... 22% 30.8M 4s 2024-04-25T16:34:06.237 INFO:tasks.workunit.client.0.smithi098.stderr: 31000K .......... .......... .......... .......... .......... 22% 29.5M 4s 2024-04-25T16:34:06.241 INFO:tasks.workunit.client.0.smithi098.stderr: 31050K .......... .......... .......... .......... .......... 22% 31.5M 4s 2024-04-25T16:34:06.241 INFO:tasks.workunit.client.0.smithi098.stderr: 31100K .......... .......... .......... .......... .......... 22% 26.2M 4s 2024-04-25T16:34:06.242 INFO:tasks.workunit.client.0.smithi098.stderr: 31150K .......... .......... .......... .......... .......... 22% 28.5M 4s 2024-04-25T16:34:06.243 INFO:tasks.workunit.client.0.smithi098.stderr: 31200K .......... .......... .......... .......... .......... 23% 28.0M 4s 2024-04-25T16:34:06.245 INFO:tasks.workunit.client.0.smithi098.stderr: 31250K .......... .......... .......... .......... .......... 23% 28.5M 4s 2024-04-25T16:34:06.247 INFO:tasks.workunit.client.0.smithi098.stderr: 31300K .......... .......... .......... .......... .......... 23% 24.5M 4s 2024-04-25T16:34:06.249 INFO:tasks.workunit.client.0.smithi098.stderr: 31350K .......... .......... .......... .......... .......... 23% 27.4M 4s 2024-04-25T16:34:06.251 INFO:tasks.workunit.client.0.smithi098.stderr: 31400K .......... .......... .......... .......... .......... 23% 28.6M 4s 2024-04-25T16:34:06.253 INFO:tasks.workunit.client.0.smithi098.stderr: 31450K .......... .......... .......... .......... .......... 23% 28.1M 4s 2024-04-25T16:34:06.254 INFO:tasks.workunit.client.0.smithi098.stderr: 31500K .......... .......... .......... .......... .......... 23% 23.9M 4s 2024-04-25T16:34:06.256 INFO:tasks.workunit.client.0.smithi098.stderr: 31550K .......... .......... .......... .......... .......... 23% 28.1M 4s 2024-04-25T16:34:06.258 INFO:tasks.workunit.client.0.smithi098.stderr: 31600K .......... .......... .......... .......... .......... 23% 30.1M 4s 2024-04-25T16:34:06.259 INFO:tasks.workunit.client.0.smithi098.stderr: 31650K .......... .......... .......... .......... .......... 23% 30.5M 4s 2024-04-25T16:34:06.261 INFO:tasks.workunit.client.0.smithi098.stderr: 31700K .......... .......... .......... .......... .......... 23% 25.7M 4s 2024-04-25T16:34:06.263 INFO:tasks.workunit.client.0.smithi098.stderr: 31750K .......... .......... .......... .......... .......... 23% 29.6M 4s 2024-04-25T16:34:06.265 INFO:tasks.workunit.client.0.smithi098.stderr: 31800K .......... .......... .......... .......... .......... 23% 30.8M 4s 2024-04-25T16:34:06.266 INFO:tasks.workunit.client.0.smithi098.stderr: 31850K .......... .......... .......... .......... .......... 23% 29.7M 4s 2024-04-25T16:34:06.268 INFO:tasks.workunit.client.0.smithi098.stderr: 31900K .......... .......... .......... .......... .......... 23% 25.8M 4s 2024-04-25T16:34:06.270 INFO:tasks.workunit.client.0.smithi098.stderr: 31950K .......... .......... .......... .......... .......... 23% 30.2M 4s 2024-04-25T16:34:06.271 INFO:tasks.workunit.client.0.smithi098.stderr: 32000K .......... .......... .......... .......... .......... 23% 29.7M 4s 2024-04-25T16:34:06.273 INFO:tasks.workunit.client.0.smithi098.stderr: 32050K .......... .......... .......... .......... .......... 23% 28.9M 4s 2024-04-25T16:34:06.275 INFO:tasks.workunit.client.0.smithi098.stderr: 32100K .......... .......... .......... .......... .......... 23% 24.9M 4s 2024-04-25T16:34:06.277 INFO:tasks.workunit.client.0.smithi098.stderr: 32150K .......... .......... .......... .......... .......... 23% 28.5M 4s 2024-04-25T16:34:06.278 INFO:tasks.workunit.client.0.smithi098.stderr: 32200K .......... .......... .......... .......... .......... 23% 28.3M 4s 2024-04-25T16:34:06.280 INFO:tasks.workunit.client.0.smithi098.stderr: 32250K .......... .......... .......... .......... .......... 23% 28.3M 4s 2024-04-25T16:34:06.282 INFO:tasks.workunit.client.0.smithi098.stderr: 32300K .......... .......... .......... .......... .......... 23% 24.2M 4s 2024-04-25T16:34:06.284 INFO:tasks.workunit.client.0.smithi098.stderr: 32350K .......... .......... .......... .......... .......... 23% 28.3M 4s 2024-04-25T16:34:06.286 INFO:tasks.workunit.client.0.smithi098.stderr: 32400K .......... .......... .......... .......... .......... 23% 29.2M 4s 2024-04-25T16:34:06.287 INFO:tasks.workunit.client.0.smithi098.stderr: 32450K .......... .......... .......... .......... .......... 23% 28.3M 4s 2024-04-25T16:34:06.289 INFO:tasks.workunit.client.0.smithi098.stderr: 32500K .......... .......... .......... .......... .......... 23% 24.8M 4s 2024-04-25T16:34:06.291 INFO:tasks.workunit.client.0.smithi098.stderr: 32550K .......... .......... .......... .......... .......... 24% 28.5M 4s 2024-04-25T16:34:06.293 INFO:tasks.workunit.client.0.smithi098.stderr: 32600K .......... .......... .......... .......... .......... 24% 28.0M 4s 2024-04-25T16:34:06.295 INFO:tasks.workunit.client.0.smithi098.stderr: 32650K .......... .......... .......... .......... .......... 24% 27.6M 4s 2024-04-25T16:34:06.297 INFO:tasks.workunit.client.0.smithi098.stderr: 32700K .......... .......... .......... .......... .......... 24% 22.9M 4s 2024-04-25T16:34:06.298 INFO:tasks.workunit.client.0.smithi098.stderr: 32750K .......... .......... .......... .......... .......... 24% 25.9M 4s 2024-04-25T16:34:06.300 INFO:tasks.workunit.client.0.smithi098.stderr: 32800K .......... .......... .......... .......... .......... 24% 34.1M 4s 2024-04-25T16:34:06.301 INFO:tasks.workunit.client.0.smithi098.stderr: 32850K .......... .......... .......... .......... .......... 24% 35.5M 4s 2024-04-25T16:34:06.302 INFO:tasks.workunit.client.0.smithi098.stderr: 32900K .......... .......... .......... .......... .......... 24% 33.5M 4s 2024-04-25T16:34:06.304 INFO:tasks.workunit.client.0.smithi098.stderr: 32950K .......... .......... .......... .......... .......... 24% 37.8M 4s 2024-04-25T16:34:06.305 INFO:tasks.workunit.client.0.smithi098.stderr: 33000K .......... .......... .......... .......... .......... 24% 38.2M 4s 2024-04-25T16:34:06.307 INFO:tasks.workunit.client.0.smithi098.stderr: 33050K .......... .......... .......... .......... .......... 24% 38.1M 4s 2024-04-25T16:34:06.308 INFO:tasks.workunit.client.0.smithi098.stderr: 33100K .......... .......... .......... .......... .......... 24% 32.7M 4s 2024-04-25T16:34:06.310 INFO:tasks.workunit.client.0.smithi098.stderr: 33150K .......... .......... .......... .......... .......... 24% 37.7M 4s 2024-04-25T16:34:06.311 INFO:tasks.workunit.client.0.smithi098.stderr: 33200K .......... .......... .......... .......... .......... 24% 36.4M 4s 2024-04-25T16:34:06.312 INFO:tasks.workunit.client.0.smithi098.stderr: 33250K .......... .......... .......... .......... .......... 24% 37.9M 4s 2024-04-25T16:34:06.313 INFO:tasks.workunit.client.0.smithi098.stderr: 33300K .......... .......... .......... .......... .......... 24% 31.7M 4s 2024-04-25T16:34:06.315 INFO:tasks.workunit.client.0.smithi098.stderr: 33350K .......... .......... .......... .......... .......... 24% 36.8M 4s 2024-04-25T16:34:06.316 INFO:tasks.workunit.client.0.smithi098.stderr: 33400K .......... .......... .......... .......... .......... 24% 32.7M 4s 2024-04-25T16:34:06.318 INFO:tasks.workunit.client.0.smithi098.stderr: 33450K .......... .......... .......... .......... .......... 24% 32.2M 4s 2024-04-25T16:34:06.320 INFO:tasks.workunit.client.0.smithi098.stderr: 33500K .......... .......... .......... .......... .......... 24% 24.6M 4s 2024-04-25T16:34:06.322 INFO:tasks.workunit.client.0.smithi098.stderr: 33550K .......... .......... .......... .......... .......... 24% 32.9M 4s 2024-04-25T16:34:06.323 INFO:tasks.workunit.client.0.smithi098.stderr: 33600K .......... .......... .......... .......... .......... 24% 33.3M 4s 2024-04-25T16:34:06.324 INFO:tasks.workunit.client.0.smithi098.stderr: 33650K .......... .......... .......... .......... .......... 24% 33.3M 4s 2024-04-25T16:34:06.326 INFO:tasks.workunit.client.0.smithi098.stderr: 33700K .......... .......... .......... .......... .......... 24% 25.9M 4s 2024-04-25T16:34:06.328 INFO:tasks.workunit.client.0.smithi098.stderr: 33750K .......... .......... .......... .......... .......... 24% 32.4M 4s 2024-04-25T16:34:06.329 INFO:tasks.workunit.client.0.smithi098.stderr: 33800K .......... .......... .......... .......... .......... 24% 33.3M 4s 2024-04-25T16:34:06.331 INFO:tasks.workunit.client.0.smithi098.stderr: 33850K .......... .......... .......... .......... .......... 24% 32.5M 4s 2024-04-25T16:34:06.333 INFO:tasks.workunit.client.0.smithi098.stderr: 33900K .......... .......... .......... .......... .......... 25% 28.5M 4s 2024-04-25T16:34:06.334 INFO:tasks.workunit.client.0.smithi098.stderr: 33950K .......... .......... .......... .......... .......... 25% 33.8M 4s 2024-04-25T16:34:06.335 INFO:tasks.workunit.client.0.smithi098.stderr: 34000K .......... .......... .......... .......... .......... 25% 34.8M 4s 2024-04-25T16:34:06.337 INFO:tasks.workunit.client.0.smithi098.stderr: 34050K .......... .......... .......... .......... .......... 25% 31.7M 4s 2024-04-25T16:34:06.339 INFO:tasks.workunit.client.0.smithi098.stderr: 34100K .......... .......... .......... .......... .......... 25% 24.5M 4s 2024-04-25T16:34:06.341 INFO:tasks.workunit.client.0.smithi098.stderr: 34150K .......... .......... .......... .......... .......... 25% 25.6M 4s 2024-04-25T16:34:06.342 INFO:tasks.workunit.client.0.smithi098.stderr: 34200K .......... .......... .......... .......... .......... 25% 31.9M 4s 2024-04-25T16:34:06.344 INFO:tasks.workunit.client.0.smithi098.stderr: 34250K .......... .......... .......... .......... .......... 25% 33.0M 4s 2024-04-25T16:34:06.345 INFO:tasks.workunit.client.0.smithi098.stderr: 34300K .......... .......... .......... .......... .......... 25% 27.8M 4s 2024-04-25T16:34:06.347 INFO:tasks.workunit.client.0.smithi098.stderr: 34350K .......... .......... .......... .......... .......... 25% 32.2M 4s 2024-04-25T16:34:06.348 INFO:tasks.workunit.client.0.smithi098.stderr: 34400K .......... .......... .......... .......... .......... 25% 33.5M 4s 2024-04-25T16:34:06.350 INFO:tasks.workunit.client.0.smithi098.stderr: 34450K .......... .......... .......... .......... .......... 25% 32.9M 4s 2024-04-25T16:34:06.352 INFO:tasks.workunit.client.0.smithi098.stderr: 34500K .......... .......... .......... .......... .......... 25% 28.3M 4s 2024-04-25T16:34:06.353 INFO:tasks.workunit.client.0.smithi098.stderr: 34550K .......... .......... .......... .......... .......... 25% 33.5M 4s 2024-04-25T16:34:06.355 INFO:tasks.workunit.client.0.smithi098.stderr: 34600K .......... .......... .......... .......... .......... 25% 32.8M 4s 2024-04-25T16:34:06.356 INFO:tasks.workunit.client.0.smithi098.stderr: 34650K .......... .......... .......... .......... .......... 25% 32.0M 4s 2024-04-25T16:34:06.358 INFO:tasks.workunit.client.0.smithi098.stderr: 34700K .......... .......... .......... .......... .......... 25% 27.2M 4s 2024-04-25T16:34:06.359 INFO:tasks.workunit.client.0.smithi098.stderr: 34750K .......... .......... .......... .......... .......... 25% 35.4M 4s 2024-04-25T16:34:06.361 INFO:tasks.workunit.client.0.smithi098.stderr: 34800K .......... .......... .......... .......... .......... 25% 34.8M 4s 2024-04-25T16:34:06.363 INFO:tasks.workunit.client.0.smithi098.stderr: 34850K .......... .......... .......... .......... .......... 25% 34.5M 4s 2024-04-25T16:34:06.364 INFO:tasks.workunit.client.0.smithi098.stderr: 34900K .......... .......... .......... .......... .......... 25% 29.9M 4s 2024-04-25T16:34:06.365 INFO:tasks.workunit.client.0.smithi098.stderr: 34950K .......... .......... .......... .......... .......... 25% 28.7M 4s 2024-04-25T16:34:06.367 INFO:tasks.workunit.client.0.smithi098.stderr: 35000K .......... .......... .......... .......... .......... 25% 28.8M 4s 2024-04-25T16:34:06.369 INFO:tasks.workunit.client.0.smithi098.stderr: 35050K .......... .......... .......... .......... .......... 25% 29.3M 4s 2024-04-25T16:34:06.371 INFO:tasks.workunit.client.0.smithi098.stderr: 35100K .......... .......... .......... .......... .......... 25% 23.9M 4s 2024-04-25T16:34:06.373 INFO:tasks.workunit.client.0.smithi098.stderr: 35150K .......... .......... .......... .......... .......... 25% 28.2M 4s 2024-04-25T16:34:06.374 INFO:tasks.workunit.client.0.smithi098.stderr: 35200K .......... .......... .......... .......... .......... 25% 32.1M 4s 2024-04-25T16:34:06.376 INFO:tasks.workunit.client.0.smithi098.stderr: 35250K .......... .......... .......... .......... .......... 26% 31.7M 4s 2024-04-25T16:34:06.377 INFO:tasks.workunit.client.0.smithi098.stderr: 35300K .......... .......... .......... .......... .......... 26% 26.8M 4s 2024-04-25T16:34:06.379 INFO:tasks.workunit.client.0.smithi098.stderr: 35350K .......... .......... .......... .......... .......... 26% 30.8M 4s 2024-04-25T16:34:06.380 INFO:tasks.workunit.client.0.smithi098.stderr: 35400K .......... .......... .......... .......... .......... 26% 30.6M 4s 2024-04-25T16:34:06.382 INFO:tasks.workunit.client.0.smithi098.stderr: 35450K .......... .......... .......... .......... .......... 26% 30.4M 4s 2024-04-25T16:34:06.384 INFO:tasks.workunit.client.0.smithi098.stderr: 35500K .......... .......... .......... .......... .......... 26% 24.0M 4s 2024-04-25T16:34:06.386 INFO:tasks.workunit.client.0.smithi098.stderr: 35550K .......... .......... .......... .......... .......... 26% 28.3M 4s 2024-04-25T16:34:06.388 INFO:tasks.workunit.client.0.smithi098.stderr: 35600K .......... .......... .......... .......... .......... 26% 30.3M 4s 2024-04-25T16:34:06.389 INFO:tasks.workunit.client.0.smithi098.stderr: 35650K .......... .......... .......... .......... .......... 26% 30.0M 4s 2024-04-25T16:34:06.391 INFO:tasks.workunit.client.0.smithi098.stderr: 35700K .......... .......... .......... .......... .......... 26% 25.7M 4s 2024-04-25T16:34:06.393 INFO:tasks.workunit.client.0.smithi098.stderr: 35750K .......... .......... .......... .......... .......... 26% 29.8M 4s 2024-04-25T16:34:06.394 INFO:tasks.workunit.client.0.smithi098.stderr: 35800K .......... .......... .......... .......... .......... 26% 29.8M 4s 2024-04-25T16:34:06.396 INFO:tasks.workunit.client.0.smithi098.stderr: 35850K .......... .......... .......... .......... .......... 26% 29.5M 4s 2024-04-25T16:34:06.398 INFO:tasks.workunit.client.0.smithi098.stderr: 35900K .......... .......... .......... .......... .......... 26% 25.4M 4s 2024-04-25T16:34:06.399 INFO:tasks.workunit.client.0.smithi098.stderr: 35950K .......... .......... .......... .......... .......... 26% 29.6M 4s 2024-04-25T16:34:06.401 INFO:tasks.workunit.client.0.smithi098.stderr: 36000K .......... .......... .......... .......... .......... 26% 29.0M 4s 2024-04-25T16:34:06.403 INFO:tasks.workunit.client.0.smithi098.stderr: 36050K .......... .......... .......... .......... .......... 26% 27.8M 4s 2024-04-25T16:34:06.405 INFO:tasks.workunit.client.0.smithi098.stderr: 36100K .......... .......... .......... .......... .......... 26% 24.2M 4s 2024-04-25T16:34:06.407 INFO:tasks.workunit.client.0.smithi098.stderr: 36150K .......... .......... .......... .......... .......... 26% 28.4M 4s 2024-04-25T16:34:06.409 INFO:tasks.workunit.client.0.smithi098.stderr: 36200K .......... .......... .......... .......... .......... 26% 28.9M 4s 2024-04-25T16:34:06.410 INFO:tasks.workunit.client.0.smithi098.stderr: 36250K .......... .......... .......... .......... .......... 26% 28.5M 4s 2024-04-25T16:34:06.412 INFO:tasks.workunit.client.0.smithi098.stderr: 36300K .......... .......... .......... .......... .......... 26% 23.7M 4s 2024-04-25T16:34:06.414 INFO:tasks.workunit.client.0.smithi098.stderr: 36350K .......... .......... .......... .......... .......... 26% 27.9M 4s 2024-04-25T16:34:06.416 INFO:tasks.workunit.client.0.smithi098.stderr: 36400K .......... .......... .......... .......... .......... 26% 26.8M 4s 2024-04-25T16:34:06.418 INFO:tasks.workunit.client.0.smithi098.stderr: 36450K .......... .......... .......... .......... .......... 26% 27.8M 4s 2024-04-25T16:34:06.420 INFO:tasks.workunit.client.0.smithi098.stderr: 36500K .......... .......... .......... .......... .......... 26% 21.7M 4s 2024-04-25T16:34:06.422 INFO:tasks.workunit.client.0.smithi098.stderr: 36550K .......... .......... .......... .......... .......... 26% 24.9M 4s 2024-04-25T16:34:06.424 INFO:tasks.workunit.client.0.smithi098.stderr: 36600K .......... .......... .......... .......... .......... 26% 24.5M 4s 2024-04-25T16:34:06.426 INFO:tasks.workunit.client.0.smithi098.stderr: 36650K .......... .......... .......... .......... .......... 27% 24.7M 4s 2024-04-25T16:34:06.428 INFO:tasks.workunit.client.0.smithi098.stderr: 36700K .......... .......... .......... .......... .......... 27% 22.6M 4s 2024-04-25T16:34:06.430 INFO:tasks.workunit.client.0.smithi098.stderr: 36750K .......... .......... .......... .......... .......... 27% 27.6M 4s 2024-04-25T16:34:06.432 INFO:tasks.workunit.client.0.smithi098.stderr: 36800K .......... .......... .......... .......... .......... 27% 28.6M 4s 2024-04-25T16:34:06.433 INFO:tasks.workunit.client.0.smithi098.stderr: 36850K .......... .......... .......... .......... .......... 27% 25.4M 4s 2024-04-25T16:34:06.435 INFO:tasks.workunit.client.0.smithi098.stderr: 36900K .......... .......... .......... .......... .......... 27% 24.5M 4s 2024-04-25T16:34:06.437 INFO:tasks.workunit.client.0.smithi098.stderr: 36950K .......... .......... .......... .......... .......... 27% 33.9M 4s 2024-04-25T16:34:06.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:06 smithi203 ceph-mon[33115]: pgmap v404: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:34:06.438 INFO:tasks.workunit.client.0.smithi098.stderr: 37000K .......... .......... .......... .......... .......... 27% 31.5M 4s 2024-04-25T16:34:06.440 INFO:tasks.workunit.client.0.smithi098.stderr: 37050K .......... .......... .......... .......... .......... 27% 29.0M 4s 2024-04-25T16:34:06.443 INFO:tasks.workunit.client.0.smithi098.stderr: 37100K .......... .......... .......... .......... .......... 27% 17.4M 4s 2024-04-25T16:34:06.445 INFO:tasks.workunit.client.0.smithi098.stderr: 37150K .......... .......... .......... .......... .......... 27% 26.6M 4s 2024-04-25T16:34:06.446 INFO:tasks.workunit.client.0.smithi098.stderr: 37200K .......... .......... .......... .......... .......... 27% 33.8M 4s 2024-04-25T16:34:06.447 INFO:tasks.workunit.client.0.smithi098.stderr: 37250K .......... .......... .......... .......... .......... 27% 35.8M 4s 2024-04-25T16:34:06.450 INFO:tasks.workunit.client.0.smithi098.stderr: 37300K .......... .......... .......... .......... .......... 27% 21.6M 4s 2024-04-25T16:34:06.451 INFO:tasks.workunit.client.0.smithi098.stderr: 37350K .......... .......... .......... .......... .......... 27% 35.9M 4s 2024-04-25T16:34:06.453 INFO:tasks.workunit.client.0.smithi098.stderr: 37400K .......... .......... .......... .......... .......... 27% 30.3M 4s 2024-04-25T16:34:06.455 INFO:tasks.workunit.client.0.smithi098.stderr: 37450K .......... .......... .......... .......... .......... 27% 29.7M 4s 2024-04-25T16:34:06.456 INFO:tasks.workunit.client.0.smithi098.stderr: 37500K .......... .......... .......... .......... .......... 27% 26.3M 4s 2024-04-25T16:34:06.458 INFO:tasks.workunit.client.0.smithi098.stderr: 37550K .......... .......... .......... .......... .......... 27% 30.5M 4s 2024-04-25T16:34:06.459 INFO:tasks.workunit.client.0.smithi098.stderr: 37600K .......... .......... .......... .......... .......... 27% 36.4M 4s 2024-04-25T16:34:06.460 INFO:tasks.workunit.client.0.smithi098.stderr: 37650K .......... .......... .......... .......... .......... 27% 36.9M 4s 2024-04-25T16:34:06.462 INFO:tasks.workunit.client.0.smithi098.stderr: 37700K .......... .......... .......... .......... .......... 27% 27.3M 4s 2024-04-25T16:34:06.464 INFO:tasks.workunit.client.0.smithi098.stderr: 37750K .......... .......... .......... .......... .......... 27% 32.1M 4s 2024-04-25T16:34:06.465 INFO:tasks.workunit.client.0.smithi098.stderr: 37800K .......... .......... .......... .......... .......... 27% 32.7M 4s 2024-04-25T16:34:06.467 INFO:tasks.workunit.client.0.smithi098.stderr: 37850K .......... .......... .......... .......... .......... 27% 32.1M 4s 2024-04-25T16:34:06.468 INFO:tasks.workunit.client.0.smithi098.stderr: 37900K .......... .......... .......... .......... .......... 27% 27.9M 4s 2024-04-25T16:34:06.470 INFO:tasks.workunit.client.0.smithi098.stderr: 37950K .......... .......... .......... .......... .......... 27% 31.9M 4s 2024-04-25T16:34:06.471 INFO:tasks.workunit.client.0.smithi098.stderr: 38000K .......... .......... .......... .......... .......... 28% 32.3M 4s 2024-04-25T16:34:06.473 INFO:tasks.workunit.client.0.smithi098.stderr: 38050K .......... .......... .......... .......... .......... 28% 32.3M 4s 2024-04-25T16:34:06.475 INFO:tasks.workunit.client.0.smithi098.stderr: 38100K .......... .......... .......... .......... .......... 28% 27.3M 4s 2024-04-25T16:34:06.476 INFO:tasks.workunit.client.0.smithi098.stderr: 38150K .......... .......... .......... .......... .......... 28% 32.0M 4s 2024-04-25T16:34:06.478 INFO:tasks.workunit.client.0.smithi098.stderr: 38200K .......... .......... .......... .......... .......... 28% 26.9M 4s 2024-04-25T16:34:06.480 INFO:tasks.workunit.client.0.smithi098.stderr: 38250K .......... .......... .......... .......... .......... 28% 34.2M 4s 2024-04-25T16:34:06.481 INFO:tasks.workunit.client.0.smithi098.stderr: 38300K .......... .......... .......... .......... .......... 28% 30.6M 4s 2024-04-25T16:34:06.483 INFO:tasks.workunit.client.0.smithi098.stderr: 38350K .......... .......... .......... .......... .......... 28% 35.1M 4s 2024-04-25T16:34:06.489 INFO:tasks.workunit.client.0.smithi098.stderr: 38400K .......... .......... .......... .......... .......... 28% 34.7M 4s 2024-04-25T16:34:06.489 INFO:tasks.workunit.client.0.smithi098.stderr: 38450K .......... .......... .......... .......... .......... 28% 35.4M 4s 2024-04-25T16:34:06.489 INFO:tasks.workunit.client.0.smithi098.stderr: 38500K .......... .......... .......... .......... .......... 28% 28.4M 4s 2024-04-25T16:34:06.489 INFO:tasks.workunit.client.0.smithi098.stderr: 38550K .......... .......... .......... .......... .......... 28% 32.9M 4s 2024-04-25T16:34:06.491 INFO:tasks.workunit.client.0.smithi098.stderr: 38600K .......... .......... .......... .......... .......... 28% 33.2M 4s 2024-04-25T16:34:06.491 INFO:tasks.workunit.client.0.smithi098.stderr: 38650K .......... .......... .......... .......... .......... 28% 34.1M 4s 2024-04-25T16:34:06.493 INFO:tasks.workunit.client.0.smithi098.stderr: 38700K .......... .......... .......... .......... .......... 28% 28.1M 4s 2024-04-25T16:34:06.495 INFO:tasks.workunit.client.0.smithi098.stderr: 38750K .......... .......... .......... .......... .......... 28% 33.4M 4s 2024-04-25T16:34:06.496 INFO:tasks.workunit.client.0.smithi098.stderr: 38800K .......... .......... .......... .......... .......... 28% 34.0M 4s 2024-04-25T16:34:06.498 INFO:tasks.workunit.client.0.smithi098.stderr: 38850K .......... .......... .......... .......... .......... 28% 33.5M 4s 2024-04-25T16:34:06.499 INFO:tasks.workunit.client.0.smithi098.stderr: 38900K .......... .......... .......... .......... .......... 28% 28.5M 4s 2024-04-25T16:34:06.501 INFO:tasks.workunit.client.0.smithi098.stderr: 38950K .......... .......... .......... .......... .......... 28% 32.7M 4s 2024-04-25T16:34:06.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:06 smithi098 ceph-mon[29162]: pgmap v404: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:34:06.502 INFO:tasks.workunit.client.0.smithi098.stderr: 39000K .......... .......... .......... .......... .......... 28% 31.4M 4s 2024-04-25T16:34:06.504 INFO:tasks.workunit.client.0.smithi098.stderr: 39050K .......... .......... .......... .......... .......... 28% 32.2M 4s 2024-04-25T16:34:06.506 INFO:tasks.workunit.client.0.smithi098.stderr: 39100K .......... .......... .......... .......... .......... 28% 28.1M 4s 2024-04-25T16:34:06.507 INFO:tasks.workunit.client.0.smithi098.stderr: 39150K .......... .......... .......... .......... .......... 28% 34.1M 4s 2024-04-25T16:34:06.509 INFO:tasks.workunit.client.0.smithi098.stderr: 39200K .......... .......... .......... .......... .......... 28% 33.0M 4s 2024-04-25T16:34:06.510 INFO:tasks.workunit.client.0.smithi098.stderr: 39250K .......... .......... .......... .......... .......... 28% 32.5M 4s 2024-04-25T16:34:06.512 INFO:tasks.workunit.client.0.smithi098.stderr: 39300K .......... .......... .......... .......... .......... 28% 27.9M 4s 2024-04-25T16:34:06.513 INFO:tasks.workunit.client.0.smithi098.stderr: 39350K .......... .......... .......... .......... .......... 29% 33.0M 4s 2024-04-25T16:34:06.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:06 smithi173 ceph-mon[32556]: pgmap v404: 129 pgs: 129 active+clean; 667 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-25T16:34:06.515 INFO:tasks.workunit.client.0.smithi098.stderr: 39400K .......... .......... .......... .......... .......... 29% 32.7M 4s 2024-04-25T16:34:06.516 INFO:tasks.workunit.client.0.smithi098.stderr: 39450K .......... .......... .......... .......... .......... 29% 32.4M 4s 2024-04-25T16:34:06.518 INFO:tasks.workunit.client.0.smithi098.stderr: 39500K .......... .......... .......... .......... .......... 29% 28.2M 4s 2024-04-25T16:34:06.520 INFO:tasks.workunit.client.0.smithi098.stderr: 39550K .......... .......... .......... .......... .......... 29% 32.4M 4s 2024-04-25T16:34:06.521 INFO:tasks.workunit.client.0.smithi098.stderr: 39600K .......... .......... .......... .......... .......... 29% 32.2M 4s 2024-04-25T16:34:06.522 INFO:tasks.workunit.client.0.smithi098.stderr: 39650K .......... .......... .......... .......... .......... 29% 33.2M 4s 2024-04-25T16:34:06.524 INFO:tasks.workunit.client.0.smithi098.stderr: 39700K .......... .......... .......... .......... .......... 29% 27.3M 4s 2024-04-25T16:34:06.526 INFO:tasks.workunit.client.0.smithi098.stderr: 39750K .......... .......... .......... .......... .......... 29% 30.9M 4s 2024-04-25T16:34:06.527 INFO:tasks.workunit.client.0.smithi098.stderr: 39800K .......... .......... .......... .......... .......... 29% 32.2M 4s 2024-04-25T16:34:06.529 INFO:tasks.workunit.client.0.smithi098.stderr: 39850K .......... .......... .......... .......... .......... 29% 30.3M 4s 2024-04-25T16:34:06.531 INFO:tasks.workunit.client.0.smithi098.stderr: 39900K .......... .......... .......... .......... .......... 29% 25.9M 4s 2024-04-25T16:34:06.533 INFO:tasks.workunit.client.0.smithi098.stderr: 39950K .......... .......... .......... .......... .......... 29% 30.6M 4s 2024-04-25T16:34:06.534 INFO:tasks.workunit.client.0.smithi098.stderr: 40000K .......... .......... .......... .......... .......... 29% 29.0M 4s 2024-04-25T16:34:06.536 INFO:tasks.workunit.client.0.smithi098.stderr: 40050K .......... .......... .......... .......... .......... 29% 30.2M 4s 2024-04-25T16:34:06.538 INFO:tasks.workunit.client.0.smithi098.stderr: 40100K .......... .......... .......... .......... .......... 29% 25.2M 4s 2024-04-25T16:34:06.540 INFO:tasks.workunit.client.0.smithi098.stderr: 40150K .......... .......... .......... .......... .......... 29% 27.0M 4s 2024-04-25T16:34:06.541 INFO:tasks.workunit.client.0.smithi098.stderr: 40200K .......... .......... .......... .......... .......... 29% 27.2M 4s 2024-04-25T16:34:06.543 INFO:tasks.workunit.client.0.smithi098.stderr: 40250K .......... .......... .......... .......... .......... 29% 28.4M 4s 2024-04-25T16:34:06.545 INFO:tasks.workunit.client.0.smithi098.stderr: 40300K .......... .......... .......... .......... .......... 29% 24.3M 4s 2024-04-25T16:34:06.547 INFO:tasks.workunit.client.0.smithi098.stderr: 40350K .......... .......... .......... .......... .......... 29% 28.2M 4s 2024-04-25T16:34:06.548 INFO:tasks.workunit.client.0.smithi098.stderr: 40400K .......... .......... .......... .......... .......... 29% 27.8M 4s 2024-04-25T16:34:06.551 INFO:tasks.workunit.client.0.smithi098.stderr: 40450K .......... .......... .......... .......... .......... 29% 27.6M 4s 2024-04-25T16:34:06.552 INFO:tasks.workunit.client.0.smithi098.stderr: 40500K .......... .......... .......... .......... .......... 29% 23.7M 4s 2024-04-25T16:34:06.554 INFO:tasks.workunit.client.0.smithi098.stderr: 40550K .......... .......... .......... .......... .......... 29% 28.2M 4s 2024-04-25T16:34:06.556 INFO:tasks.workunit.client.0.smithi098.stderr: 40600K .......... .......... .......... .......... .......... 29% 27.7M 4s 2024-04-25T16:34:06.558 INFO:tasks.workunit.client.0.smithi098.stderr: 40650K .......... .......... .......... .......... .......... 29% 26.8M 4s 2024-04-25T16:34:06.560 INFO:tasks.workunit.client.0.smithi098.stderr: 40700K .......... .......... .......... .......... .......... 30% 23.5M 4s 2024-04-25T16:34:06.562 INFO:tasks.workunit.client.0.smithi098.stderr: 40750K .......... .......... .......... .......... .......... 30% 27.1M 4s 2024-04-25T16:34:06.563 INFO:tasks.workunit.client.0.smithi098.stderr: 40800K .......... .......... .......... .......... .......... 30% 27.4M 4s 2024-04-25T16:34:06.565 INFO:tasks.workunit.client.0.smithi098.stderr: 40850K .......... .......... .......... .......... .......... 30% 27.3M 4s 2024-04-25T16:34:06.567 INFO:tasks.workunit.client.0.smithi098.stderr: 40900K .......... .......... .......... .......... .......... 30% 23.8M 4s 2024-04-25T16:34:06.569 INFO:tasks.workunit.client.0.smithi098.stderr: 40950K .......... .......... .......... .......... .......... 30% 27.3M 4s 2024-04-25T16:34:06.570 INFO:tasks.workunit.client.0.smithi098.stderr: 41000K .......... .......... .......... .......... .......... 30% 32.5M 4s 2024-04-25T16:34:06.572 INFO:tasks.workunit.client.0.smithi098.stderr: 41050K .......... .......... .......... .......... .......... 30% 34.1M 4s 2024-04-25T16:34:06.574 INFO:tasks.workunit.client.0.smithi098.stderr: 41100K .......... .......... .......... .......... .......... 30% 27.0M 4s 2024-04-25T16:34:06.575 INFO:tasks.workunit.client.0.smithi098.stderr: 41150K .......... .......... .......... .......... .......... 30% 33.9M 4s 2024-04-25T16:34:06.577 INFO:tasks.workunit.client.0.smithi098.stderr: 41200K .......... .......... .......... .......... .......... 30% 32.4M 4s 2024-04-25T16:34:06.578 INFO:tasks.workunit.client.0.smithi098.stderr: 41250K .......... .......... .......... .......... .......... 30% 35.0M 4s 2024-04-25T16:34:06.579 INFO:tasks.workunit.client.0.smithi098.stderr: 41300K .......... .......... .......... .......... .......... 30% 31.7M 4s 2024-04-25T16:34:06.581 INFO:tasks.workunit.client.0.smithi098.stderr: 41350K .......... .......... .......... .......... .......... 30% 37.9M 4s 2024-04-25T16:34:06.582 INFO:tasks.workunit.client.0.smithi098.stderr: 41400K .......... .......... .......... .......... .......... 30% 30.7M 4s 2024-04-25T16:34:06.584 INFO:tasks.workunit.client.0.smithi098.stderr: 41450K .......... .......... .......... .......... .......... 30% 24.6M 4s 2024-04-25T16:34:06.586 INFO:tasks.workunit.client.0.smithi098.stderr: 41500K .......... .......... .......... .......... .......... 30% 27.3M 4s 2024-04-25T16:34:06.588 INFO:tasks.workunit.client.0.smithi098.stderr: 41550K .......... .......... .......... .......... .......... 30% 23.2M 4s 2024-04-25T16:34:06.590 INFO:tasks.workunit.client.0.smithi098.stderr: 41600K .......... .......... .......... .......... .......... 30% 28.3M 4s 2024-04-25T16:34:06.591 INFO:tasks.workunit.client.0.smithi098.stderr: 41650K .......... .......... .......... .......... .......... 30% 36.6M 4s 2024-04-25T16:34:06.594 INFO:tasks.workunit.client.0.smithi098.stderr: 41700K .......... .......... .......... .......... .......... 30% 22.8M 4s 2024-04-25T16:34:06.595 INFO:tasks.workunit.client.0.smithi098.stderr: 41750K .......... .......... .......... .......... .......... 30% 35.8M 4s 2024-04-25T16:34:06.596 INFO:tasks.workunit.client.0.smithi098.stderr: 41800K .......... .......... .......... .......... .......... 30% 38.0M 4s 2024-04-25T16:34:06.598 INFO:tasks.workunit.client.0.smithi098.stderr: 41850K .......... .......... .......... .......... .......... 30% 38.4M 4s 2024-04-25T16:34:06.599 INFO:tasks.workunit.client.0.smithi098.stderr: 41900K .......... .......... .......... .......... .......... 30% 32.2M 4s 2024-04-25T16:34:06.600 INFO:tasks.workunit.client.0.smithi098.stderr: 41950K .......... .......... .......... .......... .......... 30% 37.9M 4s 2024-04-25T16:34:06.603 INFO:tasks.workunit.client.0.smithi098.stderr: 42000K .......... .......... .......... .......... .......... 30% 37.1M 4s 2024-04-25T16:34:06.603 INFO:tasks.workunit.client.0.smithi098.stderr: 42050K .......... .......... .......... .......... .......... 31% 36.0M 4s 2024-04-25T16:34:06.604 INFO:tasks.workunit.client.0.smithi098.stderr: 42100K .......... .......... .......... .......... .......... 31% 31.6M 4s 2024-04-25T16:34:06.606 INFO:tasks.workunit.client.0.smithi098.stderr: 42150K .......... .......... .......... .......... .......... 31% 36.8M 4s 2024-04-25T16:34:06.607 INFO:tasks.workunit.client.0.smithi098.stderr: 42200K .......... .......... .......... .......... .......... 31% 37.1M 4s 2024-04-25T16:34:06.609 INFO:tasks.workunit.client.0.smithi098.stderr: 42250K .......... .......... .......... .......... .......... 31% 36.2M 4s 2024-04-25T16:34:06.610 INFO:tasks.workunit.client.0.smithi098.stderr: 42300K .......... .......... .......... .......... .......... 31% 30.6M 3s 2024-04-25T16:34:06.612 INFO:tasks.workunit.client.0.smithi098.stderr: 42350K .......... .......... .......... .......... .......... 31% 32.1M 3s 2024-04-25T16:34:06.613 INFO:tasks.workunit.client.0.smithi098.stderr: 42400K .......... .......... .......... .......... .......... 31% 32.6M 3s 2024-04-25T16:34:06.615 INFO:tasks.workunit.client.0.smithi098.stderr: 42450K .......... .......... .......... .......... .......... 31% 30.9M 3s 2024-04-25T16:34:06.616 INFO:tasks.workunit.client.0.smithi098.stderr: 42500K .......... .......... .......... .......... .......... 31% 24.6M 3s 2024-04-25T16:34:06.618 INFO:tasks.workunit.client.0.smithi098.stderr: 42550K .......... .......... .......... .......... .......... 31% 32.0M 3s 2024-04-25T16:34:06.620 INFO:tasks.workunit.client.0.smithi098.stderr: 42600K .......... .......... .......... .......... .......... 31% 33.6M 3s 2024-04-25T16:34:06.621 INFO:tasks.workunit.client.0.smithi098.stderr: 42650K .......... .......... .......... .......... .......... 31% 33.3M 3s 2024-04-25T16:34:06.623 INFO:tasks.workunit.client.0.smithi098.stderr: 42700K .......... .......... .......... .......... .......... 31% 28.4M 3s 2024-04-25T16:34:06.624 INFO:tasks.workunit.client.0.smithi098.stderr: 42750K .......... .......... .......... .......... .......... 31% 33.3M 3s 2024-04-25T16:34:06.626 INFO:tasks.workunit.client.0.smithi098.stderr: 42800K .......... .......... .......... .......... .......... 31% 31.9M 3s 2024-04-25T16:34:06.627 INFO:tasks.workunit.client.0.smithi098.stderr: 42850K .......... .......... .......... .......... .......... 31% 32.8M 3s 2024-04-25T16:34:06.629 INFO:tasks.workunit.client.0.smithi098.stderr: 42900K .......... .......... .......... .......... .......... 31% 27.7M 3s 2024-04-25T16:34:06.630 INFO:tasks.workunit.client.0.smithi098.stderr: 42950K .......... .......... .......... .......... .......... 31% 32.7M 3s 2024-04-25T16:34:06.632 INFO:tasks.workunit.client.0.smithi098.stderr: 43000K .......... .......... .......... .......... .......... 31% 31.8M 3s 2024-04-25T16:34:06.634 INFO:tasks.workunit.client.0.smithi098.stderr: 43050K .......... .......... .......... .......... .......... 31% 32.4M 3s 2024-04-25T16:34:06.635 INFO:tasks.workunit.client.0.smithi098.stderr: 43100K .......... .......... .......... .......... .......... 31% 26.7M 3s 2024-04-25T16:34:06.637 INFO:tasks.workunit.client.0.smithi098.stderr: 43150K .......... .......... .......... .......... .......... 31% 31.5M 3s 2024-04-25T16:34:06.639 INFO:tasks.workunit.client.0.smithi098.stderr: 43200K .......... .......... .......... .......... .......... 31% 31.3M 3s 2024-04-25T16:34:06.640 INFO:tasks.workunit.client.0.smithi098.stderr: 43250K .......... .......... .......... .......... .......... 31% 30.9M 3s 2024-04-25T16:34:06.642 INFO:tasks.workunit.client.0.smithi098.stderr: 43300K .......... .......... .......... .......... .......... 31% 27.0M 3s 2024-04-25T16:34:06.644 INFO:tasks.workunit.client.0.smithi098.stderr: 43350K .......... .......... .......... .......... .......... 31% 31.2M 3s 2024-04-25T16:34:06.645 INFO:tasks.workunit.client.0.smithi098.stderr: 43400K .......... .......... .......... .......... .......... 32% 31.3M 3s 2024-04-25T16:34:06.647 INFO:tasks.workunit.client.0.smithi098.stderr: 43450K .......... .......... .......... .......... .......... 32% 30.7M 3s 2024-04-25T16:34:06.649 INFO:tasks.workunit.client.0.smithi098.stderr: 43500K .......... .......... .......... .......... .......... 32% 26.3M 3s 2024-04-25T16:34:06.650 INFO:tasks.workunit.client.0.smithi098.stderr: 43550K .......... .......... .......... .......... .......... 32% 30.4M 3s 2024-04-25T16:34:06.652 INFO:tasks.workunit.client.0.smithi098.stderr: 43600K .......... .......... .......... .......... .......... 32% 30.3M 3s 2024-04-25T16:34:06.654 INFO:tasks.workunit.client.0.smithi098.stderr: 43650K .......... .......... .......... .......... .......... 32% 30.9M 3s 2024-04-25T16:34:06.655 INFO:tasks.workunit.client.0.smithi098.stderr: 43700K .......... .......... .......... .......... .......... 32% 25.8M 3s 2024-04-25T16:34:06.657 INFO:tasks.workunit.client.0.smithi098.stderr: 43750K .......... .......... .......... .......... .......... 32% 30.4M 3s 2024-04-25T16:34:06.658 INFO:tasks.workunit.client.0.smithi098.stderr: 43800K .......... .......... .......... .......... .......... 32% 30.3M 3s 2024-04-25T16:34:06.660 INFO:tasks.workunit.client.0.smithi098.stderr: 43850K .......... .......... .......... .......... .......... 32% 30.1M 3s 2024-04-25T16:34:06.662 INFO:tasks.workunit.client.0.smithi098.stderr: 43900K .......... .......... .......... .......... .......... 32% 25.4M 3s 2024-04-25T16:34:06.663 INFO:tasks.workunit.client.0.smithi098.stderr: 43950K .......... .......... .......... .......... .......... 32% 30.0M 3s 2024-04-25T16:34:06.666 INFO:tasks.workunit.client.0.smithi098.stderr: 44000K .......... .......... .......... .......... .......... 32% 29.0M 3s 2024-04-25T16:34:06.667 INFO:tasks.workunit.client.0.smithi098.stderr: 44050K .......... .......... .......... .......... .......... 32% 30.0M 3s 2024-04-25T16:34:06.671 INFO:tasks.workunit.client.0.smithi098.stderr: 44100K .......... .......... .......... .......... .......... 32% 25.4M 3s 2024-04-25T16:34:06.671 INFO:tasks.workunit.client.0.smithi098.stderr: 44150K .......... .......... .......... .......... .......... 32% 28.7M 3s 2024-04-25T16:34:06.672 INFO:tasks.workunit.client.0.smithi098.stderr: 44200K .......... .......... .......... .......... .......... 32% 27.7M 3s 2024-04-25T16:34:06.674 INFO:tasks.workunit.client.0.smithi098.stderr: 44250K .......... .......... .......... .......... .......... 32% 28.4M 3s 2024-04-25T16:34:06.676 INFO:tasks.workunit.client.0.smithi098.stderr: 44300K .......... .......... .......... .......... .......... 32% 23.9M 3s 2024-04-25T16:34:06.678 INFO:tasks.workunit.client.0.smithi098.stderr: 44350K .......... .......... .......... .......... .......... 32% 27.9M 3s 2024-04-25T16:34:06.679 INFO:tasks.workunit.client.0.smithi098.stderr: 44400K .......... .......... .......... .......... .......... 32% 27.4M 3s 2024-04-25T16:34:06.681 INFO:tasks.workunit.client.0.smithi098.stderr: 44450K .......... .......... .......... .......... .......... 32% 27.9M 3s 2024-04-25T16:34:06.683 INFO:tasks.workunit.client.0.smithi098.stderr: 44500K .......... .......... .......... .......... .......... 32% 23.5M 3s 2024-04-25T16:34:06.685 INFO:tasks.workunit.client.0.smithi098.stderr: 44550K .......... .......... .......... .......... .......... 32% 27.7M 3s 2024-04-25T16:34:06.687 INFO:tasks.workunit.client.0.smithi098.stderr: 44600K .......... .......... .......... .......... .......... 32% 26.8M 3s 2024-04-25T16:34:06.692 INFO:tasks.workunit.client.0.smithi098.stderr: 44650K .......... .......... .......... .......... .......... 32% 27.3M 3s 2024-04-25T16:34:06.692 INFO:tasks.workunit.client.0.smithi098.stderr: 44700K .......... .......... .......... .......... .......... 32% 22.3M 3s 2024-04-25T16:34:06.693 INFO:tasks.workunit.client.0.smithi098.stderr: 44750K .......... .......... .......... .......... .......... 33% 27.2M 3s 2024-04-25T16:34:06.695 INFO:tasks.workunit.client.0.smithi098.stderr: 44800K .......... .......... .......... .......... .......... 33% 26.8M 3s 2024-04-25T16:34:06.697 INFO:tasks.workunit.client.0.smithi098.stderr: 44850K .......... .......... .......... .......... .......... 33% 27.3M 3s 2024-04-25T16:34:06.698 INFO:tasks.workunit.client.0.smithi098.stderr: 44900K .......... .......... .......... .......... .......... 33% 22.1M 3s 2024-04-25T16:34:06.701 INFO:tasks.workunit.client.0.smithi098.stderr: 44950K .......... .......... .......... .......... .......... 33% 26.6M 3s 2024-04-25T16:34:06.702 INFO:tasks.workunit.client.0.smithi098.stderr: 45000K .......... .......... .......... .......... .......... 33% 26.5M 3s 2024-04-25T16:34:06.704 INFO:tasks.workunit.client.0.smithi098.stderr: 45050K .......... .......... .......... .......... .......... 33% 30.2M 3s 2024-04-25T16:34:06.705 INFO:tasks.workunit.client.0.smithi098.stderr: 45100K .......... .......... .......... .......... .......... 33% 33.2M 3s 2024-04-25T16:34:06.707 INFO:tasks.workunit.client.0.smithi098.stderr: 45150K .......... .......... .......... .......... .......... 33% 40.3M 3s 2024-04-25T16:34:06.708 INFO:tasks.workunit.client.0.smithi098.stderr: 45200K .......... .......... .......... .......... .......... 33% 40.3M 3s 2024-04-25T16:34:06.709 INFO:tasks.workunit.client.0.smithi098.stderr: 45250K .......... .......... .......... .......... .......... 33% 40.8M 3s 2024-04-25T16:34:06.710 INFO:tasks.workunit.client.0.smithi098.stderr: 45300K .......... .......... .......... .......... .......... 33% 34.0M 3s 2024-04-25T16:34:06.712 INFO:tasks.workunit.client.0.smithi098.stderr: 45350K .......... .......... .......... .......... .......... 33% 36.6M 3s 2024-04-25T16:34:06.713 INFO:tasks.workunit.client.0.smithi098.stderr: 45400K .......... .......... .......... .......... .......... 33% 27.6M 3s 2024-04-25T16:34:06.715 INFO:tasks.workunit.client.0.smithi098.stderr: 45450K .......... .......... .......... .......... .......... 33% 28.7M 3s 2024-04-25T16:34:06.717 INFO:tasks.workunit.client.0.smithi098.stderr: 45500K .......... .......... .......... .......... .......... 33% 23.4M 3s 2024-04-25T16:34:06.719 INFO:tasks.workunit.client.0.smithi098.stderr: 45550K .......... .......... .......... .......... .......... 33% 29.0M 3s 2024-04-25T16:34:06.720 INFO:tasks.workunit.client.0.smithi098.stderr: 45600K .......... .......... .......... .......... .......... 33% 35.2M 3s 2024-04-25T16:34:06.722 INFO:tasks.workunit.client.0.smithi098.stderr: 45650K .......... .......... .......... .......... .......... 33% 36.8M 3s 2024-04-25T16:34:06.724 INFO:tasks.workunit.client.0.smithi098.stderr: 45700K .......... .......... .......... .......... .......... 33% 22.4M 3s 2024-04-25T16:34:06.725 INFO:tasks.workunit.client.0.smithi098.stderr: 45750K .......... .......... .......... .......... .......... 33% 36.8M 3s 2024-04-25T16:34:06.726 INFO:tasks.workunit.client.0.smithi098.stderr: 45800K .......... .......... .......... .......... .......... 33% 37.1M 3s 2024-04-25T16:34:06.728 INFO:tasks.workunit.client.0.smithi098.stderr: 45850K .......... .......... .......... .......... .......... 33% 36.7M 3s 2024-04-25T16:34:06.730 INFO:tasks.workunit.client.0.smithi098.stderr: 45900K .......... .......... .......... .......... .......... 33% 31.0M 3s 2024-04-25T16:34:06.731 INFO:tasks.workunit.client.0.smithi098.stderr: 45950K .......... .......... .......... .......... .......... 33% 36.7M 3s 2024-04-25T16:34:06.732 INFO:tasks.workunit.client.0.smithi098.stderr: 46000K .......... .......... .......... .......... .......... 33% 38.0M 3s 2024-04-25T16:34:06.734 INFO:tasks.workunit.client.0.smithi098.stderr: 46050K .......... .......... .......... .......... .......... 33% 38.0M 3s 2024-04-25T16:34:06.735 INFO:tasks.workunit.client.0.smithi098.stderr: 46100K .......... .......... .......... .......... .......... 33% 32.2M 3s 2024-04-25T16:34:06.736 INFO:tasks.workunit.client.0.smithi098.stderr: 46150K .......... .......... .......... .......... .......... 34% 37.5M 3s 2024-04-25T16:34:06.737 INFO:tasks.workunit.client.0.smithi098.stderr: 46200K .......... .......... .......... .......... .......... 34% 37.7M 3s 2024-04-25T16:34:06.739 INFO:tasks.workunit.client.0.smithi098.stderr: 46250K .......... .......... .......... .......... .......... 34% 35.4M 3s 2024-04-25T16:34:06.740 INFO:tasks.workunit.client.0.smithi098.stderr: 46300K .......... .......... .......... .......... .......... 34% 29.2M 3s 2024-04-25T16:34:06.742 INFO:tasks.workunit.client.0.smithi098.stderr: 46350K .......... .......... .......... .......... .......... 34% 34.8M 3s 2024-04-25T16:34:06.743 INFO:tasks.workunit.client.0.smithi098.stderr: 46400K .......... .......... .......... .......... .......... 34% 34.9M 3s 2024-04-25T16:34:06.745 INFO:tasks.workunit.client.0.smithi098.stderr: 46450K .......... .......... .......... .......... .......... 34% 34.2M 3s 2024-04-25T16:34:06.746 INFO:tasks.workunit.client.0.smithi098.stderr: 46500K .......... .......... .......... .......... .......... 34% 29.8M 3s 2024-04-25T16:34:06.748 INFO:tasks.workunit.client.0.smithi098.stderr: 46550K .......... .......... .......... .......... .......... 34% 25.2M 3s 2024-04-25T16:34:06.750 INFO:tasks.workunit.client.0.smithi098.stderr: 46600K .......... .......... .......... .......... .......... 34% 33.6M 3s 2024-04-25T16:34:06.751 INFO:tasks.workunit.client.0.smithi098.stderr: 46650K .......... .......... .......... .......... .......... 34% 34.1M 3s 2024-04-25T16:34:06.753 INFO:tasks.workunit.client.0.smithi098.stderr: 46700K .......... .......... .......... .......... .......... 34% 29.1M 3s 2024-04-25T16:34:06.754 INFO:tasks.workunit.client.0.smithi098.stderr: 46750K .......... .......... .......... .......... .......... 34% 33.2M 3s 2024-04-25T16:34:06.756 INFO:tasks.workunit.client.0.smithi098.stderr: 46800K .......... .......... .......... .......... .......... 34% 32.6M 3s 2024-04-25T16:34:06.757 INFO:tasks.workunit.client.0.smithi098.stderr: 46850K .......... .......... .......... .......... .......... 34% 31.6M 3s 2024-04-25T16:34:06.759 INFO:tasks.workunit.client.0.smithi098.stderr: 46900K .......... .......... .......... .......... .......... 34% 26.1M 3s 2024-04-25T16:34:06.761 INFO:tasks.workunit.client.0.smithi098.stderr: 46950K .......... .......... .......... .......... .......... 34% 30.0M 3s 2024-04-25T16:34:06.762 INFO:tasks.workunit.client.0.smithi098.stderr: 47000K .......... .......... .......... .......... .......... 34% 31.0M 3s 2024-04-25T16:34:06.764 INFO:tasks.workunit.client.0.smithi098.stderr: 47050K .......... .......... .......... .......... .......... 34% 28.9M 3s 2024-04-25T16:34:06.766 INFO:tasks.workunit.client.0.smithi098.stderr: 47100K .......... .......... .......... .......... .......... 34% 25.1M 3s 2024-04-25T16:34:06.767 INFO:tasks.workunit.client.0.smithi098.stderr: 47150K .......... .......... .......... .......... .......... 34% 30.3M 3s 2024-04-25T16:34:06.769 INFO:tasks.workunit.client.0.smithi098.stderr: 47200K .......... .......... .......... .......... .......... 34% 30.2M 3s 2024-04-25T16:34:06.771 INFO:tasks.workunit.client.0.smithi098.stderr: 47250K .......... .......... .......... .......... .......... 34% 29.0M 3s 2024-04-25T16:34:06.773 INFO:tasks.workunit.client.0.smithi098.stderr: 47300K .......... .......... .......... .......... .......... 34% 24.6M 3s 2024-04-25T16:34:06.775 INFO:tasks.workunit.client.0.smithi098.stderr: 47350K .......... .......... .......... .......... .......... 34% 28.9M 3s 2024-04-25T16:34:06.776 INFO:tasks.workunit.client.0.smithi098.stderr: 47400K .......... .......... .......... .......... .......... 34% 29.3M 3s 2024-04-25T16:34:06.778 INFO:tasks.workunit.client.0.smithi098.stderr: 47450K .......... .......... .......... .......... .......... 34% 28.7M 3s 2024-04-25T16:34:06.780 INFO:tasks.workunit.client.0.smithi098.stderr: 47500K .......... .......... .......... .......... .......... 35% 24.0M 3s 2024-04-25T16:34:06.782 INFO:tasks.workunit.client.0.smithi098.stderr: 47550K .......... .......... .......... .......... .......... 35% 28.5M 3s 2024-04-25T16:34:06.783 INFO:tasks.workunit.client.0.smithi098.stderr: 47600K .......... .......... .......... .......... .......... 35% 28.7M 3s 2024-04-25T16:34:06.785 INFO:tasks.workunit.client.0.smithi098.stderr: 47650K .......... .......... .......... .......... .......... 35% 28.0M 3s 2024-04-25T16:34:06.787 INFO:tasks.workunit.client.0.smithi098.stderr: 47700K .......... .......... .......... .......... .......... 35% 24.0M 3s 2024-04-25T16:34:06.789 INFO:tasks.workunit.client.0.smithi098.stderr: 47750K .......... .......... .......... .......... .......... 35% 28.0M 3s 2024-04-25T16:34:06.791 INFO:tasks.workunit.client.0.smithi098.stderr: 47800K .......... .......... .......... .......... .......... 35% 27.2M 3s 2024-04-25T16:34:06.793 INFO:tasks.workunit.client.0.smithi098.stderr: 47850K .......... .......... .......... .......... .......... 35% 28.3M 3s 2024-04-25T16:34:06.795 INFO:tasks.workunit.client.0.smithi098.stderr: 47900K .......... .......... .......... .......... .......... 35% 23.9M 3s 2024-04-25T16:34:06.796 INFO:tasks.workunit.client.0.smithi098.stderr: 47950K .......... .......... .......... .......... .......... 35% 27.8M 3s 2024-04-25T16:34:06.798 INFO:tasks.workunit.client.0.smithi098.stderr: 48000K .......... .......... .......... .......... .......... 35% 27.3M 3s 2024-04-25T16:34:06.800 INFO:tasks.workunit.client.0.smithi098.stderr: 48050K .......... .......... .......... .......... .......... 35% 27.2M 3s 2024-04-25T16:34:06.802 INFO:tasks.workunit.client.0.smithi098.stderr: 48100K .......... .......... .......... .......... .......... 35% 25.2M 3s 2024-04-25T16:34:06.804 INFO:tasks.workunit.client.0.smithi098.stderr: 48150K .......... .......... .......... .......... .......... 35% 29.5M 3s 2024-04-25T16:34:06.805 INFO:tasks.workunit.client.0.smithi098.stderr: 48200K .......... .......... .......... .......... .......... 35% 29.2M 3s 2024-04-25T16:34:06.807 INFO:tasks.workunit.client.0.smithi098.stderr: 48250K .......... .......... .......... .......... .......... 35% 28.3M 3s 2024-04-25T16:34:06.809 INFO:tasks.workunit.client.0.smithi098.stderr: 48300K .......... .......... .......... .......... .......... 35% 24.6M 3s 2024-04-25T16:34:06.811 INFO:tasks.workunit.client.0.smithi098.stderr: 48350K .......... .......... .......... .......... .......... 35% 27.8M 3s 2024-04-25T16:34:06.812 INFO:tasks.workunit.client.0.smithi098.stderr: 48400K .......... .......... .......... .......... .......... 35% 28.4M 3s 2024-04-25T16:34:06.814 INFO:tasks.workunit.client.0.smithi098.stderr: 48450K .......... .......... .......... .......... .......... 35% 28.5M 3s 2024-04-25T16:34:06.816 INFO:tasks.workunit.client.0.smithi098.stderr: 48500K .......... .......... .......... .......... .......... 35% 24.0M 3s 2024-04-25T16:34:06.818 INFO:tasks.workunit.client.0.smithi098.stderr: 48550K .......... .......... .......... .......... .......... 35% 28.3M 3s 2024-04-25T16:34:06.819 INFO:tasks.workunit.client.0.smithi098.stderr: 48600K .......... .......... .......... .......... .......... 35% 28.1M 3s 2024-04-25T16:34:06.821 INFO:tasks.workunit.client.0.smithi098.stderr: 48650K .......... .......... .......... .......... .......... 35% 25.6M 3s 2024-04-25T16:34:06.823 INFO:tasks.workunit.client.0.smithi098.stderr: 48700K .......... .......... .......... .......... .......... 35% 22.2M 3s 2024-04-25T16:34:06.825 INFO:tasks.workunit.client.0.smithi098.stderr: 48750K .......... .......... .......... .......... .......... 35% 26.3M 3s 2024-04-25T16:34:06.827 INFO:tasks.workunit.client.0.smithi098.stderr: 48800K .......... .......... .......... .......... .......... 35% 27.0M 3s 2024-04-25T16:34:06.829 INFO:tasks.workunit.client.0.smithi098.stderr: 48850K .......... .......... .......... .......... .......... 36% 27.0M 3s 2024-04-25T16:34:06.831 INFO:tasks.workunit.client.0.smithi098.stderr: 48900K .......... .......... .......... .......... .......... 36% 23.3M 3s 2024-04-25T16:34:06.833 INFO:tasks.workunit.client.0.smithi098.stderr: 48950K .......... .......... .......... .......... .......... 36% 27.7M 3s 2024-04-25T16:34:06.835 INFO:tasks.workunit.client.0.smithi098.stderr: 49000K .......... .......... .......... .......... .......... 36% 26.5M 3s 2024-04-25T16:34:06.837 INFO:tasks.workunit.client.0.smithi098.stderr: 49050K .......... .......... .......... .......... .......... 36% 26.1M 3s 2024-04-25T16:34:06.839 INFO:tasks.workunit.client.0.smithi098.stderr: 49100K .......... .......... .......... .......... .......... 36% 21.8M 3s 2024-04-25T16:34:06.841 INFO:tasks.workunit.client.0.smithi098.stderr: 49150K .......... .......... .......... .......... .......... 36% 27.2M 3s 2024-04-25T16:34:06.843 INFO:tasks.workunit.client.0.smithi098.stderr: 49200K .......... .......... .......... .......... .......... 36% 19.6M 3s 2024-04-25T16:34:06.846 INFO:tasks.workunit.client.0.smithi098.stderr: 49250K .......... .......... .......... .......... .......... 36% 23.1M 3s 2024-04-25T16:34:06.850 INFO:tasks.workunit.client.0.smithi098.stderr: 49300K .......... .......... .......... .......... .......... 36% 26.4M 3s 2024-04-25T16:34:06.850 INFO:tasks.workunit.client.0.smithi098.stderr: 49350K .......... .......... .......... .......... .......... 36% 30.0M 3s 2024-04-25T16:34:06.851 INFO:tasks.workunit.client.0.smithi098.stderr: 49400K .......... .......... .......... .......... .......... 36% 30.0M 3s 2024-04-25T16:34:06.852 INFO:tasks.workunit.client.0.smithi098.stderr: 49450K .......... .......... .......... .......... .......... 36% 33.5M 3s 2024-04-25T16:34:06.853 INFO:tasks.workunit.client.0.smithi098.stderr: 49500K .......... .......... .......... .......... .......... 36% 30.7M 3s 2024-04-25T16:34:06.855 INFO:tasks.workunit.client.0.smithi098.stderr: 49550K .......... .......... .......... .......... .......... 36% 35.0M 3s 2024-04-25T16:34:06.856 INFO:tasks.workunit.client.0.smithi098.stderr: 49600K .......... .......... .......... .......... .......... 36% 36.1M 3s 2024-04-25T16:34:06.858 INFO:tasks.workunit.client.0.smithi098.stderr: 49650K .......... .......... .......... .......... .......... 36% 34.5M 3s 2024-04-25T16:34:06.859 INFO:tasks.workunit.client.0.smithi098.stderr: 49700K .......... .......... .......... .......... .......... 36% 31.1M 3s 2024-04-25T16:34:06.860 INFO:tasks.workunit.client.0.smithi098.stderr: 49750K .......... .......... .......... .......... .......... 36% 32.0M 3s 2024-04-25T16:34:06.862 INFO:tasks.workunit.client.0.smithi098.stderr: 49800K .......... .......... .......... .......... .......... 36% 35.9M 3s 2024-04-25T16:34:06.864 INFO:tasks.workunit.client.0.smithi098.stderr: 49850K .......... .......... .......... .......... .......... 36% 31.8M 3s 2024-04-25T16:34:06.865 INFO:tasks.workunit.client.0.smithi098.stderr: 49900K .......... .......... .......... .......... .......... 36% 30.1M 3s 2024-04-25T16:34:06.867 INFO:tasks.workunit.client.0.smithi098.stderr: 49950K .......... .......... .......... .......... .......... 36% 34.9M 3s 2024-04-25T16:34:06.868 INFO:tasks.workunit.client.0.smithi098.stderr: 50000K .......... .......... .......... .......... .......... 36% 33.7M 3s 2024-04-25T16:34:06.869 INFO:tasks.workunit.client.0.smithi098.stderr: 50050K .......... .......... .......... .......... .......... 36% 32.6M 3s 2024-04-25T16:34:06.871 INFO:tasks.workunit.client.0.smithi098.stderr: 50100K .......... .......... .......... .......... .......... 36% 29.0M 3s 2024-04-25T16:34:06.873 INFO:tasks.workunit.client.0.smithi098.stderr: 50150K .......... .......... .......... .......... .......... 36% 32.0M 3s 2024-04-25T16:34:06.874 INFO:tasks.workunit.client.0.smithi098.stderr: 50200K .......... .......... .......... .......... .......... 37% 33.2M 3s 2024-04-25T16:34:06.876 INFO:tasks.workunit.client.0.smithi098.stderr: 50250K .......... .......... .......... .......... .......... 37% 32.8M 3s 2024-04-25T16:34:06.877 INFO:tasks.workunit.client.0.smithi098.stderr: 50300K .......... .......... .......... .......... .......... 37% 29.2M 3s 2024-04-25T16:34:06.879 INFO:tasks.workunit.client.0.smithi098.stderr: 50350K .......... .......... .......... .......... .......... 37% 31.5M 3s 2024-04-25T16:34:06.880 INFO:tasks.workunit.client.0.smithi098.stderr: 50400K .......... .......... .......... .......... .......... 37% 31.7M 3s 2024-04-25T16:34:06.882 INFO:tasks.workunit.client.0.smithi098.stderr: 50450K .......... .......... .......... .......... .......... 37% 33.6M 3s 2024-04-25T16:34:06.884 INFO:tasks.workunit.client.0.smithi098.stderr: 50500K .......... .......... .......... .......... .......... 37% 27.9M 3s 2024-04-25T16:34:06.885 INFO:tasks.workunit.client.0.smithi098.stderr: 50550K .......... .......... .......... .......... .......... 37% 32.6M 3s 2024-04-25T16:34:06.886 INFO:tasks.workunit.client.0.smithi098.stderr: 50600K .......... .......... .......... .......... .......... 37% 32.1M 3s 2024-04-25T16:34:06.888 INFO:tasks.workunit.client.0.smithi098.stderr: 50650K .......... .......... .......... .......... .......... 37% 32.3M 3s 2024-04-25T16:34:06.890 INFO:tasks.workunit.client.0.smithi098.stderr: 50700K .......... .......... .......... .......... .......... 37% 24.3M 3s 2024-04-25T16:34:06.892 INFO:tasks.workunit.client.0.smithi098.stderr: 50750K .......... .......... .......... .......... .......... 37% 31.6M 3s 2024-04-25T16:34:06.893 INFO:tasks.workunit.client.0.smithi098.stderr: 50800K .......... .......... .......... .......... .......... 37% 30.5M 3s 2024-04-25T16:34:06.895 INFO:tasks.workunit.client.0.smithi098.stderr: 50850K .......... .......... .......... .......... .......... 37% 29.2M 3s 2024-04-25T16:34:06.897 INFO:tasks.workunit.client.0.smithi098.stderr: 50900K .......... .......... .......... .......... .......... 37% 24.8M 3s 2024-04-25T16:34:06.899 INFO:tasks.workunit.client.0.smithi098.stderr: 50950K .......... .......... .......... .......... .......... 37% 28.0M 3s 2024-04-25T16:34:06.900 INFO:tasks.workunit.client.0.smithi098.stderr: 51000K .......... .......... .......... .......... .......... 37% 29.7M 3s 2024-04-25T16:34:06.903 INFO:tasks.workunit.client.0.smithi098.stderr: 51050K .......... .......... .......... .......... .......... 37% 29.1M 3s 2024-04-25T16:34:06.905 INFO:tasks.workunit.client.0.smithi098.stderr: 51100K .......... .......... .......... .......... .......... 37% 25.4M 3s 2024-04-25T16:34:06.906 INFO:tasks.workunit.client.0.smithi098.stderr: 51150K .......... .......... .......... .......... .......... 37% 29.6M 3s 2024-04-25T16:34:06.908 INFO:tasks.workunit.client.0.smithi098.stderr: 51200K .......... .......... .......... .......... .......... 37% 29.5M 3s 2024-04-25T16:34:06.910 INFO:tasks.workunit.client.0.smithi098.stderr: 51250K .......... .......... .......... .......... .......... 37% 29.6M 3s 2024-04-25T16:34:06.911 INFO:tasks.workunit.client.0.smithi098.stderr: 51300K .......... .......... .......... .......... .......... 37% 25.1M 3s 2024-04-25T16:34:06.912 INFO:tasks.workunit.client.0.smithi098.stderr: 51350K .......... .......... .......... .......... .......... 37% 29.2M 3s 2024-04-25T16:34:06.914 INFO:tasks.workunit.client.0.smithi098.stderr: 51400K .......... .......... .......... .......... .......... 37% 28.8M 3s 2024-04-25T16:34:06.916 INFO:tasks.workunit.client.0.smithi098.stderr: 51450K .......... .......... .......... .......... .......... 37% 29.8M 3s 2024-04-25T16:34:06.918 INFO:tasks.workunit.client.0.smithi098.stderr: 51500K .......... .......... .......... .......... .......... 37% 26.6M 3s 2024-04-25T16:34:06.920 INFO:tasks.workunit.client.0.smithi098.stderr: 51550K .......... .......... .......... .......... .......... 38% 31.1M 3s 2024-04-25T16:34:06.922 INFO:tasks.workunit.client.0.smithi098.stderr: 51600K .......... .......... .......... .......... .......... 38% 28.8M 3s 2024-04-25T16:34:06.923 INFO:tasks.workunit.client.0.smithi098.stderr: 51650K .......... .......... .......... .......... .......... 38% 28.3M 3s 2024-04-25T16:34:06.925 INFO:tasks.workunit.client.0.smithi098.stderr: 51700K .......... .......... .......... .......... .......... 38% 24.2M 3s 2024-04-25T16:34:06.926 INFO:tasks.workunit.client.0.smithi098.stderr: 51750K .......... .......... .......... .......... .......... 38% 27.8M 3s 2024-04-25T16:34:06.928 INFO:tasks.workunit.client.0.smithi098.stderr: 51800K .......... .......... .......... .......... .......... 38% 28.1M 3s 2024-04-25T16:34:06.930 INFO:tasks.workunit.client.0.smithi098.stderr: 51850K .......... .......... .......... .......... .......... 38% 27.6M 3s 2024-04-25T16:34:06.932 INFO:tasks.workunit.client.0.smithi098.stderr: 51900K .......... .......... .......... .......... .......... 38% 23.8M 3s 2024-04-25T16:34:06.935 INFO:tasks.workunit.client.0.smithi098.stderr: 51950K .......... .......... .......... .......... .......... 38% 27.4M 3s 2024-04-25T16:34:06.937 INFO:tasks.workunit.client.0.smithi098.stderr: 52000K .......... .......... .......... .......... .......... 38% 27.5M 3s 2024-04-25T16:34:06.939 INFO:tasks.workunit.client.0.smithi098.stderr: 52050K .......... .......... .......... .......... .......... 38% 27.2M 3s 2024-04-25T16:34:06.941 INFO:tasks.workunit.client.0.smithi098.stderr: 52100K .......... .......... .......... .......... .......... 38% 23.4M 3s 2024-04-25T16:34:06.943 INFO:tasks.workunit.client.0.smithi098.stderr: 52150K .......... .......... .......... .......... .......... 38% 28.1M 3s 2024-04-25T16:34:06.943 INFO:tasks.workunit.client.0.smithi098.stderr: 52200K .......... .......... .......... .......... .......... 38% 29.1M 3s 2024-04-25T16:34:06.945 INFO:tasks.workunit.client.0.smithi098.stderr: 52250K .......... .......... .......... .......... .......... 38% 29.3M 3s 2024-04-25T16:34:06.948 INFO:tasks.workunit.client.0.smithi098.stderr: 52300K .......... .......... .......... .......... .......... 38% 24.2M 3s 2024-04-25T16:34:06.950 INFO:tasks.workunit.client.0.smithi098.stderr: 52350K .......... .......... .......... .......... .......... 38% 26.5M 3s 2024-04-25T16:34:06.951 INFO:tasks.workunit.client.0.smithi098.stderr: 52400K .......... .......... .......... .......... .......... 38% 26.7M 3s 2024-04-25T16:34:06.953 INFO:tasks.workunit.client.0.smithi098.stderr: 52450K .......... .......... .......... .......... .......... 38% 26.4M 3s 2024-04-25T16:34:06.955 INFO:tasks.workunit.client.0.smithi098.stderr: 52500K .......... .......... .......... .......... .......... 38% 22.8M 3s 2024-04-25T16:34:06.958 INFO:tasks.workunit.client.0.smithi098.stderr: 52550K .......... .......... .......... .......... .......... 38% 26.2M 3s 2024-04-25T16:34:06.959 INFO:tasks.workunit.client.0.smithi098.stderr: 52600K .......... .......... .......... .......... .......... 38% 26.2M 3s 2024-04-25T16:34:06.960 INFO:tasks.workunit.client.0.smithi098.stderr: 52650K .......... .......... .......... .......... .......... 38% 26.2M 3s 2024-04-25T16:34:06.962 INFO:tasks.workunit.client.0.smithi098.stderr: 52700K .......... .......... .......... .......... .......... 38% 22.2M 3s 2024-04-25T16:34:06.964 INFO:tasks.workunit.client.0.smithi098.stderr: 52750K .......... .......... .......... .......... .......... 38% 25.6M 3s 2024-04-25T16:34:06.967 INFO:tasks.workunit.client.0.smithi098.stderr: 52800K .......... .......... .......... .......... .......... 38% 25.4M 3s 2024-04-25T16:34:06.968 INFO:tasks.workunit.client.0.smithi098.stderr: 52850K .......... .......... .......... .......... .......... 38% 25.5M 3s 2024-04-25T16:34:06.970 INFO:tasks.workunit.client.0.smithi098.stderr: 52900K .......... .......... .......... .......... .......... 39% 21.6M 3s 2024-04-25T16:34:06.972 INFO:tasks.workunit.client.0.smithi098.stderr: 52950K .......... .......... .......... .......... .......... 39% 25.3M 3s 2024-04-25T16:34:06.974 INFO:tasks.workunit.client.0.smithi098.stderr: 53000K .......... .......... .......... .......... .......... 39% 25.5M 3s 2024-04-25T16:34:06.977 INFO:tasks.workunit.client.0.smithi098.stderr: 53050K .......... .......... .......... .......... .......... 39% 25.4M 3s 2024-04-25T16:34:06.978 INFO:tasks.workunit.client.0.smithi098.stderr: 53100K .......... .......... .......... .......... .......... 39% 21.7M 3s 2024-04-25T16:34:06.980 INFO:tasks.workunit.client.0.smithi098.stderr: 53150K .......... .......... .......... .......... .......... 39% 24.5M 3s 2024-04-25T16:34:06.982 INFO:tasks.workunit.client.0.smithi098.stderr: 53200K .......... .......... .......... .......... .......... 39% 24.3M 3s 2024-04-25T16:34:06.984 INFO:tasks.workunit.client.0.smithi098.stderr: 53250K .......... .......... .......... .......... .......... 39% 24.5M 3s 2024-04-25T16:34:06.986 INFO:tasks.workunit.client.0.smithi098.stderr: 53300K .......... .......... .......... .......... .......... 39% 25.4M 3s 2024-04-25T16:34:06.987 INFO:tasks.workunit.client.0.smithi098.stderr: 53350K .......... .......... .......... .......... .......... 39% 32.3M 3s 2024-04-25T16:34:06.989 INFO:tasks.workunit.client.0.smithi098.stderr: 53400K .......... .......... .......... .......... .......... 39% 37.1M 3s 2024-04-25T16:34:06.990 INFO:tasks.workunit.client.0.smithi098.stderr: 53450K .......... .......... .......... .......... .......... 39% 38.5M 3s 2024-04-25T16:34:06.992 INFO:tasks.workunit.client.0.smithi098.stderr: 53500K .......... .......... .......... .......... .......... 39% 32.2M 3s 2024-04-25T16:34:06.993 INFO:tasks.workunit.client.0.smithi098.stderr: 53550K .......... .......... .......... .......... .......... 39% 32.5M 3s 2024-04-25T16:34:06.994 INFO:tasks.workunit.client.0.smithi098.stderr: 53600K .......... .......... .......... .......... .......... 39% 33.3M 3s 2024-04-25T16:34:06.996 INFO:tasks.workunit.client.0.smithi098.stderr: 53650K .......... .......... .......... .......... .......... 39% 28.5M 3s 2024-04-25T16:34:06.998 INFO:tasks.workunit.client.0.smithi098.stderr: 53700K .......... .......... .......... .......... .......... 39% 27.8M 3s 2024-04-25T16:34:06.999 INFO:tasks.workunit.client.0.smithi098.stderr: 53750K .......... .......... .......... .......... .......... 39% 30.7M 3s 2024-04-25T16:34:07.001 INFO:tasks.workunit.client.0.smithi098.stderr: 53800K .......... .......... .......... .......... .......... 39% 34.4M 3s 2024-04-25T16:34:07.003 INFO:tasks.workunit.client.0.smithi098.stderr: 53850K .......... .......... .......... .......... .......... 39% 34.1M 3s 2024-04-25T16:34:07.004 INFO:tasks.workunit.client.0.smithi098.stderr: 53900K .......... .......... .......... .......... .......... 39% 26.4M 3s 2024-04-25T16:34:07.006 INFO:tasks.workunit.client.0.smithi098.stderr: 53950K .......... .......... .......... .......... .......... 39% 33.3M 3s 2024-04-25T16:34:07.007 INFO:tasks.workunit.client.0.smithi098.stderr: 54000K .......... .......... .......... .......... .......... 39% 33.0M 3s 2024-04-25T16:34:07.009 INFO:tasks.workunit.client.0.smithi098.stderr: 54050K .......... .......... .......... .......... .......... 39% 34.8M 3s 2024-04-25T16:34:07.010 INFO:tasks.workunit.client.0.smithi098.stderr: 54100K .......... .......... .......... .......... .......... 39% 30.8M 3s 2024-04-25T16:34:07.012 INFO:tasks.workunit.client.0.smithi098.stderr: 54150K .......... .......... .......... .......... .......... 39% 33.5M 3s 2024-04-25T16:34:07.013 INFO:tasks.workunit.client.0.smithi098.stderr: 54200K .......... .......... .......... .......... .......... 39% 35.9M 3s 2024-04-25T16:34:07.014 INFO:tasks.workunit.client.0.smithi098.stderr: 54250K .......... .......... .......... .......... .......... 39% 37.3M 3s 2024-04-25T16:34:07.016 INFO:tasks.workunit.client.0.smithi098.stderr: 54300K .......... .......... .......... .......... .......... 40% 29.7M 3s 2024-04-25T16:34:07.017 INFO:tasks.workunit.client.0.smithi098.stderr: 54350K .......... .......... .......... .......... .......... 40% 34.1M 3s 2024-04-25T16:34:07.019 INFO:tasks.workunit.client.0.smithi098.stderr: 54400K .......... .......... .......... .......... .......... 40% 34.1M 3s 2024-04-25T16:34:07.020 INFO:tasks.workunit.client.0.smithi098.stderr: 54450K .......... .......... .......... .......... .......... 40% 34.0M 3s 2024-04-25T16:34:07.022 INFO:tasks.workunit.client.0.smithi098.stderr: 54500K .......... .......... .......... .......... .......... 40% 28.8M 3s 2024-04-25T16:34:07.023 INFO:tasks.workunit.client.0.smithi098.stderr: 54550K .......... .......... .......... .......... .......... 40% 33.9M 3s 2024-04-25T16:34:07.025 INFO:tasks.workunit.client.0.smithi098.stderr: 54600K .......... .......... .......... .......... .......... 40% 35.0M 3s 2024-04-25T16:34:07.027 INFO:tasks.workunit.client.0.smithi098.stderr: 54650K .......... .......... .......... .......... .......... 40% 34.1M 3s 2024-04-25T16:34:07.028 INFO:tasks.workunit.client.0.smithi098.stderr: 54700K .......... .......... .......... .......... .......... 40% 28.6M 3s 2024-04-25T16:34:07.029 INFO:tasks.workunit.client.0.smithi098.stderr: 54750K .......... .......... .......... .......... .......... 40% 33.2M 3s 2024-04-25T16:34:07.031 INFO:tasks.workunit.client.0.smithi098.stderr: 54800K .......... .......... .......... .......... .......... 40% 31.1M 3s 2024-04-25T16:34:07.033 INFO:tasks.workunit.client.0.smithi098.stderr: 54850K .......... .......... .......... .......... .......... 40% 33.0M 3s 2024-04-25T16:34:07.034 INFO:tasks.workunit.client.0.smithi098.stderr: 54900K .......... .......... .......... .......... .......... 40% 28.3M 3s 2024-04-25T16:34:07.036 INFO:tasks.workunit.client.0.smithi098.stderr: 54950K .......... .......... .......... .......... .......... 40% 33.1M 3s 2024-04-25T16:34:07.037 INFO:tasks.workunit.client.0.smithi098.stderr: 55000K .......... .......... .......... .......... .......... 40% 33.3M 3s 2024-04-25T16:34:07.039 INFO:tasks.workunit.client.0.smithi098.stderr: 55050K .......... .......... .......... .......... .......... 40% 30.9M 3s 2024-04-25T16:34:07.040 INFO:tasks.workunit.client.0.smithi098.stderr: 55100K .......... .......... .......... .......... .......... 40% 27.1M 3s 2024-04-25T16:34:07.042 INFO:tasks.workunit.client.0.smithi098.stderr: 55150K .......... .......... .......... .......... .......... 40% 29.0M 3s 2024-04-25T16:34:07.044 INFO:tasks.workunit.client.0.smithi098.stderr: 55200K .......... .......... .......... .......... .......... 40% 29.1M 3s 2024-04-25T16:34:07.045 INFO:tasks.workunit.client.0.smithi098.stderr: 55250K .......... .......... .......... .......... .......... 40% 32.4M 3s 2024-04-25T16:34:07.047 INFO:tasks.workunit.client.0.smithi098.stderr: 55300K .......... .......... .......... .......... .......... 40% 27.9M 3s 2024-04-25T16:34:07.049 INFO:tasks.workunit.client.0.smithi098.stderr: 55350K .......... .......... .......... .......... .......... 40% 29.5M 3s 2024-04-25T16:34:07.050 INFO:tasks.workunit.client.0.smithi098.stderr: 55400K .......... .......... .......... .......... .......... 40% 31.2M 3s 2024-04-25T16:34:07.052 INFO:tasks.workunit.client.0.smithi098.stderr: 55450K .......... .......... .......... .......... .......... 40% 30.6M 3s 2024-04-25T16:34:07.054 INFO:tasks.workunit.client.0.smithi098.stderr: 55500K .......... .......... .......... .......... .......... 40% 26.1M 3s 2024-04-25T16:34:07.055 INFO:tasks.workunit.client.0.smithi098.stderr: 55550K .......... .......... .......... .......... .......... 40% 31.3M 3s 2024-04-25T16:34:07.057 INFO:tasks.workunit.client.0.smithi098.stderr: 55600K .......... .......... .......... .......... .......... 40% 31.6M 3s 2024-04-25T16:34:07.059 INFO:tasks.workunit.client.0.smithi098.stderr: 55650K .......... .......... .......... .......... .......... 41% 31.3M 3s 2024-04-25T16:34:07.060 INFO:tasks.workunit.client.0.smithi098.stderr: 55700K .......... .......... .......... .......... .......... 41% 27.3M 3s 2024-04-25T16:34:07.062 INFO:tasks.workunit.client.0.smithi098.stderr: 55750K .......... .......... .......... .......... .......... 41% 31.1M 3s 2024-04-25T16:34:07.063 INFO:tasks.workunit.client.0.smithi098.stderr: 55800K .......... .......... .......... .......... .......... 41% 31.8M 3s 2024-04-25T16:34:07.065 INFO:tasks.workunit.client.0.smithi098.stderr: 55850K .......... .......... .......... .......... .......... 41% 31.2M 3s 2024-04-25T16:34:07.067 INFO:tasks.workunit.client.0.smithi098.stderr: 55900K .......... .......... .......... .......... .......... 41% 26.6M 3s 2024-04-25T16:34:07.068 INFO:tasks.workunit.client.0.smithi098.stderr: 55950K .......... .......... .......... .......... .......... 41% 31.2M 3s 2024-04-25T16:34:07.070 INFO:tasks.workunit.client.0.smithi098.stderr: 56000K .......... .......... .......... .......... .......... 41% 30.2M 3s 2024-04-25T16:34:07.072 INFO:tasks.workunit.client.0.smithi098.stderr: 56050K .......... .......... .......... .......... .......... 41% 30.4M 3s 2024-04-25T16:34:07.073 INFO:tasks.workunit.client.0.smithi098.stderr: 56100K .......... .......... .......... .......... .......... 41% 26.3M 3s 2024-04-25T16:34:07.075 INFO:tasks.workunit.client.0.smithi098.stderr: 56150K .......... .......... .......... .......... .......... 41% 30.2M 3s 2024-04-25T16:34:07.077 INFO:tasks.workunit.client.0.smithi098.stderr: 56200K .......... .......... .......... .......... .......... 41% 29.6M 3s 2024-04-25T16:34:07.078 INFO:tasks.workunit.client.0.smithi098.stderr: 56250K .......... .......... .......... .......... .......... 41% 30.0M 3s 2024-04-25T16:34:07.080 INFO:tasks.workunit.client.0.smithi098.stderr: 56300K .......... .......... .......... .......... .......... 41% 25.4M 3s 2024-04-25T16:34:07.082 INFO:tasks.workunit.client.0.smithi098.stderr: 56350K .......... .......... .......... .......... .......... 41% 29.7M 3s 2024-04-25T16:34:07.084 INFO:tasks.workunit.client.0.smithi098.stderr: 56400K .......... .......... .......... .......... .......... 41% 29.6M 3s 2024-04-25T16:34:07.085 INFO:tasks.workunit.client.0.smithi098.stderr: 56450K .......... .......... .......... .......... .......... 41% 29.8M 3s 2024-04-25T16:34:07.087 INFO:tasks.workunit.client.0.smithi098.stderr: 56500K .......... .......... .......... .......... .......... 41% 25.4M 3s 2024-04-25T16:34:07.089 INFO:tasks.workunit.client.0.smithi098.stderr: 56550K .......... .......... .......... .......... .......... 41% 28.8M 3s 2024-04-25T16:34:07.091 INFO:tasks.workunit.client.0.smithi098.stderr: 56600K .......... .......... .......... .......... .......... 41% 29.0M 3s 2024-04-25T16:34:07.092 INFO:tasks.workunit.client.0.smithi098.stderr: 56650K .......... .......... .......... .......... .......... 41% 28.9M 3s 2024-04-25T16:34:07.094 INFO:tasks.workunit.client.0.smithi098.stderr: 56700K .......... .......... .......... .......... .......... 41% 24.4M 3s 2024-04-25T16:34:07.096 INFO:tasks.workunit.client.0.smithi098.stderr: 56750K .......... .......... .......... .......... .......... 41% 28.1M 3s 2024-04-25T16:34:07.098 INFO:tasks.workunit.client.0.smithi098.stderr: 56800K .......... .......... .......... .......... .......... 41% 26.9M 3s 2024-04-25T16:34:07.100 INFO:tasks.workunit.client.0.smithi098.stderr: 56850K .......... .......... .......... .......... .......... 41% 24.7M 3s 2024-04-25T16:34:07.102 INFO:tasks.workunit.client.0.smithi098.stderr: 56900K .......... .......... .......... .......... .......... 41% 22.5M 3s 2024-04-25T16:34:07.104 INFO:tasks.workunit.client.0.smithi098.stderr: 56950K .......... .......... .......... .......... .......... 41% 26.1M 3s 2024-04-25T16:34:07.106 INFO:tasks.workunit.client.0.smithi098.stderr: 57000K .......... .......... .......... .......... .......... 42% 24.5M 3s 2024-04-25T16:34:07.108 INFO:tasks.workunit.client.0.smithi098.stderr: 57050K .......... .......... .......... .......... .......... 42% 27.0M 3s 2024-04-25T16:34:07.110 INFO:tasks.workunit.client.0.smithi098.stderr: 57100K .......... .......... .......... .......... .......... 42% 23.2M 3s 2024-04-25T16:34:07.112 INFO:tasks.workunit.client.0.smithi098.stderr: 57150K .......... .......... .......... .......... .......... 42% 25.2M 3s 2024-04-25T16:34:07.114 INFO:tasks.workunit.client.0.smithi098.stderr: 57200K .......... .......... .......... .......... .......... 42% 24.7M 3s 2024-04-25T16:34:07.116 INFO:tasks.workunit.client.0.smithi098.stderr: 57250K .......... .......... .......... .......... .......... 42% 24.5M 3s 2024-04-25T16:34:07.118 INFO:tasks.workunit.client.0.smithi098.stderr: 57300K .......... .......... .......... .......... .......... 42% 21.6M 3s 2024-04-25T16:34:07.120 INFO:tasks.workunit.client.0.smithi098.stderr: 57350K .......... .......... .......... .......... .......... 42% 24.3M 3s 2024-04-25T16:34:07.122 INFO:tasks.workunit.client.0.smithi098.stderr: 57400K .......... .......... .......... .......... .......... 42% 28.2M 3s 2024-04-25T16:34:07.123 INFO:tasks.workunit.client.0.smithi098.stderr: 57450K .......... .......... .......... .......... .......... 42% 32.2M 3s 2024-04-25T16:34:07.125 INFO:tasks.workunit.client.0.smithi098.stderr: 57500K .......... .......... .......... .......... .......... 42% 27.6M 3s 2024-04-25T16:34:07.127 INFO:tasks.workunit.client.0.smithi098.stderr: 57550K .......... .......... .......... .......... .......... 42% 30.2M 3s 2024-04-25T16:34:07.128 INFO:tasks.workunit.client.0.smithi098.stderr: 57600K .......... .......... .......... .......... .......... 42% 31.1M 3s 2024-04-25T16:34:07.130 INFO:tasks.workunit.client.0.smithi098.stderr: 57650K .......... .......... .......... .......... .......... 42% 32.3M 3s 2024-04-25T16:34:07.131 INFO:tasks.workunit.client.0.smithi098.stderr: 57700K .......... .......... .......... .......... .......... 42% 28.2M 3s 2024-04-25T16:34:07.133 INFO:tasks.workunit.client.0.smithi098.stderr: 57750K .......... .......... .......... .......... .......... 42% 35.3M 3s 2024-04-25T16:34:07.134 INFO:tasks.workunit.client.0.smithi098.stderr: 57800K .......... .......... .......... .......... .......... 42% 31.9M 3s 2024-04-25T16:34:07.136 INFO:tasks.workunit.client.0.smithi098.stderr: 57850K .......... .......... .......... .......... .......... 42% 25.8M 3s 2024-04-25T16:34:07.138 INFO:tasks.workunit.client.0.smithi098.stderr: 57900K .......... .......... .......... .......... .......... 42% 27.0M 3s 2024-04-25T16:34:07.140 INFO:tasks.workunit.client.0.smithi098.stderr: 57950K .......... .......... .......... .......... .......... 42% 30.0M 3s 2024-04-25T16:34:07.141 INFO:tasks.workunit.client.0.smithi098.stderr: 58000K .......... .......... .......... .......... .......... 42% 35.0M 3s 2024-04-25T16:34:07.143 INFO:tasks.workunit.client.0.smithi098.stderr: 58050K .......... .......... .......... .......... .......... 42% 27.8M 3s 2024-04-25T16:34:07.145 INFO:tasks.workunit.client.0.smithi098.stderr: 58100K .......... .......... .......... .......... .......... 42% 24.9M 3s 2024-04-25T16:34:07.146 INFO:tasks.workunit.client.0.smithi098.stderr: 58150K .......... .......... .......... .......... .......... 42% 30.1M 3s 2024-04-25T16:34:07.148 INFO:tasks.workunit.client.0.smithi098.stderr: 58200K .......... .......... .......... .......... .......... 42% 30.3M 3s 2024-04-25T16:34:07.149 INFO:tasks.workunit.client.0.smithi098.stderr: 58250K .......... .......... .......... .......... .......... 42% 32.5M 3s 2024-04-25T16:34:07.151 INFO:tasks.workunit.client.0.smithi098.stderr: 58300K .......... .......... .......... .......... .......... 42% 29.4M 3s 2024-04-25T16:34:07.152 INFO:tasks.workunit.client.0.smithi098.stderr: 58350K .......... .......... .......... .......... .......... 43% 34.4M 3s 2024-04-25T16:34:07.154 INFO:tasks.workunit.client.0.smithi098.stderr: 58400K .......... .......... .......... .......... .......... 43% 37.3M 3s 2024-04-25T16:34:07.156 INFO:tasks.workunit.client.0.smithi098.stderr: 58450K .......... .......... .......... .......... .......... 43% 31.4M 3s 2024-04-25T16:34:07.157 INFO:tasks.workunit.client.0.smithi098.stderr: 58500K .......... .......... .......... .......... .......... 43% 24.4M 3s 2024-04-25T16:34:07.159 INFO:tasks.workunit.client.0.smithi098.stderr: 58550K .......... .......... .......... .......... .......... 43% 33.1M 3s 2024-04-25T16:34:07.160 INFO:tasks.workunit.client.0.smithi098.stderr: 58600K .......... .......... .......... .......... .......... 43% 36.1M 3s 2024-04-25T16:34:07.162 INFO:tasks.workunit.client.0.smithi098.stderr: 58650K .......... .......... .......... .......... .......... 43% 35.0M 3s 2024-04-25T16:34:07.163 INFO:tasks.workunit.client.0.smithi098.stderr: 58700K .......... .......... .......... .......... .......... 43% 28.4M 3s 2024-04-25T16:34:07.165 INFO:tasks.workunit.client.0.smithi098.stderr: 58750K .......... .......... .......... .......... .......... 43% 33.7M 3s 2024-04-25T16:34:07.166 INFO:tasks.workunit.client.0.smithi098.stderr: 58800K .......... .......... .......... .......... .......... 43% 34.5M 3s 2024-04-25T16:34:07.168 INFO:tasks.workunit.client.0.smithi098.stderr: 58850K .......... .......... .......... .......... .......... 43% 33.7M 3s 2024-04-25T16:34:07.169 INFO:tasks.workunit.client.0.smithi098.stderr: 58900K .......... .......... .......... .......... .......... 43% 28.6M 3s 2024-04-25T16:34:07.171 INFO:tasks.workunit.client.0.smithi098.stderr: 58950K .......... .......... .......... .......... .......... 43% 32.2M 3s 2024-04-25T16:34:07.172 INFO:tasks.workunit.client.0.smithi098.stderr: 59000K .......... .......... .......... .......... .......... 43% 33.6M 3s 2024-04-25T16:34:07.174 INFO:tasks.workunit.client.0.smithi098.stderr: 59050K .......... .......... .......... .......... .......... 43% 34.1M 3s 2024-04-25T16:34:07.175 INFO:tasks.workunit.client.0.smithi098.stderr: 59100K .......... .......... .......... .......... .......... 43% 27.6M 3s 2024-04-25T16:34:07.177 INFO:tasks.workunit.client.0.smithi098.stderr: 59150K .......... .......... .......... .......... .......... 43% 30.8M 3s 2024-04-25T16:34:07.179 INFO:tasks.workunit.client.0.smithi098.stderr: 59200K .......... .......... .......... .......... .......... 43% 32.1M 3s 2024-04-25T16:34:07.180 INFO:tasks.workunit.client.0.smithi098.stderr: 59250K .......... .......... .......... .......... .......... 43% 32.2M 3s 2024-04-25T16:34:07.182 INFO:tasks.workunit.client.0.smithi098.stderr: 59300K .......... .......... .......... .......... .......... 43% 27.5M 3s 2024-04-25T16:34:07.183 INFO:tasks.workunit.client.0.smithi098.stderr: 59350K .......... .......... .......... .......... .......... 43% 31.3M 3s 2024-04-25T16:34:07.185 INFO:tasks.workunit.client.0.smithi098.stderr: 59400K .......... .......... .......... .......... .......... 43% 30.6M 3s 2024-04-25T16:34:07.187 INFO:tasks.workunit.client.0.smithi098.stderr: 59450K .......... .......... .......... .......... .......... 43% 27.9M 3s 2024-04-25T16:34:07.189 INFO:tasks.workunit.client.0.smithi098.stderr: 59500K .......... .......... .......... .......... .......... 43% 23.9M 3s 2024-04-25T16:34:07.190 INFO:tasks.workunit.client.0.smithi098.stderr: 59550K .......... .......... .......... .......... .......... 43% 28.1M 3s 2024-04-25T16:34:07.192 INFO:tasks.workunit.client.0.smithi098.stderr: 59600K .......... .......... .......... .......... .......... 43% 28.3M 3s 2024-04-25T16:34:07.194 INFO:tasks.workunit.client.0.smithi098.stderr: 59650K .......... .......... .......... .......... .......... 43% 28.3M 3s 2024-04-25T16:34:07.196 INFO:tasks.workunit.client.0.smithi098.stderr: 59700K .......... .......... .......... .......... .......... 44% 24.1M 3s 2024-04-25T16:34:07.198 INFO:tasks.workunit.client.0.smithi098.stderr: 59750K .......... .......... .......... .......... .......... 44% 28.2M 3s 2024-04-25T16:34:07.200 INFO:tasks.workunit.client.0.smithi098.stderr: 59800K .......... .......... .......... .......... .......... 44% 28.3M 3s 2024-04-25T16:34:07.201 INFO:tasks.workunit.client.0.smithi098.stderr: 59850K .......... .......... .......... .......... .......... 44% 28.4M 3s 2024-04-25T16:34:07.203 INFO:tasks.workunit.client.0.smithi098.stderr: 59900K .......... .......... .......... .......... .......... 44% 23.9M 3s 2024-04-25T16:34:07.205 INFO:tasks.workunit.client.0.smithi098.stderr: 59950K .......... .......... .......... .......... .......... 44% 27.5M 3s 2024-04-25T16:34:07.207 INFO:tasks.workunit.client.0.smithi098.stderr: 60000K .......... .......... .......... .......... .......... 44% 28.9M 3s 2024-04-25T16:34:07.208 INFO:tasks.workunit.client.0.smithi098.stderr: 60050K .......... .......... .......... .......... .......... 44% 29.3M 3s 2024-04-25T16:34:07.210 INFO:tasks.workunit.client.0.smithi098.stderr: 60100K .......... .......... .......... .......... .......... 44% 25.1M 3s 2024-04-25T16:34:07.212 INFO:tasks.workunit.client.0.smithi098.stderr: 60150K .......... .......... .......... .......... .......... 44% 28.9M 3s 2024-04-25T16:34:07.213 INFO:tasks.workunit.client.0.smithi098.stderr: 60200K .......... .......... .......... .......... .......... 44% 28.9M 3s 2024-04-25T16:34:07.216 INFO:tasks.workunit.client.0.smithi098.stderr: 60250K .......... .......... .......... .......... .......... 44% 29.0M 3s 2024-04-25T16:34:07.217 INFO:tasks.workunit.client.0.smithi098.stderr: 60300K .......... .......... .......... .......... .......... 44% 25.0M 3s 2024-04-25T16:34:07.219 INFO:tasks.workunit.client.0.smithi098.stderr: 60350K .......... .......... .......... .......... .......... 44% 29.2M 3s 2024-04-25T16:34:07.221 INFO:tasks.workunit.client.0.smithi098.stderr: 60400K .......... .......... .......... .......... .......... 44% 27.6M 3s 2024-04-25T16:34:07.223 INFO:tasks.workunit.client.0.smithi098.stderr: 60450K .......... .......... .......... .......... .......... 44% 26.7M 3s 2024-04-25T16:34:07.224 INFO:tasks.workunit.client.0.smithi098.stderr: 60500K .......... .......... .......... .......... .......... 44% 23.6M 3s 2024-04-25T16:34:07.226 INFO:tasks.workunit.client.0.smithi098.stderr: 60550K .......... .......... .......... .......... .......... 44% 27.6M 3s 2024-04-25T16:34:07.228 INFO:tasks.workunit.client.0.smithi098.stderr: 60600K .......... .......... .......... .......... .......... 44% 28.0M 3s 2024-04-25T16:34:07.230 INFO:tasks.workunit.client.0.smithi098.stderr: 60650K .......... .......... .......... .......... .......... 44% 27.8M 3s 2024-04-25T16:34:07.232 INFO:tasks.workunit.client.0.smithi098.stderr: 60700K .......... .......... .......... .......... .......... 44% 24.3M 3s 2024-04-25T16:34:07.234 INFO:tasks.workunit.client.0.smithi098.stderr: 60750K .......... .......... .......... .......... .......... 44% 27.8M 3s 2024-04-25T16:34:07.235 INFO:tasks.workunit.client.0.smithi098.stderr: 60800K .......... .......... .......... .......... .......... 44% 28.1M 3s 2024-04-25T16:34:07.237 INFO:tasks.workunit.client.0.smithi098.stderr: 60850K .......... .......... .......... .......... .......... 44% 27.8M 3s 2024-04-25T16:34:07.239 INFO:tasks.workunit.client.0.smithi098.stderr: 60900K .......... .......... .......... .......... .......... 44% 23.4M 3s 2024-04-25T16:34:07.241 INFO:tasks.workunit.client.0.smithi098.stderr: 60950K .......... .......... .......... .......... .......... 44% 26.9M 3s 2024-04-25T16:34:07.243 INFO:tasks.workunit.client.0.smithi098.stderr: 61000K .......... .......... .......... .......... .......... 44% 26.0M 3s 2024-04-25T16:34:07.245 INFO:tasks.workunit.client.0.smithi098.stderr: 61050K .......... .......... .......... .......... .......... 45% 26.7M 3s 2024-04-25T16:34:07.247 INFO:tasks.workunit.client.0.smithi098.stderr: 61100K .......... .......... .......... .......... .......... 45% 23.3M 3s 2024-04-25T16:34:07.249 INFO:tasks.workunit.client.0.smithi098.stderr: 61150K .......... .......... .......... .......... .......... 45% 27.0M 3s 2024-04-25T16:34:07.251 INFO:tasks.workunit.client.0.smithi098.stderr: 61200K .......... .......... .......... .......... .......... 45% 26.5M 3s 2024-04-25T16:34:07.253 INFO:tasks.workunit.client.0.smithi098.stderr: 61250K .......... .......... .......... .......... .......... 45% 25.7M 3s 2024-04-25T16:34:07.255 INFO:tasks.workunit.client.0.smithi098.stderr: 61300K .......... .......... .......... .......... .......... 45% 21.8M 3s 2024-04-25T16:34:07.256 INFO:tasks.workunit.client.0.smithi098.stderr: 61350K .......... .......... .......... .......... .......... 45% 25.4M 3s 2024-04-25T16:34:07.259 INFO:tasks.workunit.client.0.smithi098.stderr: 61400K .......... .......... .......... .......... .......... 45% 27.0M 3s 2024-04-25T16:34:07.260 INFO:tasks.workunit.client.0.smithi098.stderr: 61450K .......... .......... .......... .......... .......... 45% 29.7M 3s 2024-04-25T16:34:07.262 INFO:tasks.workunit.client.0.smithi098.stderr: 61500K .......... .......... .......... .......... .......... 45% 28.8M 3s 2024-04-25T16:34:07.263 INFO:tasks.workunit.client.0.smithi098.stderr: 61550K .......... .......... .......... .......... .......... 45% 29.9M 3s 2024-04-25T16:34:07.265 INFO:tasks.workunit.client.0.smithi098.stderr: 61600K .......... .......... .......... .......... .......... 45% 36.0M 3s 2024-04-25T16:34:07.266 INFO:tasks.workunit.client.0.smithi098.stderr: 61650K .......... .......... .......... .......... .......... 45% 37.5M 3s 2024-04-25T16:34:07.268 INFO:tasks.workunit.client.0.smithi098.stderr: 61700K .......... .......... .......... .......... .......... 45% 32.2M 3s 2024-04-25T16:34:07.269 INFO:tasks.workunit.client.0.smithi098.stderr: 61750K .......... .......... .......... .......... .......... 45% 32.3M 3s 2024-04-25T16:34:07.271 INFO:tasks.workunit.client.0.smithi098.stderr: 61800K .......... .......... .......... .......... .......... 45% 32.7M 3s 2024-04-25T16:34:07.272 INFO:tasks.workunit.client.0.smithi098.stderr: 61850K .......... .......... .......... .......... .......... 45% 30.8M 3s 2024-04-25T16:34:07.274 INFO:tasks.workunit.client.0.smithi098.stderr: 61900K .......... .......... .......... .......... .......... 45% 27.3M 3s 2024-04-25T16:34:07.276 INFO:tasks.workunit.client.0.smithi098.stderr: 61950K .......... .......... .......... .......... .......... 45% 28.6M 3s 2024-04-25T16:34:07.277 INFO:tasks.workunit.client.0.smithi098.stderr: 62000K .......... .......... .......... .......... .......... 45% 28.4M 3s 2024-04-25T16:34:07.279 INFO:tasks.workunit.client.0.smithi098.stderr: 62050K .......... .......... .......... .......... .......... 45% 35.9M 3s 2024-04-25T16:34:07.281 INFO:tasks.workunit.client.0.smithi098.stderr: 62100K .......... .......... .......... .......... .......... 45% 29.5M 3s 2024-04-25T16:34:07.282 INFO:tasks.workunit.client.0.smithi098.stderr: 62150K .......... .......... .......... .......... .......... 45% 36.6M 3s 2024-04-25T16:34:07.283 INFO:tasks.workunit.client.0.smithi098.stderr: 62200K .......... .......... .......... .......... .......... 45% 35.4M 3s 2024-04-25T16:34:07.284 INFO:tasks.workunit.client.0.smithi098.stderr: 62250K .......... .......... .......... .......... .......... 45% 36.5M 3s 2024-04-25T16:34:07.286 INFO:tasks.workunit.client.0.smithi098.stderr: 62300K .......... .......... .......... .......... .......... 45% 31.2M 3s 2024-04-25T16:34:07.287 INFO:tasks.workunit.client.0.smithi098.stderr: 62350K .......... .......... .......... .......... .......... 45% 36.8M 3s 2024-04-25T16:34:07.289 INFO:tasks.workunit.client.0.smithi098.stderr: 62400K .......... .......... .......... .......... .......... 46% 36.1M 3s 2024-04-25T16:34:07.290 INFO:tasks.workunit.client.0.smithi098.stderr: 62450K .......... .......... .......... .......... .......... 46% 35.9M 3s 2024-04-25T16:34:07.292 INFO:tasks.workunit.client.0.smithi098.stderr: 62500K .......... .......... .......... .......... .......... 46% 30.5M 3s 2024-04-25T16:34:07.293 INFO:tasks.workunit.client.0.smithi098.stderr: 62550K .......... .......... .......... .......... .......... 46% 34.5M 3s 2024-04-25T16:34:07.294 INFO:tasks.workunit.client.0.smithi098.stderr: 62600K .......... .......... .......... .......... .......... 46% 35.7M 3s 2024-04-25T16:34:07.296 INFO:tasks.workunit.client.0.smithi098.stderr: 62650K .......... .......... .......... .......... .......... 46% 34.7M 3s 2024-04-25T16:34:07.297 INFO:tasks.workunit.client.0.smithi098.stderr: 62700K .......... .......... .......... .......... .......... 46% 30.7M 3s 2024-04-25T16:34:07.299 INFO:tasks.workunit.client.0.smithi098.stderr: 62750K .......... .......... .......... .......... .......... 46% 34.7M 3s 2024-04-25T16:34:07.300 INFO:tasks.workunit.client.0.smithi098.stderr: 62800K .......... .......... .......... .......... .......... 46% 34.4M 3s 2024-04-25T16:34:07.302 INFO:tasks.workunit.client.0.smithi098.stderr: 62850K .......... .......... .......... .......... .......... 46% 34.7M 3s 2024-04-25T16:34:07.303 INFO:tasks.workunit.client.0.smithi098.stderr: 62900K .......... .......... .......... .......... .......... 46% 29.3M 3s 2024-04-25T16:34:07.305 INFO:tasks.workunit.client.0.smithi098.stderr: 62950K .......... .......... .......... .......... .......... 46% 32.4M 3s 2024-04-25T16:34:07.306 INFO:tasks.workunit.client.0.smithi098.stderr: 63000K .......... .......... .......... .......... .......... 46% 33.6M 3s 2024-04-25T16:34:07.308 INFO:tasks.workunit.client.0.smithi098.stderr: 63050K .......... .......... .......... .......... .......... 46% 33.4M 3s 2024-04-25T16:34:07.310 INFO:tasks.workunit.client.0.smithi098.stderr: 63100K .......... .......... .......... .......... .......... 46% 28.1M 3s 2024-04-25T16:34:07.311 INFO:tasks.workunit.client.0.smithi098.stderr: 63150K .......... .......... .......... .......... .......... 46% 32.4M 3s 2024-04-25T16:34:07.313 INFO:tasks.workunit.client.0.smithi098.stderr: 63200K .......... .......... .......... .......... .......... 46% 32.3M 3s 2024-04-25T16:34:07.314 INFO:tasks.workunit.client.0.smithi098.stderr: 63250K .......... .......... .......... .......... .......... 46% 32.6M 3s 2024-04-25T16:34:07.316 INFO:tasks.workunit.client.0.smithi098.stderr: 63300K .......... .......... .......... .......... .......... 46% 27.6M 3s 2024-04-25T16:34:07.317 INFO:tasks.workunit.client.0.smithi098.stderr: 63350K .......... .......... .......... .......... .......... 46% 32.3M 3s 2024-04-25T16:34:07.319 INFO:tasks.workunit.client.0.smithi098.stderr: 63400K .......... .......... .......... .......... .......... 46% 31.8M 3s 2024-04-25T16:34:07.321 INFO:tasks.workunit.client.0.smithi098.stderr: 63450K .......... .......... .......... .......... .......... 46% 31.5M 3s 2024-04-25T16:34:07.322 INFO:tasks.workunit.client.0.smithi098.stderr: 63500K .......... .......... .......... .......... .......... 46% 27.1M 3s 2024-04-25T16:34:07.324 INFO:tasks.workunit.client.0.smithi098.stderr: 63550K .......... .......... .......... .......... .......... 46% 31.2M 3s 2024-04-25T16:34:07.325 INFO:tasks.workunit.client.0.smithi098.stderr: 63600K .......... .......... .......... .......... .......... 46% 31.8M 3s 2024-04-25T16:34:07.327 INFO:tasks.workunit.client.0.smithi098.stderr: 63650K .......... .......... .......... .......... .......... 46% 30.9M 3s 2024-04-25T16:34:07.329 INFO:tasks.workunit.client.0.smithi098.stderr: 63700K .......... .......... .......... .......... .......... 46% 26.3M 3s 2024-04-25T16:34:07.330 INFO:tasks.workunit.client.0.smithi098.stderr: 63750K .......... .......... .......... .......... .......... 46% 31.1M 3s 2024-04-25T16:34:07.332 INFO:tasks.workunit.client.0.smithi098.stderr: 63800K .......... .......... .......... .......... .......... 47% 31.1M 3s 2024-04-25T16:34:07.333 INFO:tasks.workunit.client.0.smithi098.stderr: 63850K .......... .......... .......... .......... .......... 47% 31.7M 3s 2024-04-25T16:34:07.335 INFO:tasks.workunit.client.0.smithi098.stderr: 63900K .......... .......... .......... .......... .......... 47% 26.0M 3s 2024-04-25T16:34:07.337 INFO:tasks.workunit.client.0.smithi098.stderr: 63950K .......... .......... .......... .......... .......... 47% 30.6M 3s 2024-04-25T16:34:07.339 INFO:tasks.workunit.client.0.smithi098.stderr: 64000K .......... .......... .......... .......... .......... 47% 30.5M 3s 2024-04-25T16:34:07.340 INFO:tasks.workunit.client.0.smithi098.stderr: 64050K .......... .......... .......... .......... .......... 47% 30.4M 3s 2024-04-25T16:34:07.342 INFO:tasks.workunit.client.0.smithi098.stderr: 64100K .......... .......... .......... .......... .......... 47% 25.3M 3s 2024-04-25T16:34:07.344 INFO:tasks.workunit.client.0.smithi098.stderr: 64150K .......... .......... .......... .......... .......... 47% 29.5M 3s 2024-04-25T16:34:07.345 INFO:tasks.workunit.client.0.smithi098.stderr: 64200K .......... .......... .......... .......... .......... 47% 30.2M 3s 2024-04-25T16:34:07.347 INFO:tasks.workunit.client.0.smithi098.stderr: 64250K .......... .......... .......... .......... .......... 47% 29.2M 3s 2024-04-25T16:34:07.349 INFO:tasks.workunit.client.0.smithi098.stderr: 64300K .......... .......... .......... .......... .......... 47% 25.4M 3s 2024-04-25T16:34:07.351 INFO:tasks.workunit.client.0.smithi098.stderr: 64350K .......... .......... .......... .......... .......... 47% 27.3M 3s 2024-04-25T16:34:07.352 INFO:tasks.workunit.client.0.smithi098.stderr: 64400K .......... .......... .......... .......... .......... 47% 29.0M 3s 2024-04-25T16:34:07.354 INFO:tasks.workunit.client.0.smithi098.stderr: 64450K .......... .......... .......... .......... .......... 47% 29.4M 3s 2024-04-25T16:34:07.356 INFO:tasks.workunit.client.0.smithi098.stderr: 64500K .......... .......... .......... .......... .......... 47% 22.7M 3s 2024-04-25T16:34:07.358 INFO:tasks.workunit.client.0.smithi098.stderr: 64550K .......... .......... .......... .......... .......... 47% 28.3M 3s 2024-04-25T16:34:07.360 INFO:tasks.workunit.client.0.smithi098.stderr: 64600K .......... .......... .......... .......... .......... 47% 29.0M 3s 2024-04-25T16:34:07.361 INFO:tasks.workunit.client.0.smithi098.stderr: 64650K .......... .......... .......... .......... .......... 47% 29.2M 3s 2024-04-25T16:34:07.363 INFO:tasks.workunit.client.0.smithi098.stderr: 64700K .......... .......... .......... .......... .......... 47% 24.5M 3s 2024-04-25T16:34:07.365 INFO:tasks.workunit.client.0.smithi098.stderr: 64750K .......... .......... .......... .......... .......... 47% 27.0M 3s 2024-04-25T16:34:07.367 INFO:tasks.workunit.client.0.smithi098.stderr: 64800K .......... .......... .......... .......... .......... 47% 27.8M 3s 2024-04-25T16:34:07.369 INFO:tasks.workunit.client.0.smithi098.stderr: 64850K .......... .......... .......... .......... .......... 47% 27.7M 3s 2024-04-25T16:34:07.371 INFO:tasks.workunit.client.0.smithi098.stderr: 64900K .......... .......... .......... .......... .......... 47% 23.5M 3s 2024-04-25T16:34:07.372 INFO:tasks.workunit.client.0.smithi098.stderr: 64950K .......... .......... .......... .......... .......... 47% 25.5M 3s 2024-04-25T16:34:07.374 INFO:tasks.workunit.client.0.smithi098.stderr: 65000K .......... .......... .......... .......... .......... 47% 26.5M 3s 2024-04-25T16:34:07.376 INFO:tasks.workunit.client.0.smithi098.stderr: 65050K .......... .......... .......... .......... .......... 47% 27.8M 3s 2024-04-25T16:34:07.378 INFO:tasks.workunit.client.0.smithi098.stderr: 65100K .......... .......... .......... .......... .......... 47% 21.4M 3s 2024-04-25T16:34:07.380 INFO:tasks.workunit.client.0.smithi098.stderr: 65150K .......... .......... .......... .......... .......... 48% 25.2M 3s 2024-04-25T16:34:07.382 INFO:tasks.workunit.client.0.smithi098.stderr: 65200K .......... .......... .......... .......... .......... 48% 24.4M 3s 2024-04-25T16:34:07.384 INFO:tasks.workunit.client.0.smithi098.stderr: 65250K .......... .......... .......... .......... .......... 48% 25.4M 3s 2024-04-25T16:34:07.387 INFO:tasks.workunit.client.0.smithi098.stderr: 65300K .......... .......... .......... .......... .......... 48% 20.2M 3s 2024-04-25T16:34:07.389 INFO:tasks.workunit.client.0.smithi098.stderr: 65350K .......... .......... .......... .......... .......... 48% 25.1M 3s 2024-04-25T16:34:07.391 INFO:tasks.workunit.client.0.smithi098.stderr: 65400K .......... .......... .......... .......... .......... 48% 25.1M 3s 2024-04-25T16:34:07.393 INFO:tasks.workunit.client.0.smithi098.stderr: 65450K .......... .......... .......... .......... .......... 48% 24.2M 3s 2024-04-25T16:34:07.395 INFO:tasks.workunit.client.0.smithi098.stderr: 65500K .......... .......... .......... .......... .......... 48% 20.2M 3s 2024-04-25T16:34:07.397 INFO:tasks.workunit.client.0.smithi098.stderr: 65550K .......... .......... .......... .......... .......... 48% 31.2M 3s 2024-04-25T16:34:07.398 INFO:tasks.workunit.client.0.smithi098.stderr: 65600K .......... .......... .......... .......... .......... 48% 40.6M 3s 2024-04-25T16:34:07.399 INFO:tasks.workunit.client.0.smithi098.stderr: 65650K .......... .......... .......... .......... .......... 48% 34.4M 3s 2024-04-25T16:34:07.401 INFO:tasks.workunit.client.0.smithi098.stderr: 65700K .......... .......... .......... .......... .......... 48% 29.1M 3s 2024-04-25T16:34:07.403 INFO:tasks.workunit.client.0.smithi098.stderr: 65750K .......... .......... .......... .......... .......... 48% 29.0M 3s 2024-04-25T16:34:07.405 INFO:tasks.workunit.client.0.smithi098.stderr: 65800K .......... .......... .......... .......... .......... 48% 22.1M 3s 2024-04-25T16:34:07.407 INFO:tasks.workunit.client.0.smithi098.stderr: 65850K .......... .......... .......... .......... .......... 48% 27.7M 3s 2024-04-25T16:34:07.408 INFO:tasks.workunit.client.0.smithi098.stderr: 65900K .......... .......... .......... .......... .......... 48% 27.8M 3s 2024-04-25T16:34:07.410 INFO:tasks.workunit.client.0.smithi098.stderr: 65950K .......... .......... .......... .......... .......... 48% 27.3M 3s 2024-04-25T16:34:07.412 INFO:tasks.workunit.client.0.smithi098.stderr: 66000K .......... .......... .......... .......... .......... 48% 31.4M 3s 2024-04-25T16:34:07.413 INFO:tasks.workunit.client.0.smithi098.stderr: 66050K .......... .......... .......... .......... .......... 48% 36.9M 3s 2024-04-25T16:34:07.415 INFO:tasks.workunit.client.0.smithi098.stderr: 66100K .......... .......... .......... .......... .......... 48% 31.2M 3s 2024-04-25T16:34:07.416 INFO:tasks.workunit.client.0.smithi098.stderr: 66150K .......... .......... .......... .......... .......... 48% 34.3M 3s 2024-04-25T16:34:07.417 INFO:tasks.workunit.client.0.smithi098.stderr: 66200K .......... .......... .......... .......... .......... 48% 34.6M 3s 2024-04-25T16:34:07.419 INFO:tasks.workunit.client.0.smithi098.stderr: 66250K .......... .......... .......... .......... .......... 48% 34.4M 3s 2024-04-25T16:34:07.420 INFO:tasks.workunit.client.0.smithi098.stderr: 66300K .......... .......... .......... .......... .......... 48% 29.5M 3s 2024-04-25T16:34:07.422 INFO:tasks.workunit.client.0.smithi098.stderr: 66350K .......... .......... .......... .......... .......... 48% 32.6M 3s 2024-04-25T16:34:07.424 INFO:tasks.workunit.client.0.smithi098.stderr: 66400K .......... .......... .......... .......... .......... 48% 30.8M 3s 2024-04-25T16:34:07.425 INFO:tasks.workunit.client.0.smithi098.stderr: 66450K .......... .......... .......... .......... .......... 48% 36.2M 3s 2024-04-25T16:34:07.426 INFO:tasks.workunit.client.0.smithi098.stderr: 66500K .......... .......... .......... .......... .......... 49% 29.7M 2s 2024-04-25T16:34:07.428 INFO:tasks.workunit.client.0.smithi098.stderr: 66550K .......... .......... .......... .......... .......... 49% 35.8M 2s 2024-04-25T16:34:07.429 INFO:tasks.workunit.client.0.smithi098.stderr: 66600K .......... .......... .......... .......... .......... 49% 33.9M 2s 2024-04-25T16:34:07.431 INFO:tasks.workunit.client.0.smithi098.stderr: 66650K .......... .......... .......... .......... .......... 49% 33.3M 2s 2024-04-25T16:34:07.432 INFO:tasks.workunit.client.0.smithi098.stderr: 66700K .......... .......... .......... .......... .......... 49% 29.6M 2s 2024-04-25T16:34:07.434 INFO:tasks.workunit.client.0.smithi098.stderr: 66750K .......... .......... .......... .......... .......... 49% 31.5M 2s 2024-04-25T16:34:07.435 INFO:tasks.workunit.client.0.smithi098.stderr: 66800K .......... .......... .......... .......... .......... 49% 34.2M 2s 2024-04-25T16:34:07.437 INFO:tasks.workunit.client.0.smithi098.stderr: 66850K .......... .......... .......... .......... .......... 49% 33.5M 2s 2024-04-25T16:34:07.439 INFO:tasks.workunit.client.0.smithi098.stderr: 66900K .......... .......... .......... .......... .......... 49% 28.8M 2s 2024-04-25T16:34:07.440 INFO:tasks.workunit.client.0.smithi098.stderr: 66950K .......... .......... .......... .......... .......... 49% 33.4M 2s 2024-04-25T16:34:07.442 INFO:tasks.workunit.client.0.smithi098.stderr: 67000K .......... .......... .......... .......... .......... 49% 32.6M 2s 2024-04-25T16:34:07.443 INFO:tasks.workunit.client.0.smithi098.stderr: 67050K .......... .......... .......... .......... .......... 49% 33.5M 2s 2024-04-25T16:34:07.445 INFO:tasks.workunit.client.0.smithi098.stderr: 67100K .......... .......... .......... .......... .......... 49% 28.4M 2s 2024-04-25T16:34:07.446 INFO:tasks.workunit.client.0.smithi098.stderr: 67150K .......... .......... .......... .......... .......... 49% 32.1M 2s 2024-04-25T16:34:07.448 INFO:tasks.workunit.client.0.smithi098.stderr: 67200K .......... .......... .......... .......... .......... 49% 29.8M 2s 2024-04-25T16:34:07.450 INFO:tasks.workunit.client.0.smithi098.stderr: 67250K .......... .......... .......... .......... .......... 49% 30.7M 2s 2024-04-25T16:34:07.451 INFO:tasks.workunit.client.0.smithi098.stderr: 67300K .......... .......... .......... .......... .......... 49% 25.7M 2s 2024-04-25T16:34:07.453 INFO:tasks.workunit.client.0.smithi098.stderr: 67350K .......... .......... .......... .......... .......... 49% 30.9M 2s 2024-04-25T16:34:07.455 INFO:tasks.workunit.client.0.smithi098.stderr: 67400K .......... .......... .......... .......... .......... 49% 31.9M 2s 2024-04-25T16:34:07.456 INFO:tasks.workunit.client.0.smithi098.stderr: 67450K .......... .......... .......... .......... .......... 49% 32.5M 2s 2024-04-25T16:34:07.458 INFO:tasks.workunit.client.0.smithi098.stderr: 67500K .......... .......... .......... .......... .......... 49% 27.2M 2s 2024-04-25T16:34:07.459 INFO:tasks.workunit.client.0.smithi098.stderr: 67550K .......... .......... .......... .......... .......... 49% 32.6M 2s 2024-04-25T16:34:07.461 INFO:tasks.workunit.client.0.smithi098.stderr: 67600K .......... .......... .......... .......... .......... 49% 31.5M 2s 2024-04-25T16:34:07.462 INFO:tasks.workunit.client.0.smithi098.stderr: 67650K .......... .......... .......... .......... .......... 49% 32.1M 2s 2024-04-25T16:34:07.464 INFO:tasks.workunit.client.0.smithi098.stderr: 67700K .......... .......... .......... .......... .......... 49% 27.3M 2s 2024-04-25T16:34:07.466 INFO:tasks.workunit.client.0.smithi098.stderr: 67750K .......... .......... .......... .......... .......... 49% 30.9M 2s 2024-04-25T16:34:07.467 INFO:tasks.workunit.client.0.smithi098.stderr: 67800K .......... .......... .......... .......... .......... 49% 31.1M 2s 2024-04-25T16:34:07.469 INFO:tasks.workunit.client.0.smithi098.stderr: 67850K .......... .......... .......... .......... .......... 50% 31.4M 2s 2024-04-25T16:34:07.471 INFO:tasks.workunit.client.0.smithi098.stderr: 67900K .......... .......... .......... .......... .......... 50% 26.6M 2s 2024-04-25T16:34:07.472 INFO:tasks.workunit.client.0.smithi098.stderr: 67950K .......... .......... .......... .......... .......... 50% 30.0M 2s 2024-04-25T16:34:07.474 INFO:tasks.workunit.client.0.smithi098.stderr: 68000K .......... .......... .......... .......... .......... 50% 30.5M 2s 2024-04-25T16:34:07.476 INFO:tasks.workunit.client.0.smithi098.stderr: 68050K .......... .......... .......... .......... .......... 50% 31.0M 2s 2024-04-25T16:34:07.477 INFO:tasks.workunit.client.0.smithi098.stderr: 68100K .......... .......... .......... .......... .......... 50% 26.0M 2s 2024-04-25T16:34:07.479 INFO:tasks.workunit.client.0.smithi098.stderr: 68150K .......... .......... .......... .......... .......... 50% 29.8M 2s 2024-04-25T16:34:07.481 INFO:tasks.workunit.client.0.smithi098.stderr: 68200K .......... .......... .......... .......... .......... 50% 28.6M 2s 2024-04-25T16:34:07.482 INFO:tasks.workunit.client.0.smithi098.stderr: 68250K .......... .......... .......... .......... .......... 50% 29.3M 2s 2024-04-25T16:34:07.484 INFO:tasks.workunit.client.0.smithi098.stderr: 68300K .......... .......... .......... .......... .......... 50% 25.2M 2s 2024-04-25T16:34:07.486 INFO:tasks.workunit.client.0.smithi098.stderr: 68350K .......... .......... .......... .......... .......... 50% 29.8M 2s 2024-04-25T16:34:07.488 INFO:tasks.workunit.client.0.smithi098.stderr: 68400K .......... .......... .......... .......... .......... 50% 30.3M 2s 2024-04-25T16:34:07.489 INFO:tasks.workunit.client.0.smithi098.stderr: 68450K .......... .......... .......... .......... .......... 50% 30.1M 2s 2024-04-25T16:34:07.491 INFO:tasks.workunit.client.0.smithi098.stderr: 68500K .......... .......... .......... .......... .......... 50% 25.5M 2s 2024-04-25T16:34:07.492 INFO:tasks.workunit.client.0.smithi098.stderr: 68550K .......... .......... .......... .......... .......... 50% 30.8M 2s 2024-04-25T16:34:07.494 INFO:tasks.workunit.client.0.smithi098.stderr: 68600K .......... .......... .......... .......... .......... 50% 30.3M 2s 2024-04-25T16:34:07.496 INFO:tasks.workunit.client.0.smithi098.stderr: 68650K .......... .......... .......... .......... .......... 50% 30.7M 2s 2024-04-25T16:34:07.498 INFO:tasks.workunit.client.0.smithi098.stderr: 68700K .......... .......... .......... .......... .......... 50% 25.5M 2s 2024-04-25T16:34:07.499 INFO:tasks.workunit.client.0.smithi098.stderr: 68750K .......... .......... .......... .......... .......... 50% 28.8M 2s 2024-04-25T16:34:07.502 INFO:tasks.workunit.client.0.smithi098.stderr: 68800K .......... .......... .......... .......... .......... 50% 25.6M 2s 2024-04-25T16:34:07.503 INFO:tasks.workunit.client.0.smithi098.stderr: 68850K .......... .......... .......... .......... .......... 50% 26.3M 2s 2024-04-25T16:34:07.505 INFO:tasks.workunit.client.0.smithi098.stderr: 68900K .......... .......... .......... .......... .......... 50% 22.2M 2s 2024-04-25T16:34:07.507 INFO:tasks.workunit.client.0.smithi098.stderr: 68950K .......... .......... .......... .......... .......... 50% 24.9M 2s 2024-04-25T16:34:07.510 INFO:tasks.workunit.client.0.smithi098.stderr: 69000K .......... .......... .......... .......... .......... 50% 25.5M 2s 2024-04-25T16:34:07.511 INFO:tasks.workunit.client.0.smithi098.stderr: 69050K .......... .......... .......... .......... .......... 50% 25.6M 2s 2024-04-25T16:34:07.513 INFO:tasks.workunit.client.0.smithi098.stderr: 69100K .......... .......... .......... .......... .......... 50% 22.3M 2s 2024-04-25T16:34:07.515 INFO:tasks.workunit.client.0.smithi098.stderr: 69150K .......... .......... .......... .......... .......... 50% 24.9M 2s 2024-04-25T16:34:07.517 INFO:tasks.workunit.client.0.smithi098.stderr: 69200K .......... .......... .......... .......... .......... 51% 25.0M 2s 2024-04-25T16:34:07.520 INFO:tasks.workunit.client.0.smithi098.stderr: 69250K .......... .......... .......... .......... .......... 51% 25.0M 2s 2024-04-25T16:34:07.522 INFO:tasks.workunit.client.0.smithi098.stderr: 69300K .......... .......... .......... .......... .......... 51% 21.0M 2s 2024-04-25T16:34:07.523 INFO:tasks.workunit.client.0.smithi098.stderr: 69350K .......... .......... .......... .......... .......... 51% 25.4M 2s 2024-04-25T16:34:07.525 INFO:tasks.workunit.client.0.smithi098.stderr: 69400K .......... .......... .......... .......... .......... 51% 25.3M 2s 2024-04-25T16:34:07.528 INFO:tasks.workunit.client.0.smithi098.stderr: 69450K .......... .......... .......... .......... .......... 51% 23.8M 2s 2024-04-25T16:34:07.530 INFO:tasks.workunit.client.0.smithi098.stderr: 69500K .......... .......... .......... .......... .......... 51% 21.4M 2s 2024-04-25T16:34:07.532 INFO:tasks.workunit.client.0.smithi098.stderr: 69550K .......... .......... .......... .......... .......... 51% 23.3M 2s 2024-04-25T16:34:07.534 INFO:tasks.workunit.client.0.smithi098.stderr: 69600K .......... .......... .......... .......... .......... 51% 23.0M 2s 2024-04-25T16:34:07.535 INFO:tasks.workunit.client.0.smithi098.stderr: 69650K .......... .......... .......... .......... .......... 51% 37.6M 2s 2024-04-25T16:34:07.537 INFO:tasks.workunit.client.0.smithi098.stderr: 69700K .......... .......... .......... .......... .......... 51% 32.9M 2s 2024-04-25T16:34:07.538 INFO:tasks.workunit.client.0.smithi098.stderr: 69750K .......... .......... .......... .......... .......... 51% 33.5M 2s 2024-04-25T16:34:07.540 INFO:tasks.workunit.client.0.smithi098.stderr: 69800K .......... .......... .......... .......... .......... 51% 34.1M 2s 2024-04-25T16:34:07.541 INFO:tasks.workunit.client.0.smithi098.stderr: 69850K .......... .......... .......... .......... .......... 51% 36.5M 2s 2024-04-25T16:34:07.543 INFO:tasks.workunit.client.0.smithi098.stderr: 69900K .......... .......... .......... .......... .......... 51% 30.6M 2s 2024-04-25T16:34:07.544 INFO:tasks.workunit.client.0.smithi098.stderr: 69950K .......... .......... .......... .......... .......... 51% 37.3M 2s 2024-04-25T16:34:07.545 INFO:tasks.workunit.client.0.smithi098.stderr: 70000K .......... .......... .......... .......... .......... 51% 37.2M 2s 2024-04-25T16:34:07.547 INFO:tasks.workunit.client.0.smithi098.stderr: 70050K .......... .......... .......... .......... .......... 51% 36.1M 2s 2024-04-25T16:34:07.549 INFO:tasks.workunit.client.0.smithi098.stderr: 70100K .......... .......... .......... .......... .......... 51% 24.0M 2s 2024-04-25T16:34:07.550 INFO:tasks.workunit.client.0.smithi098.stderr: 70150K .......... .......... .......... .......... .......... 51% 29.7M 2s 2024-04-25T16:34:07.552 INFO:tasks.workunit.client.0.smithi098.stderr: 70200K .......... .......... .......... .......... .......... 51% 26.9M 2s 2024-04-25T16:34:07.554 INFO:tasks.workunit.client.0.smithi098.stderr: 70250K .......... .......... .......... .......... .......... 51% 27.6M 2s 2024-04-25T16:34:07.556 INFO:tasks.workunit.client.0.smithi098.stderr: 70300K .......... .......... .......... .......... .......... 51% 23.9M 2s 2024-04-25T16:34:07.558 INFO:tasks.workunit.client.0.smithi098.stderr: 70350K .......... .......... .......... .......... .......... 51% 29.2M 2s 2024-04-25T16:34:07.559 INFO:tasks.workunit.client.0.smithi098.stderr: 70400K .......... .......... .......... .......... .......... 51% 29.0M 2s 2024-04-25T16:34:07.561 INFO:tasks.workunit.client.0.smithi098.stderr: 70450K .......... .......... .......... .......... .......... 51% 28.9M 2s 2024-04-25T16:34:07.563 INFO:tasks.workunit.client.0.smithi098.stderr: 70500K .......... .......... .......... .......... .......... 51% 25.1M 2s 2024-04-25T16:34:07.565 INFO:tasks.workunit.client.0.smithi098.stderr: 70550K .......... .......... .......... .......... .......... 52% 28.4M 2s 2024-04-25T16:34:07.566 INFO:tasks.workunit.client.0.smithi098.stderr: 70600K .......... .......... .......... .......... .......... 52% 28.6M 2s 2024-04-25T16:34:07.568 INFO:tasks.workunit.client.0.smithi098.stderr: 70650K .......... .......... .......... .......... .......... 52% 28.5M 2s 2024-04-25T16:34:07.570 INFO:tasks.workunit.client.0.smithi098.stderr: 70700K .......... .......... .......... .......... .......... 52% 24.4M 2s 2024-04-25T16:34:07.572 INFO:tasks.workunit.client.0.smithi098.stderr: 70750K .......... .......... .......... .......... .......... 52% 27.8M 2s 2024-04-25T16:34:07.573 INFO:tasks.workunit.client.0.smithi098.stderr: 70800K .......... .......... .......... .......... .......... 52% 31.2M 2s 2024-04-25T16:34:07.575 INFO:tasks.workunit.client.0.smithi098.stderr: 70850K .......... .......... .......... .......... .......... 52% 34.4M 2s 2024-04-25T16:34:07.577 INFO:tasks.workunit.client.0.smithi098.stderr: 70900K .......... .......... .......... .......... .......... 52% 29.8M 2s 2024-04-25T16:34:07.578 INFO:tasks.workunit.client.0.smithi098.stderr: 70950K .......... .......... .......... .......... .......... 52% 32.3M 2s 2024-04-25T16:34:07.579 INFO:tasks.workunit.client.0.smithi098.stderr: 71000K .......... .......... .......... .......... .......... 52% 35.6M 2s 2024-04-25T16:34:07.581 INFO:tasks.workunit.client.0.smithi098.stderr: 71050K .......... .......... .......... .......... .......... 52% 35.3M 2s 2024-04-25T16:34:07.582 INFO:tasks.workunit.client.0.smithi098.stderr: 71100K .......... .......... .......... .......... .......... 52% 29.1M 2s 2024-04-25T16:34:07.584 INFO:tasks.workunit.client.0.smithi098.stderr: 71150K .......... .......... .......... .......... .......... 52% 33.7M 2s 2024-04-25T16:34:07.585 INFO:tasks.workunit.client.0.smithi098.stderr: 71200K .......... .......... .......... .......... .......... 52% 33.9M 2s 2024-04-25T16:34:07.587 INFO:tasks.workunit.client.0.smithi098.stderr: 71250K .......... .......... .......... .......... .......... 52% 34.3M 2s 2024-04-25T16:34:07.588 INFO:tasks.workunit.client.0.smithi098.stderr: 71300K .......... .......... .......... .......... .......... 52% 30.5M 2s 2024-04-25T16:34:07.590 INFO:tasks.workunit.client.0.smithi098.stderr: 71350K .......... .......... .......... .......... .......... 52% 34.8M 2s 2024-04-25T16:34:07.591 INFO:tasks.workunit.client.0.smithi098.stderr: 71400K .......... .......... .......... .......... .......... 52% 34.8M 2s 2024-04-25T16:34:07.592 INFO:tasks.workunit.client.0.smithi098.stderr: 71450K .......... .......... .......... .......... .......... 52% 35.3M 2s 2024-04-25T16:34:07.594 INFO:tasks.workunit.client.0.smithi098.stderr: 71500K .......... .......... .......... .......... .......... 52% 29.0M 2s 2024-04-25T16:34:07.596 INFO:tasks.workunit.client.0.smithi098.stderr: 71550K .......... .......... .......... .......... .......... 52% 32.9M 2s 2024-04-25T16:34:07.597 INFO:tasks.workunit.client.0.smithi098.stderr: 71600K .......... .......... .......... .......... .......... 52% 33.7M 2s 2024-04-25T16:34:07.599 INFO:tasks.workunit.client.0.smithi098.stderr: 71650K .......... .......... .......... .......... .......... 52% 33.6M 2s 2024-04-25T16:34:07.600 INFO:tasks.workunit.client.0.smithi098.stderr: 71700K .......... .......... .......... .......... .......... 52% 28.5M 2s 2024-04-25T16:34:07.601 INFO:tasks.workunit.client.0.smithi098.stderr: 71750K .......... .......... .......... .......... .......... 52% 33.8M 2s 2024-04-25T16:34:07.603 INFO:tasks.workunit.client.0.smithi098.stderr: 71800K .......... .......... .......... .......... .......... 52% 32.7M 2s 2024-04-25T16:34:07.605 INFO:tasks.workunit.client.0.smithi098.stderr: 71850K .......... .......... .......... .......... .......... 52% 31.0M 2s 2024-04-25T16:34:07.607 INFO:tasks.workunit.client.0.smithi098.stderr: 71900K .......... .......... .......... .......... .......... 53% 23.9M 2s 2024-04-25T16:34:07.608 INFO:tasks.workunit.client.0.smithi098.stderr: 71950K .......... .......... .......... .......... .......... 53% 31.5M 2s 2024-04-25T16:34:07.610 INFO:tasks.workunit.client.0.smithi098.stderr: 72000K .......... .......... .......... .......... .......... 53% 31.5M 2s 2024-04-25T16:34:07.612 INFO:tasks.workunit.client.0.smithi098.stderr: 72050K .......... .......... .......... .......... .......... 53% 31.5M 2s 2024-04-25T16:34:07.613 INFO:tasks.workunit.client.0.smithi098.stderr: 72100K .......... .......... .......... .......... .......... 53% 26.9M 2s 2024-04-25T16:34:07.615 INFO:tasks.workunit.client.0.smithi098.stderr: 72150K .......... .......... .......... .......... .......... 53% 30.8M 2s 2024-04-25T16:34:07.616 INFO:tasks.workunit.client.0.smithi098.stderr: 72200K .......... .......... .......... .......... .......... 53% 29.8M 2s 2024-04-25T16:34:07.618 INFO:tasks.workunit.client.0.smithi098.stderr: 72250K .......... .......... .......... .......... .......... 53% 31.0M 2s 2024-04-25T16:34:07.620 INFO:tasks.workunit.client.0.smithi098.stderr: 72300K .......... .......... .......... .......... .......... 53% 24.3M 2s 2024-04-25T16:34:07.622 INFO:tasks.workunit.client.0.smithi098.stderr: 72350K .......... .......... .......... .......... .......... 53% 29.0M 2s 2024-04-25T16:34:07.623 INFO:tasks.workunit.client.0.smithi098.stderr: 72400K .......... .......... .......... .......... .......... 53% 30.3M 2s 2024-04-25T16:34:07.625 INFO:tasks.workunit.client.0.smithi098.stderr: 72450K .......... .......... .......... .......... .......... 53% 30.3M 2s 2024-04-25T16:34:07.627 INFO:tasks.workunit.client.0.smithi098.stderr: 72500K .......... .......... .......... .......... .......... 53% 25.7M 2s 2024-04-25T16:34:07.629 INFO:tasks.workunit.client.0.smithi098.stderr: 72550K .......... .......... .......... .......... .......... 53% 30.9M 2s 2024-04-25T16:34:07.630 INFO:tasks.workunit.client.0.smithi098.stderr: 72600K .......... .......... .......... .......... .......... 53% 29.9M 2s 2024-04-25T16:34:07.632 INFO:tasks.workunit.client.0.smithi098.stderr: 72650K .......... .......... .......... .......... .......... 53% 29.6M 2s 2024-04-25T16:34:07.634 INFO:tasks.workunit.client.0.smithi098.stderr: 72700K .......... .......... .......... .......... .......... 53% 25.9M 2s 2024-04-25T16:34:07.635 INFO:tasks.workunit.client.0.smithi098.stderr: 72750K .......... .......... .......... .......... .......... 53% 27.6M 2s 2024-04-25T16:34:07.637 INFO:tasks.workunit.client.0.smithi098.stderr: 72800K .......... .......... .......... .......... .......... 53% 29.0M 2s 2024-04-25T16:34:07.639 INFO:tasks.workunit.client.0.smithi098.stderr: 72850K .......... .......... .......... .......... .......... 53% 29.6M 2s 2024-04-25T16:34:07.641 INFO:tasks.workunit.client.0.smithi098.stderr: 72900K .......... .......... .......... .......... .......... 53% 25.4M 2s 2024-04-25T16:34:07.642 INFO:tasks.workunit.client.0.smithi098.stderr: 72950K .......... .......... .......... .......... .......... 53% 29.4M 2s 2024-04-25T16:34:07.644 INFO:tasks.workunit.client.0.smithi098.stderr: 73000K .......... .......... .......... .......... .......... 53% 28.7M 2s 2024-04-25T16:34:07.646 INFO:tasks.workunit.client.0.smithi098.stderr: 73050K .......... .......... .......... .......... .......... 53% 29.2M 2s 2024-04-25T16:34:07.648 INFO:tasks.workunit.client.0.smithi098.stderr: 73100K .......... .......... .......... .......... .......... 53% 24.3M 2s 2024-04-25T16:34:07.649 INFO:tasks.workunit.client.0.smithi098.stderr: 73150K .......... .......... .......... .......... .......... 53% 28.8M 2s 2024-04-25T16:34:07.651 INFO:tasks.workunit.client.0.smithi098.stderr: 73200K .......... .......... .......... .......... .......... 53% 29.0M 2s 2024-04-25T16:34:07.653 INFO:tasks.workunit.client.0.smithi098.stderr: 73250K .......... .......... .......... .......... .......... 53% 27.8M 2s 2024-04-25T16:34:07.655 INFO:tasks.workunit.client.0.smithi098.stderr: 73300K .......... .......... .......... .......... .......... 54% 24.3M 2s 2024-04-25T16:34:07.657 INFO:tasks.workunit.client.0.smithi098.stderr: 73350K .......... .......... .......... .......... .......... 54% 26.2M 2s 2024-04-25T16:34:07.658 INFO:tasks.workunit.client.0.smithi098.stderr: 73400K .......... .......... .......... .......... .......... 54% 26.7M 2s 2024-04-25T16:34:07.661 INFO:tasks.workunit.client.0.smithi098.stderr: 73450K .......... .......... .......... .......... .......... 54% 27.0M 2s 2024-04-25T16:34:07.662 INFO:tasks.workunit.client.0.smithi098.stderr: 73500K .......... .......... .......... .......... .......... 54% 23.1M 2s 2024-04-25T16:34:07.664 INFO:tasks.workunit.client.0.smithi098.stderr: 73550K .......... .......... .......... .......... .......... 54% 26.3M 2s 2024-04-25T16:34:07.666 INFO:tasks.workunit.client.0.smithi098.stderr: 73600K .......... .......... .......... .......... .......... 54% 26.5M 2s 2024-04-25T16:34:07.668 INFO:tasks.workunit.client.0.smithi098.stderr: 73650K .......... .......... .......... .......... .......... 54% 26.1M 2s 2024-04-25T16:34:07.670 INFO:tasks.workunit.client.0.smithi098.stderr: 73700K .......... .......... .......... .......... .......... 54% 22.0M 2s 2024-04-25T16:34:07.671 INFO:tasks.workunit.client.0.smithi098.stderr: 73750K .......... .......... .......... .......... .......... 54% 41.6M 2s 2024-04-25T16:34:07.673 INFO:tasks.workunit.client.0.smithi098.stderr: 73800K .......... .......... .......... .......... .......... 54% 33.7M 2s 2024-04-25T16:34:07.674 INFO:tasks.workunit.client.0.smithi098.stderr: 73850K .......... .......... .......... .......... .......... 54% 39.8M 2s 2024-04-25T16:34:07.676 INFO:tasks.workunit.client.0.smithi098.stderr: 73900K .......... .......... .......... .......... .......... 54% 32.4M 2s 2024-04-25T16:34:07.677 INFO:tasks.workunit.client.0.smithi098.stderr: 73950K .......... .......... .......... .......... .......... 54% 37.8M 2s 2024-04-25T16:34:07.678 INFO:tasks.workunit.client.0.smithi098.stderr: 74000K .......... .......... .......... .......... .......... 54% 35.9M 2s 2024-04-25T16:34:07.680 INFO:tasks.workunit.client.0.smithi098.stderr: 74050K .......... .......... .......... .......... .......... 54% 37.5M 2s 2024-04-25T16:34:07.681 INFO:tasks.workunit.client.0.smithi098.stderr: 74100K .......... .......... .......... .......... .......... 54% 32.2M 2s 2024-04-25T16:34:07.682 INFO:tasks.workunit.client.0.smithi098.stderr: 74150K .......... .......... .......... .......... .......... 54% 37.7M 2s 2024-04-25T16:34:07.684 INFO:tasks.workunit.client.0.smithi098.stderr: 74200K .......... .......... .......... .......... .......... 54% 38.6M 2s 2024-04-25T16:34:07.685 INFO:tasks.workunit.client.0.smithi098.stderr: 74250K .......... .......... .......... .......... .......... 54% 38.9M 2s 2024-04-25T16:34:07.686 INFO:tasks.workunit.client.0.smithi098.stderr: 74300K .......... .......... .......... .......... .......... 54% 32.3M 2s 2024-04-25T16:34:07.688 INFO:tasks.workunit.client.0.smithi098.stderr: 74350K .......... .......... .......... .......... .......... 54% 37.7M 2s 2024-04-25T16:34:07.689 INFO:tasks.workunit.client.0.smithi098.stderr: 74400K .......... .......... .......... .......... .......... 54% 34.7M 2s 2024-04-25T16:34:07.691 INFO:tasks.workunit.client.0.smithi098.stderr: 74450K .......... .......... .......... .......... .......... 54% 30.4M 2s 2024-04-25T16:34:07.692 INFO:tasks.workunit.client.0.smithi098.stderr: 74500K .......... .......... .......... .......... .......... 54% 30.8M 2s 2024-04-25T16:34:07.694 INFO:tasks.workunit.client.0.smithi098.stderr: 74550K .......... .......... .......... .......... .......... 54% 37.0M 2s 2024-04-25T16:34:07.695 INFO:tasks.workunit.client.0.smithi098.stderr: 74600K .......... .......... .......... .......... .......... 54% 34.0M 2s 2024-04-25T16:34:07.696 INFO:tasks.workunit.client.0.smithi098.stderr: 74650K .......... .......... .......... .......... .......... 55% 35.0M 2s 2024-04-25T16:34:07.698 INFO:tasks.workunit.client.0.smithi098.stderr: 74700K .......... .......... .......... .......... .......... 55% 30.2M 2s 2024-04-25T16:34:07.700 INFO:tasks.workunit.client.0.smithi098.stderr: 74750K .......... .......... .......... .......... .......... 55% 35.8M 2s 2024-04-25T16:34:07.701 INFO:tasks.workunit.client.0.smithi098.stderr: 74800K .......... .......... .......... .......... .......... 55% 31.5M 2s 2024-04-25T16:34:07.703 INFO:tasks.workunit.client.0.smithi098.stderr: 74850K .......... .......... .......... .......... .......... 55% 31.3M 2s 2024-04-25T16:34:07.704 INFO:tasks.workunit.client.0.smithi098.stderr: 74900K .......... .......... .......... .......... .......... 55% 27.2M 2s 2024-04-25T16:34:07.706 INFO:tasks.workunit.client.0.smithi098.stderr: 74950K .......... .......... .......... .......... .......... 55% 31.7M 2s 2024-04-25T16:34:07.707 INFO:tasks.workunit.client.0.smithi098.stderr: 75000K .......... .......... .......... .......... .......... 55% 31.7M 2s 2024-04-25T16:34:07.710 INFO:tasks.workunit.client.0.smithi098.stderr: 75050K .......... .......... .......... .......... .......... 55% 32.8M 2s 2024-04-25T16:34:07.711 INFO:tasks.workunit.client.0.smithi098.stderr: 75100K .......... .......... .......... .......... .......... 55% 27.8M 2s 2024-04-25T16:34:07.712 INFO:tasks.workunit.client.0.smithi098.stderr: 75150K .......... .......... .......... .......... .......... 55% 31.3M 2s 2024-04-25T16:34:07.714 INFO:tasks.workunit.client.0.smithi098.stderr: 75200K .......... .......... .......... .......... .......... 55% 31.4M 2s 2024-04-25T16:34:07.715 INFO:tasks.workunit.client.0.smithi098.stderr: 75250K .......... .......... .......... .......... .......... 55% 31.7M 2s 2024-04-25T16:34:07.717 INFO:tasks.workunit.client.0.smithi098.stderr: 75300K .......... .......... .......... .......... .......... 55% 26.3M 2s 2024-04-25T16:34:07.719 INFO:tasks.workunit.client.0.smithi098.stderr: 75350K .......... .......... .......... .......... .......... 55% 31.3M 2s 2024-04-25T16:34:07.720 INFO:tasks.workunit.client.0.smithi098.stderr: 75400K .......... .......... .......... .......... .......... 55% 29.1M 2s 2024-04-25T16:34:07.722 INFO:tasks.workunit.client.0.smithi098.stderr: 75450K .......... .......... .......... .......... .......... 55% 29.9M 2s 2024-04-25T16:34:07.724 INFO:tasks.workunit.client.0.smithi098.stderr: 75500K .......... .......... .......... .......... .......... 55% 26.1M 2s 2024-04-25T16:34:07.726 INFO:tasks.workunit.client.0.smithi098.stderr: 75550K .......... .......... .......... .......... .......... 55% 31.2M 2s 2024-04-25T16:34:07.727 INFO:tasks.workunit.client.0.smithi098.stderr: 75600K .......... .......... .......... .......... .......... 55% 29.3M 2s 2024-04-25T16:34:07.729 INFO:tasks.workunit.client.0.smithi098.stderr: 75650K .......... .......... .......... .......... .......... 55% 30.5M 2s 2024-04-25T16:34:07.730 INFO:tasks.workunit.client.0.smithi098.stderr: 75700K .......... .......... .......... .......... .......... 55% 27.3M 2s 2024-04-25T16:34:07.732 INFO:tasks.workunit.client.0.smithi098.stderr: 75750K .......... .......... .......... .......... .......... 55% 31.7M 2s 2024-04-25T16:34:07.734 INFO:tasks.workunit.client.0.smithi098.stderr: 75800K .......... .......... .......... .......... .......... 55% 32.1M 2s 2024-04-25T16:34:07.735 INFO:tasks.workunit.client.0.smithi098.stderr: 75850K .......... .......... .......... .......... .......... 55% 32.3M 2s 2024-04-25T16:34:07.737 INFO:tasks.workunit.client.0.smithi098.stderr: 75900K .......... .......... .......... .......... .......... 55% 26.3M 2s 2024-04-25T16:34:07.739 INFO:tasks.workunit.client.0.smithi098.stderr: 75950K .......... .......... .......... .......... .......... 55% 32.0M 2s 2024-04-25T16:34:07.741 INFO:tasks.workunit.client.0.smithi098.stderr: 76000K .......... .......... .......... .......... .......... 56% 24.8M 2s 2024-04-25T16:34:07.742 INFO:tasks.workunit.client.0.smithi098.stderr: 76050K .......... .......... .......... .......... .......... 56% 29.8M 2s 2024-04-25T16:34:07.744 INFO:tasks.workunit.client.0.smithi098.stderr: 76100K .......... .......... .......... .......... .......... 56% 27.1M 2s 2024-04-25T16:34:07.746 INFO:tasks.workunit.client.0.smithi098.stderr: 76150K .......... .......... .......... .......... .......... 56% 31.4M 2s 2024-04-25T16:34:07.747 INFO:tasks.workunit.client.0.smithi098.stderr: 76200K .......... .......... .......... .......... .......... 56% 28.0M 2s 2024-04-25T16:34:07.749 INFO:tasks.workunit.client.0.smithi098.stderr: 76250K .......... .......... .......... .......... .......... 56% 27.4M 2s 2024-04-25T16:34:07.751 INFO:tasks.workunit.client.0.smithi098.stderr: 76300K .......... .......... .......... .......... .......... 56% 23.4M 2s 2024-04-25T16:34:07.753 INFO:tasks.workunit.client.0.smithi098.stderr: 76350K .......... .......... .......... .......... .......... 56% 27.9M 2s 2024-04-25T16:34:07.755 INFO:tasks.workunit.client.0.smithi098.stderr: 76400K .......... .......... .......... .......... .......... 56% 28.0M 2s 2024-04-25T16:34:07.756 INFO:tasks.workunit.client.0.smithi098.stderr: 76450K .......... .......... .......... .......... .......... 56% 29.9M 2s 2024-04-25T16:34:07.758 INFO:tasks.workunit.client.0.smithi098.stderr: 76500K .......... .......... .......... .......... .......... 56% 25.4M 2s 2024-04-25T16:34:07.760 INFO:tasks.workunit.client.0.smithi098.stderr: 76550K .......... .......... .......... .......... .......... 56% 30.1M 2s 2024-04-25T16:34:07.761 INFO:tasks.workunit.client.0.smithi098.stderr: 76600K .......... .......... .......... .......... .......... 56% 29.7M 2s 2024-04-25T16:34:07.763 INFO:tasks.workunit.client.0.smithi098.stderr: 76650K .......... .......... .......... .......... .......... 56% 29.7M 2s 2024-04-25T16:34:07.765 INFO:tasks.workunit.client.0.smithi098.stderr: 76700K .......... .......... .......... .......... .......... 56% 25.4M 2s 2024-04-25T16:34:07.767 INFO:tasks.workunit.client.0.smithi098.stderr: 76750K .......... .......... .......... .......... .......... 56% 29.6M 2s 2024-04-25T16:34:07.768 INFO:tasks.workunit.client.0.smithi098.stderr: 76800K .......... .......... .......... .......... .......... 56% 29.5M 2s 2024-04-25T16:34:07.770 INFO:tasks.workunit.client.0.smithi098.stderr: 76850K .......... .......... .......... .......... .......... 56% 29.1M 2s 2024-04-25T16:34:07.772 INFO:tasks.workunit.client.0.smithi098.stderr: 76900K .......... .......... .......... .......... .......... 56% 23.8M 2s 2024-04-25T16:34:07.774 INFO:tasks.workunit.client.0.smithi098.stderr: 76950K .......... .......... .......... .......... .......... 56% 28.5M 2s 2024-04-25T16:34:07.775 INFO:tasks.workunit.client.0.smithi098.stderr: 77000K .......... .......... .......... .......... .......... 56% 28.4M 2s 2024-04-25T16:34:07.777 INFO:tasks.workunit.client.0.smithi098.stderr: 77050K .......... .......... .......... .......... .......... 56% 28.2M 2s 2024-04-25T16:34:07.779 INFO:tasks.workunit.client.0.smithi098.stderr: 77100K .......... .......... .......... .......... .......... 56% 23.8M 2s 2024-04-25T16:34:07.781 INFO:tasks.workunit.client.0.smithi098.stderr: 77150K .......... .......... .......... .......... .......... 56% 27.3M 2s 2024-04-25T16:34:07.783 INFO:tasks.workunit.client.0.smithi098.stderr: 77200K .......... .......... .......... .......... .......... 56% 27.8M 2s 2024-04-25T16:34:07.785 INFO:tasks.workunit.client.0.smithi098.stderr: 77250K .......... .......... .......... .......... .......... 56% 27.1M 2s 2024-04-25T16:34:07.787 INFO:tasks.workunit.client.0.smithi098.stderr: 77300K .......... .......... .......... .......... .......... 56% 22.7M 2s 2024-04-25T16:34:07.788 INFO:tasks.workunit.client.0.smithi098.stderr: 77350K .......... .......... .......... .......... .......... 57% 27.4M 2s 2024-04-25T16:34:07.790 INFO:tasks.workunit.client.0.smithi098.stderr: 77400K .......... .......... .......... .......... .......... 57% 27.7M 2s 2024-04-25T16:34:07.792 INFO:tasks.workunit.client.0.smithi098.stderr: 77450K .......... .......... .......... .......... .......... 57% 27.2M 2s 2024-04-25T16:34:07.794 INFO:tasks.workunit.client.0.smithi098.stderr: 77500K .......... .......... .......... .......... .......... 57% 23.1M 2s 2024-04-25T16:34:07.796 INFO:tasks.workunit.client.0.smithi098.stderr: 77550K .......... .......... .......... .......... .......... 57% 27.2M 2s 2024-04-25T16:34:07.798 INFO:tasks.workunit.client.0.smithi098.stderr: 77600K .......... .......... .......... .......... .......... 57% 26.8M 2s 2024-04-25T16:34:07.800 INFO:tasks.workunit.client.0.smithi098.stderr: 77650K .......... .......... .......... .......... .......... 57% 27.8M 2s 2024-04-25T16:34:07.802 INFO:tasks.workunit.client.0.smithi098.stderr: 77700K .......... .......... .......... .......... .......... 57% 23.2M 2s 2024-04-25T16:34:07.803 INFO:tasks.workunit.client.0.smithi098.stderr: 77750K .......... .......... .......... .......... .......... 57% 26.5M 2s 2024-04-25T16:34:07.805 INFO:tasks.workunit.client.0.smithi098.stderr: 77800K .......... .......... .......... .......... .......... 57% 26.3M 2s 2024-04-25T16:34:07.807 INFO:tasks.workunit.client.0.smithi098.stderr: 77850K .......... .......... .......... .......... .......... 57% 37.3M 2s 2024-04-25T16:34:07.808 INFO:tasks.workunit.client.0.smithi098.stderr: 77900K .......... .......... .......... .......... .......... 57% 33.1M 2s 2024-04-25T16:34:07.809 INFO:tasks.workunit.client.0.smithi098.stderr: 77950K .......... .......... .......... .......... .......... 57% 38.7M 2s 2024-04-25T16:34:07.811 INFO:tasks.workunit.client.0.smithi098.stderr: 78000K .......... .......... .......... .......... .......... 57% 38.2M 2s 2024-04-25T16:34:07.812 INFO:tasks.workunit.client.0.smithi098.stderr: 78050K .......... .......... .......... .......... .......... 57% 37.9M 2s 2024-04-25T16:34:07.814 INFO:tasks.workunit.client.0.smithi098.stderr: 78100K .......... .......... .......... .......... .......... 57% 33.3M 2s 2024-04-25T16:34:07.815 INFO:tasks.workunit.client.0.smithi098.stderr: 78150K .......... .......... .......... .......... .......... 57% 37.8M 2s 2024-04-25T16:34:07.816 INFO:tasks.workunit.client.0.smithi098.stderr: 78200K .......... .......... .......... .......... .......... 57% 37.4M 2s 2024-04-25T16:34:07.817 INFO:tasks.workunit.client.0.smithi098.stderr: 78250K .......... .......... .......... .......... .......... 57% 37.9M 2s 2024-04-25T16:34:07.819 INFO:tasks.workunit.client.0.smithi098.stderr: 78300K .......... .......... .......... .......... .......... 57% 24.0M 2s 2024-04-25T16:34:07.821 INFO:tasks.workunit.client.0.smithi098.stderr: 78350K .......... .......... .......... .......... .......... 57% 31.3M 2s 2024-04-25T16:34:07.823 INFO:tasks.workunit.client.0.smithi098.stderr: 78400K .......... .......... .......... .......... .......... 57% 25.0M 2s 2024-04-25T16:34:07.824 INFO:tasks.workunit.client.0.smithi098.stderr: 78450K .......... .......... .......... .......... .......... 57% 36.0M 2s 2024-04-25T16:34:07.826 INFO:tasks.workunit.client.0.smithi098.stderr: 78500K .......... .......... .......... .......... .......... 57% 31.1M 2s 2024-04-25T16:34:07.827 INFO:tasks.workunit.client.0.smithi098.stderr: 78550K .......... .......... .......... .......... .......... 57% 34.2M 2s 2024-04-25T16:34:07.829 INFO:tasks.workunit.client.0.smithi098.stderr: 78600K .......... .......... .......... .......... .......... 57% 35.4M 2s 2024-04-25T16:34:07.830 INFO:tasks.workunit.client.0.smithi098.stderr: 78650K .......... .......... .......... .......... .......... 57% 35.3M 2s 2024-04-25T16:34:07.832 INFO:tasks.workunit.client.0.smithi098.stderr: 78700K .......... .......... .......... .......... .......... 58% 30.4M 2s 2024-04-25T16:34:07.833 INFO:tasks.workunit.client.0.smithi098.stderr: 78750K .......... .......... .......... .......... .......... 58% 36.1M 2s 2024-04-25T16:34:07.834 INFO:tasks.workunit.client.0.smithi098.stderr: 78800K .......... .......... .......... .......... .......... 58% 36.1M 2s 2024-04-25T16:34:07.836 INFO:tasks.workunit.client.0.smithi098.stderr: 78850K .......... .......... .......... .......... .......... 58% 35.3M 2s 2024-04-25T16:34:07.837 INFO:tasks.workunit.client.0.smithi098.stderr: 78900K .......... .......... .......... .......... .......... 58% 30.4M 2s 2024-04-25T16:34:07.839 INFO:tasks.workunit.client.0.smithi098.stderr: 78950K .......... .......... .......... .......... .......... 58% 34.7M 2s 2024-04-25T16:34:07.840 INFO:tasks.workunit.client.0.smithi098.stderr: 79000K .......... .......... .......... .......... .......... 58% 34.9M 2s 2024-04-25T16:34:07.842 INFO:tasks.workunit.client.0.smithi098.stderr: 79050K .......... .......... .......... .......... .......... 58% 34.0M 2s 2024-04-25T16:34:07.843 INFO:tasks.workunit.client.0.smithi098.stderr: 79100K .......... .......... .......... .......... .......... 58% 29.6M 2s 2024-04-25T16:34:07.845 INFO:tasks.workunit.client.0.smithi098.stderr: 79150K .......... .......... .......... .......... .......... 58% 34.5M 2s 2024-04-25T16:34:07.846 INFO:tasks.workunit.client.0.smithi098.stderr: 79200K .......... .......... .......... .......... .......... 58% 34.8M 2s 2024-04-25T16:34:07.847 INFO:tasks.workunit.client.0.smithi098.stderr: 79250K .......... .......... .......... .......... .......... 58% 34.5M 2s 2024-04-25T16:34:07.849 INFO:tasks.workunit.client.0.smithi098.stderr: 79300K .......... .......... .......... .......... .......... 58% 29.9M 2s 2024-04-25T16:34:07.851 INFO:tasks.workunit.client.0.smithi098.stderr: 79350K .......... .......... .......... .......... .......... 58% 33.1M 2s 2024-04-25T16:34:07.852 INFO:tasks.workunit.client.0.smithi098.stderr: 79400K .......... .......... .......... .......... .......... 58% 34.6M 2s 2024-04-25T16:34:07.854 INFO:tasks.workunit.client.0.smithi098.stderr: 79450K .......... .......... .......... .......... .......... 58% 33.8M 2s 2024-04-25T16:34:07.855 INFO:tasks.workunit.client.0.smithi098.stderr: 79500K .......... .......... .......... .......... .......... 58% 27.8M 2s 2024-04-25T16:34:07.857 INFO:tasks.workunit.client.0.smithi098.stderr: 79550K .......... .......... .......... .......... .......... 58% 29.8M 2s 2024-04-25T16:34:07.858 INFO:tasks.workunit.client.0.smithi098.stderr: 79600K .......... .......... .......... .......... .......... 58% 31.1M 2s 2024-04-25T16:34:07.860 INFO:tasks.workunit.client.0.smithi098.stderr: 79650K .......... .......... .......... .......... .......... 58% 30.8M 2s 2024-04-25T16:34:07.862 INFO:tasks.workunit.client.0.smithi098.stderr: 79700K .......... .......... .......... .......... .......... 58% 27.0M 2s 2024-04-25T16:34:07.863 INFO:tasks.workunit.client.0.smithi098.stderr: 79750K .......... .......... .......... .......... .......... 58% 31.7M 2s 2024-04-25T16:34:07.865 INFO:tasks.workunit.client.0.smithi098.stderr: 79800K .......... .......... .......... .......... .......... 58% 31.1M 2s 2024-04-25T16:34:07.866 INFO:tasks.workunit.client.0.smithi098.stderr: 79850K .......... .......... .......... .......... .......... 58% 32.0M 2s 2024-04-25T16:34:07.868 INFO:tasks.workunit.client.0.smithi098.stderr: 79900K .......... .......... .......... .......... .......... 58% 25.3M 2s 2024-04-25T16:34:07.870 INFO:tasks.workunit.client.0.smithi098.stderr: 79950K .......... .......... .......... .......... .......... 58% 28.7M 2s 2024-04-25T16:34:07.872 INFO:tasks.workunit.client.0.smithi098.stderr: 80000K .......... .......... .......... .......... .......... 58% 28.4M 2s 2024-04-25T16:34:07.873 INFO:tasks.workunit.client.0.smithi098.stderr: 80050K .......... .......... .......... .......... .......... 59% 31.8M 2s 2024-04-25T16:34:07.875 INFO:tasks.workunit.client.0.smithi098.stderr: 80100K .......... .......... .......... .......... .......... 59% 27.0M 2s 2024-04-25T16:34:07.877 INFO:tasks.workunit.client.0.smithi098.stderr: 80150K .......... .......... .......... .......... .......... 59% 32.3M 2s 2024-04-25T16:34:07.878 INFO:tasks.workunit.client.0.smithi098.stderr: 80200K .......... .......... .......... .......... .......... 59% 30.2M 2s 2024-04-25T16:34:07.880 INFO:tasks.workunit.client.0.smithi098.stderr: 80250K .......... .......... .......... .......... .......... 59% 30.5M 2s 2024-04-25T16:34:07.882 INFO:tasks.workunit.client.0.smithi098.stderr: 80300K .......... .......... .......... .......... .......... 59% 26.4M 2s 2024-04-25T16:34:07.883 INFO:tasks.workunit.client.0.smithi098.stderr: 80350K .......... .......... .......... .......... .......... 59% 32.3M 2s 2024-04-25T16:34:07.885 INFO:tasks.workunit.client.0.smithi098.stderr: 80400K .......... .......... .......... .......... .......... 59% 32.1M 2s 2024-04-25T16:34:07.886 INFO:tasks.workunit.client.0.smithi098.stderr: 80450K .......... .......... .......... .......... .......... 59% 32.0M 2s 2024-04-25T16:34:07.888 INFO:tasks.workunit.client.0.smithi098.stderr: 80500K .......... .......... .......... .......... .......... 59% 27.1M 2s 2024-04-25T16:34:07.890 INFO:tasks.workunit.client.0.smithi098.stderr: 80550K .......... .......... .......... .......... .......... 59% 31.5M 2s 2024-04-25T16:34:07.891 INFO:tasks.workunit.client.0.smithi098.stderr: 80600K .......... .......... .......... .......... .......... 59% 31.5M 2s 2024-04-25T16:34:07.893 INFO:tasks.workunit.client.0.smithi098.stderr: 80650K .......... .......... .......... .......... .......... 59% 29.8M 2s 2024-04-25T16:34:07.895 INFO:tasks.workunit.client.0.smithi098.stderr: 80700K .......... .......... .......... .......... .......... 59% 25.5M 2s 2024-04-25T16:34:07.897 INFO:tasks.workunit.client.0.smithi098.stderr: 80750K .......... .......... .......... .......... .......... 59% 22.8M 2s 2024-04-25T16:34:07.899 INFO:tasks.workunit.client.0.smithi098.stderr: 80800K .......... .......... .......... .......... .......... 59% 26.4M 2s 2024-04-25T16:34:07.901 INFO:tasks.workunit.client.0.smithi098.stderr: 80850K .......... .......... .......... .......... .......... 59% 24.1M 2s 2024-04-25T16:34:07.903 INFO:tasks.workunit.client.0.smithi098.stderr: 80900K .......... .......... .......... .......... .......... 59% 23.0M 2s 2024-04-25T16:34:07.905 INFO:tasks.workunit.client.0.smithi098.stderr: 80950K .......... .......... .......... .......... .......... 59% 26.3M 2s 2024-04-25T16:34:07.907 INFO:tasks.workunit.client.0.smithi098.stderr: 81000K .......... .......... .......... .......... .......... 59% 24.4M 2s 2024-04-25T16:34:07.909 INFO:tasks.workunit.client.0.smithi098.stderr: 81050K .......... .......... .......... .......... .......... 59% 24.7M 2s 2024-04-25T16:34:07.911 INFO:tasks.workunit.client.0.smithi098.stderr: 81100K .......... .......... .......... .......... .......... 59% 23.2M 2s 2024-04-25T16:34:07.912 INFO:tasks.workunit.client.0.smithi098.stderr: 81150K .......... .......... .......... .......... .......... 59% 26.7M 2s 2024-04-25T16:34:07.914 INFO:tasks.workunit.client.0.smithi098.stderr: 81200K .......... .......... .......... .......... .......... 59% 25.7M 2s 2024-04-25T16:34:07.916 INFO:tasks.workunit.client.0.smithi098.stderr: 81250K .......... .......... .......... .......... .......... 59% 26.3M 2s 2024-04-25T16:34:07.918 INFO:tasks.workunit.client.0.smithi098.stderr: 81300K .......... .......... .......... .......... .......... 59% 22.4M 2s 2024-04-25T16:34:07.920 INFO:tasks.workunit.client.0.smithi098.stderr: 81350K .......... .......... .......... .......... .......... 59% 24.4M 2s 2024-04-25T16:34:07.922 INFO:tasks.workunit.client.0.smithi098.stderr: 81400K .......... .......... .......... .......... .......... 59% 25.8M 2s 2024-04-25T16:34:07.924 INFO:tasks.workunit.client.0.smithi098.stderr: 81450K .......... .......... .......... .......... .......... 60% 25.0M 2s 2024-04-25T16:34:07.927 INFO:tasks.workunit.client.0.smithi098.stderr: 81500K .......... .......... .......... .......... .......... 60% 18.1M 2s 2024-04-25T16:34:07.929 INFO:tasks.workunit.client.0.smithi098.stderr: 81550K .......... .......... .......... .......... .......... 60% 27.5M 2s 2024-04-25T16:34:07.931 INFO:tasks.workunit.client.0.smithi098.stderr: 81600K .......... .......... .......... .......... .......... 60% 26.3M 2s 2024-04-25T16:34:07.933 INFO:tasks.workunit.client.0.smithi098.stderr: 81650K .......... .......... .......... .......... .......... 60% 28.1M 2s 2024-04-25T16:34:07.935 INFO:tasks.workunit.client.0.smithi098.stderr: 81700K .......... .......... .......... .......... .......... 60% 23.8M 2s 2024-04-25T16:34:07.936 INFO:tasks.workunit.client.0.smithi098.stderr: 81750K .......... .......... .......... .......... .......... 60% 26.4M 2s 2024-04-25T16:34:07.938 INFO:tasks.workunit.client.0.smithi098.stderr: 81800K .......... .......... .......... .......... .......... 60% 27.0M 2s 2024-04-25T16:34:07.940 INFO:tasks.workunit.client.0.smithi098.stderr: 81850K .......... .......... .......... .......... .......... 60% 27.0M 2s 2024-04-25T16:34:07.942 INFO:tasks.workunit.client.0.smithi098.stderr: 81900K .......... .......... .......... .......... .......... 60% 21.4M 2s 2024-04-25T16:34:07.944 INFO:tasks.workunit.client.0.smithi098.stderr: 81950K .......... .......... .......... .......... .......... 60% 27.6M 2s 2024-04-25T16:34:07.945 INFO:tasks.workunit.client.0.smithi098.stderr: 82000K .......... .......... .......... .......... .......... 60% 39.3M 2s 2024-04-25T16:34:07.946 INFO:tasks.workunit.client.0.smithi098.stderr: 82050K .......... .......... .......... .......... .......... 60% 41.0M 2s 2024-04-25T16:34:07.949 INFO:tasks.workunit.client.0.smithi098.stderr: 82100K .......... .......... .......... .......... .......... 60% 22.5M 2s 2024-04-25T16:34:07.951 INFO:tasks.workunit.client.0.smithi098.stderr: 82150K .......... .......... .......... .......... .......... 60% 19.7M 2s 2024-04-25T16:34:07.952 INFO:tasks.workunit.client.0.smithi098.stderr: 82200K .......... .......... .......... .......... .......... 60% 37.6M 2s 2024-04-25T16:34:07.954 INFO:tasks.workunit.client.0.smithi098.stderr: 82250K .......... .......... .......... .......... .......... 60% 34.0M 2s 2024-04-25T16:34:07.956 INFO:tasks.workunit.client.0.smithi098.stderr: 82300K .......... .......... .......... .......... .......... 60% 21.3M 2s 2024-04-25T16:34:07.957 INFO:tasks.workunit.client.0.smithi098.stderr: 82350K .......... .......... .......... .......... .......... 60% 35.7M 2s 2024-04-25T16:34:07.959 INFO:tasks.workunit.client.0.smithi098.stderr: 82400K .......... .......... .......... .......... .......... 60% 35.6M 2s 2024-04-25T16:34:07.960 INFO:tasks.workunit.client.0.smithi098.stderr: 82450K .......... .......... .......... .......... .......... 60% 35.6M 2s 2024-04-25T16:34:07.962 INFO:tasks.workunit.client.0.smithi098.stderr: 82500K .......... .......... .......... .......... .......... 60% 30.5M 2s 2024-04-25T16:34:07.963 INFO:tasks.workunit.client.0.smithi098.stderr: 82550K .......... .......... .......... .......... .......... 60% 36.2M 2s 2024-04-25T16:34:07.964 INFO:tasks.workunit.client.0.smithi098.stderr: 82600K .......... .......... .......... .......... .......... 60% 36.7M 2s 2024-04-25T16:34:07.966 INFO:tasks.workunit.client.0.smithi098.stderr: 82650K .......... .......... .......... .......... .......... 60% 35.3M 2s 2024-04-25T16:34:07.967 INFO:tasks.workunit.client.0.smithi098.stderr: 82700K .......... .......... .......... .......... .......... 60% 31.3M 2s 2024-04-25T16:34:07.969 INFO:tasks.workunit.client.0.smithi098.stderr: 82750K .......... .......... .......... .......... .......... 60% 35.7M 2s 2024-04-25T16:34:07.970 INFO:tasks.workunit.client.0.smithi098.stderr: 82800K .......... .......... .......... .......... .......... 61% 35.2M 2s 2024-04-25T16:34:07.972 INFO:tasks.workunit.client.0.smithi098.stderr: 82850K .......... .......... .......... .......... .......... 61% 31.9M 2s 2024-04-25T16:34:07.974 INFO:tasks.workunit.client.0.smithi098.stderr: 82900K .......... .......... .......... .......... .......... 61% 27.9M 2s 2024-04-25T16:34:07.975 INFO:tasks.workunit.client.0.smithi098.stderr: 82950K .......... .......... .......... .......... .......... 61% 33.4M 2s 2024-04-25T16:34:07.977 INFO:tasks.workunit.client.0.smithi098.stderr: 83000K .......... .......... .......... .......... .......... 61% 35.0M 2s 2024-04-25T16:34:07.978 INFO:tasks.workunit.client.0.smithi098.stderr: 83050K .......... .......... .......... .......... .......... 61% 35.1M 2s 2024-04-25T16:34:07.980 INFO:tasks.workunit.client.0.smithi098.stderr: 83100K .......... .......... .......... .......... .......... 61% 30.0M 2s 2024-04-25T16:34:07.981 INFO:tasks.workunit.client.0.smithi098.stderr: 83150K .......... .......... .......... .......... .......... 61% 34.5M 2s 2024-04-25T16:34:07.982 INFO:tasks.workunit.client.0.smithi098.stderr: 83200K .......... .......... .......... .......... .......... 61% 32.5M 2s 2024-04-25T16:34:07.984 INFO:tasks.workunit.client.0.smithi098.stderr: 83250K .......... .......... .......... .......... .......... 61% 34.1M 2s 2024-04-25T16:34:07.985 INFO:tasks.workunit.client.0.smithi098.stderr: 83300K .......... .......... .......... .......... .......... 61% 29.4M 2s 2024-04-25T16:34:07.987 INFO:tasks.workunit.client.0.smithi098.stderr: 83350K .......... .......... .......... .......... .......... 61% 32.5M 2s 2024-04-25T16:34:07.988 INFO:tasks.workunit.client.0.smithi098.stderr: 83400K .......... .......... .......... .......... .......... 61% 34.7M 2s 2024-04-25T16:34:07.990 INFO:tasks.workunit.client.0.smithi098.stderr: 83450K .......... .......... .......... .......... .......... 61% 33.0M 2s 2024-04-25T16:34:07.991 INFO:tasks.workunit.client.0.smithi098.stderr: 83500K .......... .......... .......... .......... .......... 61% 28.9M 2s 2024-04-25T16:34:07.993 INFO:tasks.workunit.client.0.smithi098.stderr: 83550K .......... .......... .......... .......... .......... 61% 32.9M 2s 2024-04-25T16:34:07.995 INFO:tasks.workunit.client.0.smithi098.stderr: 83600K .......... .......... .......... .......... .......... 61% 29.6M 2s 2024-04-25T16:34:07.996 INFO:tasks.workunit.client.0.smithi098.stderr: 83650K .......... .......... .......... .......... .......... 61% 32.9M 2s 2024-04-25T16:34:07.998 INFO:tasks.workunit.client.0.smithi098.stderr: 83700K .......... .......... .......... .......... .......... 61% 28.7M 2s 2024-04-25T16:34:07.999 INFO:tasks.workunit.client.0.smithi098.stderr: 83750K .......... .......... .......... .......... .......... 61% 32.1M 2s 2024-04-25T16:34:08.001 INFO:tasks.workunit.client.0.smithi098.stderr: 83800K .......... .......... .......... .......... .......... 61% 32.8M 2s 2024-04-25T16:34:08.002 INFO:tasks.workunit.client.0.smithi098.stderr: 83850K .......... .......... .......... .......... .......... 61% 31.4M 2s 2024-04-25T16:34:08.004 INFO:tasks.workunit.client.0.smithi098.stderr: 83900K .......... .......... .......... .......... .......... 61% 25.4M 2s 2024-04-25T16:34:08.006 INFO:tasks.workunit.client.0.smithi098.stderr: 83950K .......... .......... .......... .......... .......... 61% 29.8M 2s 2024-04-25T16:34:08.008 INFO:tasks.workunit.client.0.smithi098.stderr: 84000K .......... .......... .......... .......... .......... 61% 29.9M 2s 2024-04-25T16:34:08.009 INFO:tasks.workunit.client.0.smithi098.stderr: 84050K .......... .......... .......... .......... .......... 61% 30.1M 2s 2024-04-25T16:34:08.011 INFO:tasks.workunit.client.0.smithi098.stderr: 84100K .......... .......... .......... .......... .......... 61% 25.6M 2s 2024-04-25T16:34:08.013 INFO:tasks.workunit.client.0.smithi098.stderr: 84150K .......... .......... .......... .......... .......... 62% 29.9M 2s 2024-04-25T16:34:08.014 INFO:tasks.workunit.client.0.smithi098.stderr: 84200K .......... .......... .......... .......... .......... 62% 30.0M 2s 2024-04-25T16:34:08.016 INFO:tasks.workunit.client.0.smithi098.stderr: 84250K .......... .......... .......... .......... .......... 62% 29.4M 2s 2024-04-25T16:34:08.018 INFO:tasks.workunit.client.0.smithi098.stderr: 84300K .......... .......... .......... .......... .......... 62% 24.9M 2s 2024-04-25T16:34:08.019 INFO:tasks.workunit.client.0.smithi098.stderr: 84350K .......... .......... .......... .......... .......... 62% 28.8M 2s 2024-04-25T16:34:08.021 INFO:tasks.workunit.client.0.smithi098.stderr: 84400K .......... .......... .......... .......... .......... 62% 31.3M 2s 2024-04-25T16:34:08.023 INFO:tasks.workunit.client.0.smithi098.stderr: 84450K .......... .......... .......... .......... .......... 62% 30.9M 2s 2024-04-25T16:34:08.025 INFO:tasks.workunit.client.0.smithi098.stderr: 84500K .......... .......... .......... .......... .......... 62% 27.0M 2s 2024-04-25T16:34:08.026 INFO:tasks.workunit.client.0.smithi098.stderr: 84550K .......... .......... .......... .......... .......... 62% 30.9M 2s 2024-04-25T16:34:08.028 INFO:tasks.workunit.client.0.smithi098.stderr: 84600K .......... .......... .......... .......... .......... 62% 30.5M 2s 2024-04-25T16:34:08.029 INFO:tasks.workunit.client.0.smithi098.stderr: 84650K .......... .......... .......... .......... .......... 62% 31.0M 2s 2024-04-25T16:34:08.031 INFO:tasks.workunit.client.0.smithi098.stderr: 84700K .......... .......... .......... .......... .......... 62% 26.5M 2s 2024-04-25T16:34:08.033 INFO:tasks.workunit.client.0.smithi098.stderr: 84750K .......... .......... .......... .......... .......... 62% 30.6M 2s 2024-04-25T16:34:08.034 INFO:tasks.workunit.client.0.smithi098.stderr: 84800K .......... .......... .......... .......... .......... 62% 30.5M 2s 2024-04-25T16:34:08.036 INFO:tasks.workunit.client.0.smithi098.stderr: 84850K .......... .......... .......... .......... .......... 62% 30.0M 2s 2024-04-25T16:34:08.038 INFO:tasks.workunit.client.0.smithi098.stderr: 84900K .......... .......... .......... .......... .......... 62% 25.1M 2s 2024-04-25T16:34:08.040 INFO:tasks.workunit.client.0.smithi098.stderr: 84950K .......... .......... .......... .......... .......... 62% 28.3M 2s 2024-04-25T16:34:08.041 INFO:tasks.workunit.client.0.smithi098.stderr: 85000K .......... .......... .......... .......... .......... 62% 28.9M 2s 2024-04-25T16:34:08.043 INFO:tasks.workunit.client.0.smithi098.stderr: 85050K .......... .......... .......... .......... .......... 62% 28.8M 2s 2024-04-25T16:34:08.045 INFO:tasks.workunit.client.0.smithi098.stderr: 85100K .......... .......... .......... .......... .......... 62% 24.3M 2s 2024-04-25T16:34:08.047 INFO:tasks.workunit.client.0.smithi098.stderr: 85150K .......... .......... .......... .......... .......... 62% 24.8M 2s 2024-04-25T16:34:08.049 INFO:tasks.workunit.client.0.smithi098.stderr: 85200K .......... .......... .......... .......... .......... 62% 23.2M 2s 2024-04-25T16:34:08.051 INFO:tasks.workunit.client.0.smithi098.stderr: 85250K .......... .......... .......... .......... .......... 62% 24.3M 2s 2024-04-25T16:34:08.053 INFO:tasks.workunit.client.0.smithi098.stderr: 85300K .......... .......... .......... .......... .......... 62% 22.4M 2s 2024-04-25T16:34:08.055 INFO:tasks.workunit.client.0.smithi098.stderr: 85350K .......... .......... .......... .......... .......... 62% 26.4M 2s 2024-04-25T16:34:08.057 INFO:tasks.workunit.client.0.smithi098.stderr: 85400K .......... .......... .......... .......... .......... 62% 25.7M 2s 2024-04-25T16:34:08.059 INFO:tasks.workunit.client.0.smithi098.stderr: 85450K .......... .......... .......... .......... .......... 62% 25.1M 2s 2024-04-25T16:34:08.061 INFO:tasks.workunit.client.0.smithi098.stderr: 85500K .......... .......... .......... .......... .......... 63% 22.0M 2s 2024-04-25T16:34:08.063 INFO:tasks.workunit.client.0.smithi098.stderr: 85550K .......... .......... .......... .......... .......... 63% 25.2M 2s 2024-04-25T16:34:08.065 INFO:tasks.workunit.client.0.smithi098.stderr: 85600K .......... .......... .......... .......... .......... 63% 24.7M 2s 2024-04-25T16:34:08.067 INFO:tasks.workunit.client.0.smithi098.stderr: 85650K .......... .......... .......... .......... .......... 63% 24.3M 2s 2024-04-25T16:34:08.070 INFO:tasks.workunit.client.0.smithi098.stderr: 85700K .......... .......... .......... .......... .......... 63% 20.6M 2s 2024-04-25T16:34:08.072 INFO:tasks.workunit.client.0.smithi098.stderr: 85750K .......... .......... .......... .......... .......... 63% 21.8M 2s 2024-04-25T16:34:08.074 INFO:tasks.workunit.client.0.smithi098.stderr: 85800K .......... .......... .......... .......... .......... 63% 23.5M 2s 2024-04-25T16:34:08.076 INFO:tasks.workunit.client.0.smithi098.stderr: 85850K .......... .......... .......... .......... .......... 63% 26.1M 2s 2024-04-25T16:34:08.078 INFO:tasks.workunit.client.0.smithi098.stderr: 85900K .......... .......... .......... .......... .......... 63% 22.6M 2s 2024-04-25T16:34:08.080 INFO:tasks.workunit.client.0.smithi098.stderr: 85950K .......... .......... .......... .......... .......... 63% 26.2M 2s 2024-04-25T16:34:08.082 INFO:tasks.workunit.client.0.smithi098.stderr: 86000K .......... .......... .......... .......... .......... 63% 26.1M 2s 2024-04-25T16:34:08.083 INFO:tasks.workunit.client.0.smithi098.stderr: 86050K .......... .......... .......... .......... .......... 63% 29.1M 2s 2024-04-25T16:34:08.085 INFO:tasks.workunit.client.0.smithi098.stderr: 86100K .......... .......... .......... .......... .......... 63% 25.6M 2s 2024-04-25T16:34:08.087 INFO:tasks.workunit.client.0.smithi098.stderr: 86150K .......... .......... .......... .......... .......... 63% 34.9M 2s 2024-04-25T16:34:08.088 INFO:tasks.workunit.client.0.smithi098.stderr: 86200K .......... .......... .......... .......... .......... 63% 39.1M 2s 2024-04-25T16:34:08.089 INFO:tasks.workunit.client.0.smithi098.stderr: 86250K .......... .......... .......... .......... .......... 63% 38.5M 2s 2024-04-25T16:34:08.091 INFO:tasks.workunit.client.0.smithi098.stderr: 86300K .......... .......... .......... .......... .......... 63% 32.4M 2s 2024-04-25T16:34:08.092 INFO:tasks.workunit.client.0.smithi098.stderr: 86350K .......... .......... .......... .......... .......... 63% 38.4M 2s 2024-04-25T16:34:08.093 INFO:tasks.workunit.client.0.smithi098.stderr: 86400K .......... .......... .......... .......... .......... 63% 38.5M 2s 2024-04-25T16:34:08.094 INFO:tasks.workunit.client.0.smithi098.stderr: 86450K .......... .......... .......... .......... .......... 63% 38.9M 2s 2024-04-25T16:34:08.096 INFO:tasks.workunit.client.0.smithi098.stderr: 86500K .......... .......... .......... .......... .......... 63% 32.7M 2s 2024-04-25T16:34:08.097 INFO:tasks.workunit.client.0.smithi098.stderr: 86550K .......... .......... .......... .......... .......... 63% 37.4M 2s 2024-04-25T16:34:08.098 INFO:tasks.workunit.client.0.smithi098.stderr: 86600K .......... .......... .......... .......... .......... 63% 38.4M 2s 2024-04-25T16:34:08.100 INFO:tasks.workunit.client.0.smithi098.stderr: 86650K .......... .......... .......... .......... .......... 63% 37.4M 2s 2024-04-25T16:34:08.101 INFO:tasks.workunit.client.0.smithi098.stderr: 86700K .......... .......... .......... .......... .......... 63% 30.5M 2s 2024-04-25T16:34:08.103 INFO:tasks.workunit.client.0.smithi098.stderr: 86750K .......... .......... .......... .......... .......... 63% 36.4M 2s 2024-04-25T16:34:08.104 INFO:tasks.workunit.client.0.smithi098.stderr: 86800K .......... .......... .......... .......... .......... 63% 35.3M 2s 2024-04-25T16:34:08.105 INFO:tasks.workunit.client.0.smithi098.stderr: 86850K .......... .......... .......... .......... .......... 64% 36.6M 2s 2024-04-25T16:34:08.107 INFO:tasks.workunit.client.0.smithi098.stderr: 86900K .......... .......... .......... .......... .......... 64% 29.4M 2s 2024-04-25T16:34:08.108 INFO:tasks.workunit.client.0.smithi098.stderr: 86950K .......... .......... .......... .......... .......... 64% 36.6M 2s 2024-04-25T16:34:08.110 INFO:tasks.workunit.client.0.smithi098.stderr: 87000K .......... .......... .......... .......... .......... 64% 34.8M 2s 2024-04-25T16:34:08.111 INFO:tasks.workunit.client.0.smithi098.stderr: 87050K .......... .......... .......... .......... .......... 64% 36.4M 2s 2024-04-25T16:34:08.113 INFO:tasks.workunit.client.0.smithi098.stderr: 87100K .......... .......... .......... .......... .......... 64% 31.2M 2s 2024-04-25T16:34:08.114 INFO:tasks.workunit.client.0.smithi098.stderr: 87150K .......... .......... .......... .......... .......... 64% 35.9M 2s 2024-04-25T16:34:08.115 INFO:tasks.workunit.client.0.smithi098.stderr: 87200K .......... .......... .......... .......... .......... 64% 36.0M 2s 2024-04-25T16:34:08.117 INFO:tasks.workunit.client.0.smithi098.stderr: 87250K .......... .......... .......... .......... .......... 64% 35.3M 2s 2024-04-25T16:34:08.118 INFO:tasks.workunit.client.0.smithi098.stderr: 87300K .......... .......... .......... .......... .......... 64% 30.8M 2s 2024-04-25T16:34:08.120 INFO:tasks.workunit.client.0.smithi098.stderr: 87350K .......... .......... .......... .......... .......... 64% 35.0M 2s 2024-04-25T16:34:08.121 INFO:tasks.workunit.client.0.smithi098.stderr: 87400K .......... .......... .......... .......... .......... 64% 34.7M 2s 2024-04-25T16:34:08.123 INFO:tasks.workunit.client.0.smithi098.stderr: 87450K .......... .......... .......... .......... .......... 64% 34.4M 2s 2024-04-25T16:34:08.124 INFO:tasks.workunit.client.0.smithi098.stderr: 87500K .......... .......... .......... .......... .......... 64% 29.9M 2s 2024-04-25T16:34:08.126 INFO:tasks.workunit.client.0.smithi098.stderr: 87550K .......... .......... .......... .......... .......... 64% 34.7M 2s 2024-04-25T16:34:08.127 INFO:tasks.workunit.client.0.smithi098.stderr: 87600K .......... .......... .......... .......... .......... 64% 34.7M 2s 2024-04-25T16:34:08.128 INFO:tasks.workunit.client.0.smithi098.stderr: 87650K .......... .......... .......... .......... .......... 64% 33.5M 2s 2024-04-25T16:34:08.130 INFO:tasks.workunit.client.0.smithi098.stderr: 87700K .......... .......... .......... .......... .......... 64% 29.2M 2s 2024-04-25T16:34:08.132 INFO:tasks.workunit.client.0.smithi098.stderr: 87750K .......... .......... .......... .......... .......... 64% 33.8M 2s 2024-04-25T16:34:08.133 INFO:tasks.workunit.client.0.smithi098.stderr: 87800K .......... .......... .......... .......... .......... 64% 33.7M 2s 2024-04-25T16:34:08.135 INFO:tasks.workunit.client.0.smithi098.stderr: 87850K .......... .......... .......... .......... .......... 64% 33.4M 2s 2024-04-25T16:34:08.136 INFO:tasks.workunit.client.0.smithi098.stderr: 87900K .......... .......... .......... .......... .......... 64% 29.1M 2s 2024-04-25T16:34:08.138 INFO:tasks.workunit.client.0.smithi098.stderr: 87950K .......... .......... .......... .......... .......... 64% 33.0M 2s 2024-04-25T16:34:08.139 INFO:tasks.workunit.client.0.smithi098.stderr: 88000K .......... .......... .......... .......... .......... 64% 33.2M 2s 2024-04-25T16:34:08.141 INFO:tasks.workunit.client.0.smithi098.stderr: 88050K .......... .......... .......... .......... .......... 64% 32.1M 2s 2024-04-25T16:34:08.142 INFO:tasks.workunit.client.0.smithi098.stderr: 88100K .......... .......... .......... .......... .......... 64% 27.5M 2s 2024-04-25T16:34:08.144 INFO:tasks.workunit.client.0.smithi098.stderr: 88150K .......... .......... .......... .......... .......... 64% 32.1M 2s 2024-04-25T16:34:08.146 INFO:tasks.workunit.client.0.smithi098.stderr: 88200K .......... .......... .......... .......... .......... 65% 28.5M 2s 2024-04-25T16:34:08.147 INFO:tasks.workunit.client.0.smithi098.stderr: 88250K .......... .......... .......... .......... .......... 65% 31.0M 2s 2024-04-25T16:34:08.149 INFO:tasks.workunit.client.0.smithi098.stderr: 88300K .......... .......... .......... .......... .......... 65% 26.6M 2s 2024-04-25T16:34:08.151 INFO:tasks.workunit.client.0.smithi098.stderr: 88350K .......... .......... .......... .......... .......... 65% 31.4M 2s 2024-04-25T16:34:08.152 INFO:tasks.workunit.client.0.smithi098.stderr: 88400K .......... .......... .......... .......... .......... 65% 30.0M 2s 2024-04-25T16:34:08.154 INFO:tasks.workunit.client.0.smithi098.stderr: 88450K .......... .......... .......... .......... .......... 65% 30.7M 2s 2024-04-25T16:34:08.156 INFO:tasks.workunit.client.0.smithi098.stderr: 88500K .......... .......... .......... .......... .......... 65% 25.4M 2s 2024-04-25T16:34:08.158 INFO:tasks.workunit.client.0.smithi098.stderr: 88550K .......... .......... .......... .......... .......... 65% 27.7M 2s 2024-04-25T16:34:08.159 INFO:tasks.workunit.client.0.smithi098.stderr: 88600K .......... .......... .......... .......... .......... 65% 28.5M 2s 2024-04-25T16:34:08.161 INFO:tasks.workunit.client.0.smithi098.stderr: 88650K .......... .......... .......... .......... .......... 65% 28.3M 2s 2024-04-25T16:34:08.163 INFO:tasks.workunit.client.0.smithi098.stderr: 88700K .......... .......... .......... .......... .......... 65% 24.8M 2s 2024-04-25T16:34:08.165 INFO:tasks.workunit.client.0.smithi098.stderr: 88750K .......... .......... .......... .......... .......... 65% 28.2M 2s 2024-04-25T16:34:08.167 INFO:tasks.workunit.client.0.smithi098.stderr: 88800K .......... .......... .......... .......... .......... 65% 28.3M 2s 2024-04-25T16:34:08.168 INFO:tasks.workunit.client.0.smithi098.stderr: 88850K .......... .......... .......... .......... .......... 65% 27.5M 2s 2024-04-25T16:34:08.170 INFO:tasks.workunit.client.0.smithi098.stderr: 88900K .......... .......... .......... .......... .......... 65% 23.3M 2s 2024-04-25T16:34:08.172 INFO:tasks.workunit.client.0.smithi098.stderr: 88950K .......... .......... .......... .......... .......... 65% 27.8M 2s 2024-04-25T16:34:08.174 INFO:tasks.workunit.client.0.smithi098.stderr: 89000K .......... .......... .......... .......... .......... 65% 28.2M 2s 2024-04-25T16:34:08.176 INFO:tasks.workunit.client.0.smithi098.stderr: 89050K .......... .......... .......... .......... .......... 65% 27.8M 2s 2024-04-25T16:34:08.178 INFO:tasks.workunit.client.0.smithi098.stderr: 89100K .......... .......... .......... .......... .......... 65% 23.8M 2s 2024-04-25T16:34:08.180 INFO:tasks.workunit.client.0.smithi098.stderr: 89150K .......... .......... .......... .......... .......... 65% 26.6M 2s 2024-04-25T16:34:08.181 INFO:tasks.workunit.client.0.smithi098.stderr: 89200K .......... .......... .......... .......... .......... 65% 26.9M 2s 2024-04-25T16:34:08.183 INFO:tasks.workunit.client.0.smithi098.stderr: 89250K .......... .......... .......... .......... .......... 65% 27.1M 2s 2024-04-25T16:34:08.185 INFO:tasks.workunit.client.0.smithi098.stderr: 89300K .......... .......... .......... .......... .......... 65% 22.7M 2s 2024-04-25T16:34:08.187 INFO:tasks.workunit.client.0.smithi098.stderr: 89350K .......... .......... .......... .......... .......... 65% 24.9M 2s 2024-04-25T16:34:08.189 INFO:tasks.workunit.client.0.smithi098.stderr: 89400K .......... .......... .......... .......... .......... 65% 24.4M 2s 2024-04-25T16:34:08.191 INFO:tasks.workunit.client.0.smithi098.stderr: 89450K .......... .......... .......... .......... .......... 65% 25.7M 2s 2024-04-25T16:34:08.193 INFO:tasks.workunit.client.0.smithi098.stderr: 89500K .......... .......... .......... .......... .......... 65% 22.9M 2s 2024-04-25T16:34:08.195 INFO:tasks.workunit.client.0.smithi098.stderr: 89550K .......... .......... .......... .......... .......... 66% 26.5M 2s 2024-04-25T16:34:08.197 INFO:tasks.workunit.client.0.smithi098.stderr: 89600K .......... .......... .......... .......... .......... 66% 26.5M 2s 2024-04-25T16:34:08.199 INFO:tasks.workunit.client.0.smithi098.stderr: 89650K .......... .......... .......... .......... .......... 66% 26.1M 2s 2024-04-25T16:34:08.201 INFO:tasks.workunit.client.0.smithi098.stderr: 89700K .......... .......... .......... .......... .......... 66% 22.2M 2s 2024-04-25T16:34:08.203 INFO:tasks.workunit.client.0.smithi098.stderr: 89750K .......... .......... .......... .......... .......... 66% 26.0M 2s 2024-04-25T16:34:08.205 INFO:tasks.workunit.client.0.smithi098.stderr: 89800K .......... .......... .......... .......... .......... 66% 25.9M 2s 2024-04-25T16:34:08.207 INFO:tasks.workunit.client.0.smithi098.stderr: 89850K .......... .......... .......... .......... .......... 66% 25.1M 2s 2024-04-25T16:34:08.209 INFO:tasks.workunit.client.0.smithi098.stderr: 89900K .......... .......... .......... .......... .......... 66% 21.4M 2s 2024-04-25T16:34:08.211 INFO:tasks.workunit.client.0.smithi098.stderr: 89950K .......... .......... .......... .......... .......... 66% 25.4M 2s 2024-04-25T16:34:08.213 INFO:tasks.workunit.client.0.smithi098.stderr: 90000K .......... .......... .......... .......... .......... 66% 24.8M 2s 2024-04-25T16:34:08.215 INFO:tasks.workunit.client.0.smithi098.stderr: 90050K .......... .......... .......... .......... .......... 66% 24.4M 2s 2024-04-25T16:34:08.217 INFO:tasks.workunit.client.0.smithi098.stderr: 90100K .......... .......... .......... .......... .......... 66% 21.9M 2s 2024-04-25T16:34:08.219 INFO:tasks.workunit.client.0.smithi098.stderr: 90150K .......... .......... .......... .......... .......... 66% 26.9M 2s 2024-04-25T16:34:08.221 INFO:tasks.workunit.client.0.smithi098.stderr: 90200K .......... .......... .......... .......... .......... 66% 29.1M 2s 2024-04-25T16:34:08.222 INFO:tasks.workunit.client.0.smithi098.stderr: 90250K .......... .......... .......... .......... .......... 66% 30.8M 2s 2024-04-25T16:34:08.224 INFO:tasks.workunit.client.0.smithi098.stderr: 90300K .......... .......... .......... .......... .......... 66% 27.0M 2s 2024-04-25T16:34:08.226 INFO:tasks.workunit.client.0.smithi098.stderr: 90350K .......... .......... .......... .......... .......... 66% 30.8M 2s 2024-04-25T16:34:08.227 INFO:tasks.workunit.client.0.smithi098.stderr: 90400K .......... .......... .......... .......... .......... 66% 28.0M 2s 2024-04-25T16:34:08.229 INFO:tasks.workunit.client.0.smithi098.stderr: 90450K .......... .......... .......... .......... .......... 66% 26.2M 2s 2024-04-25T16:34:08.231 INFO:tasks.workunit.client.0.smithi098.stderr: 90500K .......... .......... .......... .......... .......... 66% 22.2M 2s 2024-04-25T16:34:08.233 INFO:tasks.workunit.client.0.smithi098.stderr: 90550K .......... .......... .......... .......... .......... 66% 31.4M 2s 2024-04-25T16:34:08.235 INFO:tasks.workunit.client.0.smithi098.stderr: 90600K .......... .......... .......... .......... .......... 66% 26.0M 2s 2024-04-25T16:34:08.236 INFO:tasks.workunit.client.0.smithi098.stderr: 90650K .......... .......... .......... .......... .......... 66% 34.9M 2s 2024-04-25T16:34:08.238 INFO:tasks.workunit.client.0.smithi098.stderr: 90700K .......... .......... .......... .......... .......... 66% 31.9M 2s 2024-04-25T16:34:08.239 INFO:tasks.workunit.client.0.smithi098.stderr: 90750K .......... .......... .......... .......... .......... 66% 38.0M 2s 2024-04-25T16:34:08.240 INFO:tasks.workunit.client.0.smithi098.stderr: 90800K .......... .......... .......... .......... .......... 66% 37.3M 2s 2024-04-25T16:34:08.242 INFO:tasks.workunit.client.0.smithi098.stderr: 90850K .......... .......... .......... .......... .......... 66% 36.7M 2s 2024-04-25T16:34:08.244 INFO:tasks.workunit.client.0.smithi098.stderr: 90900K .......... .......... .......... .......... .......... 66% 29.5M 2s 2024-04-25T16:34:08.245 INFO:tasks.workunit.client.0.smithi098.stderr: 90950K .......... .......... .......... .......... .......... 67% 37.5M 2s 2024-04-25T16:34:08.246 INFO:tasks.workunit.client.0.smithi098.stderr: 91000K .......... .......... .......... .......... .......... 67% 37.6M 2s 2024-04-25T16:34:08.247 INFO:tasks.workunit.client.0.smithi098.stderr: 91050K .......... .......... .......... .......... .......... 67% 37.4M 2s 2024-04-25T16:34:08.249 INFO:tasks.workunit.client.0.smithi098.stderr: 91100K .......... .......... .......... .......... .......... 67% 30.9M 2s 2024-04-25T16:34:08.250 INFO:tasks.workunit.client.0.smithi098.stderr: 91150K .......... .......... .......... .......... .......... 67% 36.2M 2s 2024-04-25T16:34:08.252 INFO:tasks.workunit.client.0.smithi098.stderr: 91200K .......... .......... .......... .......... .......... 67% 37.3M 2s 2024-04-25T16:34:08.253 INFO:tasks.workunit.client.0.smithi098.stderr: 91250K .......... .......... .......... .......... .......... 67% 36.3M 2s 2024-04-25T16:34:08.255 INFO:tasks.workunit.client.0.smithi098.stderr: 91300K .......... .......... .......... .......... .......... 67% 31.4M 2s 2024-04-25T16:34:08.256 INFO:tasks.workunit.client.0.smithi098.stderr: 91350K .......... .......... .......... .......... .......... 67% 36.2M 2s 2024-04-25T16:34:08.257 INFO:tasks.workunit.client.0.smithi098.stderr: 91400K .......... .......... .......... .......... .......... 67% 31.6M 2s 2024-04-25T16:34:08.259 INFO:tasks.workunit.client.0.smithi098.stderr: 91450K .......... .......... .......... .......... .......... 67% 32.8M 2s 2024-04-25T16:34:08.260 INFO:tasks.workunit.client.0.smithi098.stderr: 91500K .......... .......... .......... .......... .......... 67% 28.6M 2s 2024-04-25T16:34:08.262 INFO:tasks.workunit.client.0.smithi098.stderr: 91550K .......... .......... .......... .......... .......... 67% 32.5M 2s 2024-04-25T16:34:08.263 INFO:tasks.workunit.client.0.smithi098.stderr: 91600K .......... .......... .......... .......... .......... 67% 32.9M 2s 2024-04-25T16:34:08.265 INFO:tasks.workunit.client.0.smithi098.stderr: 91650K .......... .......... .......... .......... .......... 67% 32.7M 2s 2024-04-25T16:34:08.267 INFO:tasks.workunit.client.0.smithi098.stderr: 91700K .......... .......... .......... .......... .......... 67% 27.3M 2s 2024-04-25T16:34:08.268 INFO:tasks.workunit.client.0.smithi098.stderr: 91750K .......... .......... .......... .......... .......... 67% 32.1M 2s 2024-04-25T16:34:08.270 INFO:tasks.workunit.client.0.smithi098.stderr: 91800K .......... .......... .......... .......... .......... 67% 31.2M 2s 2024-04-25T16:34:08.271 INFO:tasks.workunit.client.0.smithi098.stderr: 91850K .......... .......... .......... .......... .......... 67% 31.7M 2s 2024-04-25T16:34:08.273 INFO:tasks.workunit.client.0.smithi098.stderr: 91900K .......... .......... .......... .......... .......... 67% 27.0M 2s 2024-04-25T16:34:08.275 INFO:tasks.workunit.client.0.smithi098.stderr: 91950K .......... .......... .......... .......... .......... 67% 31.2M 2s 2024-04-25T16:34:08.276 INFO:tasks.workunit.client.0.smithi098.stderr: 92000K .......... .......... .......... .......... .......... 67% 30.8M 2s 2024-04-25T16:34:08.278 INFO:tasks.workunit.client.0.smithi098.stderr: 92050K .......... .......... .......... .......... .......... 67% 30.6M 2s 2024-04-25T16:34:08.280 INFO:tasks.workunit.client.0.smithi098.stderr: 92100K .......... .......... .......... .......... .......... 67% 26.7M 2s 2024-04-25T16:34:08.281 INFO:tasks.workunit.client.0.smithi098.stderr: 92150K .......... .......... .......... .......... .......... 67% 31.4M 2s 2024-04-25T16:34:08.283 INFO:tasks.workunit.client.0.smithi098.stderr: 92200K .......... .......... .......... .......... .......... 67% 31.3M 2s 2024-04-25T16:34:08.284 INFO:tasks.workunit.client.0.smithi098.stderr: 92250K .......... .......... .......... .......... .......... 67% 31.0M 2s 2024-04-25T16:34:08.286 INFO:tasks.workunit.client.0.smithi098.stderr: 92300K .......... .......... .......... .......... .......... 68% 26.7M 2s 2024-04-25T16:34:08.288 INFO:tasks.workunit.client.0.smithi098.stderr: 92350K .......... .......... .......... .......... .......... 68% 30.9M 2s 2024-04-25T16:34:08.289 INFO:tasks.workunit.client.0.smithi098.stderr: 92400K .......... .......... .......... .......... .......... 68% 30.4M 2s 2024-04-25T16:34:08.291 INFO:tasks.workunit.client.0.smithi098.stderr: 92450K .......... .......... .......... .......... .......... 68% 29.8M 2s 2024-04-25T16:34:08.293 INFO:tasks.workunit.client.0.smithi098.stderr: 92500K .......... .......... .......... .......... .......... 68% 25.9M 2s 2024-04-25T16:34:08.294 INFO:tasks.workunit.client.0.smithi098.stderr: 92550K .......... .......... .......... .......... .......... 68% 30.2M 2s 2024-04-25T16:34:08.296 INFO:tasks.workunit.client.0.smithi098.stderr: 92600K .......... .......... .......... .......... .......... 68% 29.9M 2s 2024-04-25T16:34:08.298 INFO:tasks.workunit.client.0.smithi098.stderr: 92650K .......... .......... .......... .......... .......... 68% 28.3M 2s 2024-04-25T16:34:08.300 INFO:tasks.workunit.client.0.smithi098.stderr: 92700K .......... .......... .......... .......... .......... 68% 24.3M 2s 2024-04-25T16:34:08.302 INFO:tasks.workunit.client.0.smithi098.stderr: 92750K .......... .......... .......... .......... .......... 68% 28.5M 2s 2024-04-25T16:34:08.303 INFO:tasks.workunit.client.0.smithi098.stderr: 92800K .......... .......... .......... .......... .......... 68% 28.3M 2s 2024-04-25T16:34:08.305 INFO:tasks.workunit.client.0.smithi098.stderr: 92850K .......... .......... .......... .......... .......... 68% 30.7M 2s 2024-04-25T16:34:08.307 INFO:tasks.workunit.client.0.smithi098.stderr: 92900K .......... .......... .......... .......... .......... 68% 24.6M 2s 2024-04-25T16:34:08.309 INFO:tasks.workunit.client.0.smithi098.stderr: 92950K .......... .......... .......... .......... .......... 68% 28.3M 2s 2024-04-25T16:34:08.310 INFO:tasks.workunit.client.0.smithi098.stderr: 93000K .......... .......... .......... .......... .......... 68% 29.5M 2s 2024-04-25T16:34:08.312 INFO:tasks.workunit.client.0.smithi098.stderr: 93050K .......... .......... .......... .......... .......... 68% 29.3M 2s 2024-04-25T16:34:08.314 INFO:tasks.workunit.client.0.smithi098.stderr: 93100K .......... .......... .......... .......... .......... 68% 25.5M 2s 2024-04-25T16:34:08.316 INFO:tasks.workunit.client.0.smithi098.stderr: 93150K .......... .......... .......... .......... .......... 68% 29.1M 2s 2024-04-25T16:34:08.317 INFO:tasks.workunit.client.0.smithi098.stderr: 93200K .......... .......... .......... .......... .......... 68% 28.6M 2s 2024-04-25T16:34:08.319 INFO:tasks.workunit.client.0.smithi098.stderr: 93250K .......... .......... .......... .......... .......... 68% 28.4M 1s 2024-04-25T16:34:08.321 INFO:tasks.workunit.client.0.smithi098.stderr: 93300K .......... .......... .......... .......... .......... 68% 25.0M 1s 2024-04-25T16:34:08.323 INFO:tasks.workunit.client.0.smithi098.stderr: 93350K .......... .......... .......... .......... .......... 68% 28.1M 1s 2024-04-25T16:34:08.324 INFO:tasks.workunit.client.0.smithi098.stderr: 93400K .......... .......... .......... .......... .......... 68% 28.4M 1s 2024-04-25T16:34:08.326 INFO:tasks.workunit.client.0.smithi098.stderr: 93450K .......... .......... .......... .......... .......... 68% 29.0M 1s 2024-04-25T16:34:08.328 INFO:tasks.workunit.client.0.smithi098.stderr: 93500K .......... .......... .......... .......... .......... 68% 24.6M 1s 2024-04-25T16:34:08.330 INFO:tasks.workunit.client.0.smithi098.stderr: 93550K .......... .......... .......... .......... .......... 68% 27.7M 1s 2024-04-25T16:34:08.331 INFO:tasks.workunit.client.0.smithi098.stderr: 93600K .......... .......... .......... .......... .......... 68% 28.1M 1s 2024-04-25T16:34:08.333 INFO:tasks.workunit.client.0.smithi098.stderr: 93650K .......... .......... .......... .......... .......... 69% 28.4M 1s 2024-04-25T16:34:08.335 INFO:tasks.workunit.client.0.smithi098.stderr: 93700K .......... .......... .......... .......... .......... 69% 23.4M 1s 2024-04-25T16:34:08.337 INFO:tasks.workunit.client.0.smithi098.stderr: 93750K .......... .......... .......... .......... .......... 69% 26.4M 1s 2024-04-25T16:34:08.339 INFO:tasks.workunit.client.0.smithi098.stderr: 93800K .......... .......... .......... .......... .......... 69% 27.3M 1s 2024-04-25T16:34:08.341 INFO:tasks.workunit.client.0.smithi098.stderr: 93850K .......... .......... .......... .......... .......... 69% 26.1M 1s 2024-04-25T16:34:08.343 INFO:tasks.workunit.client.0.smithi098.stderr: 93900K .......... .......... .......... .......... .......... 69% 23.2M 1s 2024-04-25T16:34:08.345 INFO:tasks.workunit.client.0.smithi098.stderr: 93950K .......... .......... .......... .......... .......... 69% 27.0M 1s 2024-04-25T16:34:08.347 INFO:tasks.workunit.client.0.smithi098.stderr: 94000K .......... .......... .......... .......... .......... 69% 27.5M 1s 2024-04-25T16:34:08.348 INFO:tasks.workunit.client.0.smithi098.stderr: 94050K .......... .......... .......... .......... .......... 69% 27.5M 1s 2024-04-25T16:34:08.350 INFO:tasks.workunit.client.0.smithi098.stderr: 94100K .......... .......... .......... .......... .......... 69% 23.9M 1s 2024-04-25T16:34:08.352 INFO:tasks.workunit.client.0.smithi098.stderr: 94150K .......... .......... .......... .......... .......... 69% 26.8M 1s 2024-04-25T16:34:08.354 INFO:tasks.workunit.client.0.smithi098.stderr: 94200K .......... .......... .......... .......... .......... 69% 27.4M 1s 2024-04-25T16:34:08.356 INFO:tasks.workunit.client.0.smithi098.stderr: 94250K .......... .......... .......... .......... .......... 69% 27.8M 1s 2024-04-25T16:34:08.358 INFO:tasks.workunit.client.0.smithi098.stderr: 94300K .......... .......... .......... .......... .......... 69% 23.2M 1s 2024-04-25T16:34:08.359 INFO:tasks.workunit.client.0.smithi098.stderr: 94350K .......... .......... .......... .......... .......... 69% 31.2M 1s 2024-04-25T16:34:08.361 INFO:tasks.workunit.client.0.smithi098.stderr: 94400K .......... .......... .......... .......... .......... 69% 28.4M 1s 2024-04-25T16:34:08.364 INFO:tasks.workunit.client.0.smithi098.stderr: 94450K .......... .......... .......... .......... .......... 69% 18.6M 1s 2024-04-25T16:34:08.366 INFO:tasks.workunit.client.0.smithi098.stderr: 94500K .......... .......... .......... .......... .......... 69% 26.3M 1s 2024-04-25T16:34:08.367 INFO:tasks.workunit.client.0.smithi098.stderr: 94550K .......... .......... .......... .......... .......... 69% 36.0M 1s 2024-04-25T16:34:08.368 INFO:tasks.workunit.client.0.smithi098.stderr: 94600K .......... .......... .......... .......... .......... 69% 31.3M 1s 2024-04-25T16:34:08.370 INFO:tasks.workunit.client.0.smithi098.stderr: 94650K .......... .......... .......... .......... .......... 69% 31.5M 1s 2024-04-25T16:34:08.372 INFO:tasks.workunit.client.0.smithi098.stderr: 94700K .......... .......... .......... .......... .......... 69% 30.1M 1s 2024-04-25T16:34:08.373 INFO:tasks.workunit.client.0.smithi098.stderr: 94750K .......... .......... .......... .......... .......... 69% 34.3M 1s 2024-04-25T16:34:08.375 INFO:tasks.workunit.client.0.smithi098.stderr: 94800K .......... .......... .......... .......... .......... 69% 30.7M 1s 2024-04-25T16:34:08.376 INFO:tasks.workunit.client.0.smithi098.stderr: 94850K .......... .......... .......... .......... .......... 69% 33.4M 1s 2024-04-25T16:34:08.378 INFO:tasks.workunit.client.0.smithi098.stderr: 94900K .......... .......... .......... .......... .......... 69% 28.1M 1s 2024-04-25T16:34:08.380 INFO:tasks.workunit.client.0.smithi098.stderr: 94950K .......... .......... .......... .......... .......... 69% 32.2M 1s 2024-04-25T16:34:08.381 INFO:tasks.workunit.client.0.smithi098.stderr: 95000K .......... .......... .......... .......... .......... 70% 32.2M 1s 2024-04-25T16:34:08.383 INFO:tasks.workunit.client.0.smithi098.stderr: 95050K .......... .......... .......... .......... .......... 70% 31.3M 1s 2024-04-25T16:34:08.384 INFO:tasks.workunit.client.0.smithi098.stderr: 95100K .......... .......... .......... .......... .......... 70% 30.6M 1s 2024-04-25T16:34:08.386 INFO:tasks.workunit.client.0.smithi098.stderr: 95150K .......... .......... .......... .......... .......... 70% 34.4M 1s 2024-04-25T16:34:08.387 INFO:tasks.workunit.client.0.smithi098.stderr: 95200K .......... .......... .......... .......... .......... 70% 31.7M 1s 2024-04-25T16:34:08.389 INFO:tasks.workunit.client.0.smithi098.stderr: 95250K .......... .......... .......... .......... .......... 70% 33.5M 1s 2024-04-25T16:34:08.390 INFO:tasks.workunit.client.0.smithi098.stderr: 95300K .......... .......... .......... .......... .......... 70% 30.0M 1s 2024-04-25T16:34:08.392 INFO:tasks.workunit.client.0.smithi098.stderr: 95350K .......... .......... .......... .......... .......... 70% 35.4M 1s 2024-04-25T16:34:08.393 INFO:tasks.workunit.client.0.smithi098.stderr: 95400K .......... .......... .......... .......... .......... 70% 36.5M 1s 2024-04-25T16:34:08.394 INFO:tasks.workunit.client.0.smithi098.stderr: 95450K .......... .......... .......... .......... .......... 70% 37.7M 1s 2024-04-25T16:34:08.396 INFO:tasks.workunit.client.0.smithi098.stderr: 95500K .......... .......... .......... .......... .......... 70% 30.3M 1s 2024-04-25T16:34:08.397 INFO:tasks.workunit.client.0.smithi098.stderr: 95550K .......... .......... .......... .......... .......... 70% 36.6M 1s 2024-04-25T16:34:08.399 INFO:tasks.workunit.client.0.smithi098.stderr: 95600K .......... .......... .......... .......... .......... 70% 35.6M 1s 2024-04-25T16:34:08.400 INFO:tasks.workunit.client.0.smithi098.stderr: 95650K .......... .......... .......... .......... .......... 70% 36.4M 1s 2024-04-25T16:34:08.402 INFO:tasks.workunit.client.0.smithi098.stderr: 95700K .......... .......... .......... .......... .......... 70% 28.4M 1s 2024-04-25T16:34:08.403 INFO:tasks.workunit.client.0.smithi098.stderr: 95750K .......... .......... .......... .......... .......... 70% 29.6M 1s 2024-04-25T16:34:08.405 INFO:tasks.workunit.client.0.smithi098.stderr: 95800K .......... .......... .......... .......... .......... 70% 29.9M 1s 2024-04-25T16:34:08.407 INFO:tasks.workunit.client.0.smithi098.stderr: 95850K .......... .......... .......... .......... .......... 70% 29.4M 1s 2024-04-25T16:34:08.408 INFO:tasks.workunit.client.0.smithi098.stderr: 95900K .......... .......... .......... .......... .......... 70% 29.0M 1s 2024-04-25T16:34:08.410 INFO:tasks.workunit.client.0.smithi098.stderr: 95950K .......... .......... .......... .......... .......... 70% 34.2M 1s 2024-04-25T16:34:08.411 INFO:tasks.workunit.client.0.smithi098.stderr: 96000K .......... .......... .......... .......... .......... 70% 32.9M 1s 2024-04-25T16:34:08.413 INFO:tasks.workunit.client.0.smithi098.stderr: 96050K .......... .......... .......... .......... .......... 70% 32.0M 1s 2024-04-25T16:34:08.414 INFO:tasks.workunit.client.0.smithi098.stderr: 96100K .......... .......... .......... .......... .......... 70% 27.6M 1s 2024-04-25T16:34:08.416 INFO:tasks.workunit.client.0.smithi098.stderr: 96150K .......... .......... .......... .......... .......... 70% 32.3M 1s 2024-04-25T16:34:08.417 INFO:tasks.workunit.client.0.smithi098.stderr: 96200K .......... .......... .......... .......... .......... 70% 32.9M 1s 2024-04-25T16:34:08.419 INFO:tasks.workunit.client.0.smithi098.stderr: 96250K .......... .......... .......... .......... .......... 70% 28.9M 1s 2024-04-25T16:34:08.421 INFO:tasks.workunit.client.0.smithi098.stderr: 96300K .......... .......... .......... .......... .......... 70% 25.0M 1s 2024-04-25T16:34:08.423 INFO:tasks.workunit.client.0.smithi098.stderr: 96350K .......... .......... .......... .......... .......... 71% 29.1M 1s 2024-04-25T16:34:08.424 INFO:tasks.workunit.client.0.smithi098.stderr: 96400K .......... .......... .......... .......... .......... 71% 28.5M 1s 2024-04-25T16:34:08.426 INFO:tasks.workunit.client.0.smithi098.stderr: 96450K .......... .......... .......... .......... .......... 71% 29.2M 1s 2024-04-25T16:34:08.428 INFO:tasks.workunit.client.0.smithi098.stderr: 96500K .......... .......... .......... .......... .......... 71% 26.1M 1s 2024-04-25T16:34:08.430 INFO:tasks.workunit.client.0.smithi098.stderr: 96550K .......... .......... .......... .......... .......... 71% 29.8M 1s 2024-04-25T16:34:08.431 INFO:tasks.workunit.client.0.smithi098.stderr: 96600K .......... .......... .......... .......... .......... 71% 30.7M 1s 2024-04-25T16:34:08.433 INFO:tasks.workunit.client.0.smithi098.stderr: 96650K .......... .......... .......... .......... .......... 71% 30.6M 1s 2024-04-25T16:34:08.435 INFO:tasks.workunit.client.0.smithi098.stderr: 96700K .......... .......... .......... .......... .......... 71% 26.8M 1s 2024-04-25T16:34:08.436 INFO:tasks.workunit.client.0.smithi098.stderr: 96750K .......... .......... .......... .......... .......... 71% 30.7M 1s 2024-04-25T16:34:08.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:08 smithi203 ceph-mon[33115]: pgmap v405: 129 pgs: 129 active+clean; 45 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.7 MiB/s wr, 12 op/s 2024-04-25T16:34:08.438 INFO:tasks.workunit.client.0.smithi098.stderr: 96800K .......... .......... .......... .......... .......... 71% 30.2M 1s 2024-04-25T16:34:08.440 INFO:tasks.workunit.client.0.smithi098.stderr: 96850K .......... .......... .......... .......... .......... 71% 30.4M 1s 2024-04-25T16:34:08.442 INFO:tasks.workunit.client.0.smithi098.stderr: 96900K .......... .......... .......... .......... .......... 71% 25.8M 1s 2024-04-25T16:34:08.443 INFO:tasks.workunit.client.0.smithi098.stderr: 96950K .......... .......... .......... .......... .......... 71% 29.5M 1s 2024-04-25T16:34:08.445 INFO:tasks.workunit.client.0.smithi098.stderr: 97000K .......... .......... .......... .......... .......... 71% 29.0M 1s 2024-04-25T16:34:08.446 INFO:tasks.workunit.client.0.smithi098.stderr: 97050K .......... .......... .......... .......... .......... 71% 29.4M 1s 2024-04-25T16:34:08.448 INFO:tasks.workunit.client.0.smithi098.stderr: 97100K .......... .......... .......... .......... .......... 71% 25.3M 1s 2024-04-25T16:34:08.450 INFO:tasks.workunit.client.0.smithi098.stderr: 97150K .......... .......... .......... .......... .......... 71% 29.1M 1s 2024-04-25T16:34:08.452 INFO:tasks.workunit.client.0.smithi098.stderr: 97200K .......... .......... .......... .......... .......... 71% 28.5M 1s 2024-04-25T16:34:08.453 INFO:tasks.workunit.client.0.smithi098.stderr: 97250K .......... .......... .......... .......... .......... 71% 29.1M 1s 2024-04-25T16:34:08.455 INFO:tasks.workunit.client.0.smithi098.stderr: 97300K .......... .......... .......... .......... .......... 71% 24.5M 1s 2024-04-25T16:34:08.457 INFO:tasks.workunit.client.0.smithi098.stderr: 97350K .......... .......... .......... .......... .......... 71% 27.5M 1s 2024-04-25T16:34:08.459 INFO:tasks.workunit.client.0.smithi098.stderr: 97400K .......... .......... .......... .......... .......... 71% 27.6M 1s 2024-04-25T16:34:08.461 INFO:tasks.workunit.client.0.smithi098.stderr: 97450K .......... .......... .......... .......... .......... 71% 27.5M 1s 2024-04-25T16:34:08.463 INFO:tasks.workunit.client.0.smithi098.stderr: 97500K .......... .......... .......... .......... .......... 71% 23.2M 1s 2024-04-25T16:34:08.465 INFO:tasks.workunit.client.0.smithi098.stderr: 97550K .......... .......... .......... .......... .......... 71% 27.3M 1s 2024-04-25T16:34:08.466 INFO:tasks.workunit.client.0.smithi098.stderr: 97600K .......... .......... .......... .......... .......... 71% 27.5M 1s 2024-04-25T16:34:08.468 INFO:tasks.workunit.client.0.smithi098.stderr: 97650K .......... .......... .......... .......... .......... 71% 27.4M 1s 2024-04-25T16:34:08.470 INFO:tasks.workunit.client.0.smithi098.stderr: 97700K .......... .......... .......... .......... .......... 72% 23.3M 1s 2024-04-25T16:34:08.472 INFO:tasks.workunit.client.0.smithi098.stderr: 97750K .......... .......... .......... .......... .......... 72% 26.6M 1s 2024-04-25T16:34:08.474 INFO:tasks.workunit.client.0.smithi098.stderr: 97800K .......... .......... .......... .......... .......... 72% 27.1M 1s 2024-04-25T16:34:08.476 INFO:tasks.workunit.client.0.smithi098.stderr: 97850K .......... .......... .......... .......... .......... 72% 23.2M 1s 2024-04-25T16:34:08.478 INFO:tasks.workunit.client.0.smithi098.stderr: 97900K .......... .......... .......... .......... .......... 72% 20.4M 1s 2024-04-25T16:34:08.481 INFO:tasks.workunit.client.0.smithi098.stderr: 97950K .......... .......... .......... .......... .......... 72% 22.9M 1s 2024-04-25T16:34:08.483 INFO:tasks.workunit.client.0.smithi098.stderr: 98000K .......... .......... .......... .......... .......... 72% 23.4M 1s 2024-04-25T16:34:08.485 INFO:tasks.workunit.client.0.smithi098.stderr: 98050K .......... .......... .......... .......... .......... 72% 24.3M 1s 2024-04-25T16:34:08.487 INFO:tasks.workunit.client.0.smithi098.stderr: 98100K .......... .......... .......... .......... .......... 72% 21.3M 1s 2024-04-25T16:34:08.489 INFO:tasks.workunit.client.0.smithi098.stderr: 98150K .......... .......... .......... .......... .......... 72% 24.7M 1s 2024-04-25T16:34:08.491 INFO:tasks.workunit.client.0.smithi098.stderr: 98200K .......... .......... .......... .......... .......... 72% 23.9M 1s 2024-04-25T16:34:08.493 INFO:tasks.workunit.client.0.smithi098.stderr: 98250K .......... .......... .......... .......... .......... 72% 23.3M 1s 2024-04-25T16:34:08.495 INFO:tasks.workunit.client.0.smithi098.stderr: 98300K .......... .......... .......... .......... .......... 72% 22.3M 1s 2024-04-25T16:34:08.497 INFO:tasks.workunit.client.0.smithi098.stderr: 98350K .......... .......... .......... .......... .......... 72% 35.8M 1s 2024-04-25T16:34:08.498 INFO:tasks.workunit.client.0.smithi098.stderr: 98400K .......... .......... .......... .......... .......... 72% 35.1M 1s 2024-04-25T16:34:08.499 INFO:tasks.workunit.client.0.smithi098.stderr: 98450K .......... .......... .......... .......... .......... 72% 31.7M 1s 2024-04-25T16:34:08.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:08 smithi098 ceph-mon[29162]: pgmap v405: 129 pgs: 129 active+clean; 45 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.7 MiB/s wr, 12 op/s 2024-04-25T16:34:08.502 INFO:tasks.workunit.client.0.smithi098.stderr: 98500K .......... .......... .......... .......... .......... 72% 26.5M 1s 2024-04-25T16:34:08.504 INFO:tasks.workunit.client.0.smithi098.stderr: 98550K .......... .......... .......... .......... .......... 72% 23.5M 1s 2024-04-25T16:34:08.505 INFO:tasks.workunit.client.0.smithi098.stderr: 98600K .......... .......... .......... .......... .......... 72% 29.4M 1s 2024-04-25T16:34:08.506 INFO:tasks.workunit.client.0.smithi098.stderr: 98650K .......... .......... .......... .......... .......... 72% 38.4M 1s 2024-04-25T16:34:08.508 INFO:tasks.workunit.client.0.smithi098.stderr: 98700K .......... .......... .......... .......... .......... 72% 32.9M 1s 2024-04-25T16:34:08.509 INFO:tasks.workunit.client.0.smithi098.stderr: 98750K .......... .......... .......... .......... .......... 72% 37.5M 1s 2024-04-25T16:34:08.510 INFO:tasks.workunit.client.0.smithi098.stderr: 98800K .......... .......... .......... .......... .......... 72% 38.6M 1s 2024-04-25T16:34:08.512 INFO:tasks.workunit.client.0.smithi098.stderr: 98850K .......... .......... .......... .......... .......... 72% 37.9M 1s 2024-04-25T16:34:08.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:08 smithi173 ceph-mon[32556]: pgmap v405: 129 pgs: 129 active+clean; 45 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 3.7 MiB/s wr, 12 op/s 2024-04-25T16:34:08.514 INFO:tasks.workunit.client.0.smithi098.stderr: 98900K .......... .......... .......... .......... .......... 72% 24.4M 1s 2024-04-25T16:34:08.515 INFO:tasks.workunit.client.0.smithi098.stderr: 98950K .......... .......... .......... .......... .......... 72% 31.6M 1s 2024-04-25T16:34:08.517 INFO:tasks.workunit.client.0.smithi098.stderr: 99000K .......... .......... .......... .......... .......... 72% 31.8M 1s 2024-04-25T16:34:08.518 INFO:tasks.workunit.client.0.smithi098.stderr: 99050K .......... .......... .......... .......... .......... 73% 31.3M 1s 2024-04-25T16:34:08.520 INFO:tasks.workunit.client.0.smithi098.stderr: 99100K .......... .......... .......... .......... .......... 73% 28.1M 1s 2024-04-25T16:34:08.522 INFO:tasks.workunit.client.0.smithi098.stderr: 99150K .......... .......... .......... .......... .......... 73% 32.8M 1s 2024-04-25T16:34:08.523 INFO:tasks.workunit.client.0.smithi098.stderr: 99200K .......... .......... .......... .......... .......... 73% 32.6M 1s 2024-04-25T16:34:08.526 INFO:tasks.workunit.client.0.smithi098.stderr: 99250K .......... .......... .......... .......... .......... 73% 31.3M 1s 2024-04-25T16:34:08.527 INFO:tasks.workunit.client.0.smithi098.stderr: 99300K .......... .......... .......... .......... .......... 73% 27.4M 1s 2024-04-25T16:34:08.528 INFO:tasks.workunit.client.0.smithi098.stderr: 99350K .......... .......... .......... .......... .......... 73% 31.0M 1s 2024-04-25T16:34:08.529 INFO:tasks.workunit.client.0.smithi098.stderr: 99400K .......... .......... .......... .......... .......... 73% 34.5M 1s 2024-04-25T16:34:08.531 INFO:tasks.workunit.client.0.smithi098.stderr: 99450K .......... .......... .......... .......... .......... 73% 31.8M 1s 2024-04-25T16:34:08.533 INFO:tasks.workunit.client.0.smithi098.stderr: 99500K .......... .......... .......... .......... .......... 73% 26.5M 1s 2024-04-25T16:34:08.534 INFO:tasks.workunit.client.0.smithi098.stderr: 99550K .......... .......... .......... .......... .......... 73% 31.4M 1s 2024-04-25T16:34:08.536 INFO:tasks.workunit.client.0.smithi098.stderr: 99600K .......... .......... .......... .......... .......... 73% 31.0M 1s 2024-04-25T16:34:08.537 INFO:tasks.workunit.client.0.smithi098.stderr: 99650K .......... .......... .......... .......... .......... 73% 34.7M 1s 2024-04-25T16:34:08.539 INFO:tasks.workunit.client.0.smithi098.stderr: 99700K .......... .......... .......... .......... .......... 73% 30.5M 1s 2024-04-25T16:34:08.540 INFO:tasks.workunit.client.0.smithi098.stderr: 99750K .......... .......... .......... .......... .......... 73% 34.5M 1s 2024-04-25T16:34:08.542 INFO:tasks.workunit.client.0.smithi098.stderr: 99800K .......... .......... .......... .......... .......... 73% 35.3M 1s 2024-04-25T16:34:08.543 INFO:tasks.workunit.client.0.smithi098.stderr: 99850K .......... .......... .......... .......... .......... 73% 31.3M 1s 2024-04-25T16:34:08.545 INFO:tasks.workunit.client.0.smithi098.stderr: 99900K .......... .......... .......... .......... .......... 73% 25.8M 1s 2024-04-25T16:34:08.547 INFO:tasks.workunit.client.0.smithi098.stderr: 99950K .......... .......... .......... .......... .......... 73% 29.4M 1s 2024-04-25T16:34:08.549 INFO:tasks.workunit.client.0.smithi098.stderr:100000K .......... .......... .......... .......... .......... 73% 29.8M 1s 2024-04-25T16:34:08.550 INFO:tasks.workunit.client.0.smithi098.stderr:100050K .......... .......... .......... .......... .......... 73% 32.2M 1s 2024-04-25T16:34:08.552 INFO:tasks.workunit.client.0.smithi098.stderr:100100K .......... .......... .......... .......... .......... 73% 27.3M 1s 2024-04-25T16:34:08.553 INFO:tasks.workunit.client.0.smithi098.stderr:100150K .......... .......... .......... .......... .......... 73% 31.5M 1s 2024-04-25T16:34:08.555 INFO:tasks.workunit.client.0.smithi098.stderr:100200K .......... .......... .......... .......... .......... 73% 31.9M 1s 2024-04-25T16:34:08.557 INFO:tasks.workunit.client.0.smithi098.stderr:100250K .......... .......... .......... .......... .......... 73% 30.3M 1s 2024-04-25T16:34:08.559 INFO:tasks.workunit.client.0.smithi098.stderr:100300K .......... .......... .......... .......... .......... 73% 24.1M 1s 2024-04-25T16:34:08.560 INFO:tasks.workunit.client.0.smithi098.stderr:100350K .......... .......... .......... .......... .......... 73% 29.6M 1s 2024-04-25T16:34:08.562 INFO:tasks.workunit.client.0.smithi098.stderr:100400K .......... .......... .......... .......... .......... 73% 27.1M 1s 2024-04-25T16:34:08.564 INFO:tasks.workunit.client.0.smithi098.stderr:100450K .......... .......... .......... .......... .......... 74% 28.4M 1s 2024-04-25T16:34:08.566 INFO:tasks.workunit.client.0.smithi098.stderr:100500K .......... .......... .......... .......... .......... 74% 23.8M 1s 2024-04-25T16:34:08.567 INFO:tasks.workunit.client.0.smithi098.stderr:100550K .......... .......... .......... .......... .......... 74% 28.3M 1s 2024-04-25T16:34:08.569 INFO:tasks.workunit.client.0.smithi098.stderr:100600K .......... .......... .......... .......... .......... 74% 27.7M 1s 2024-04-25T16:34:08.571 INFO:tasks.workunit.client.0.smithi098.stderr:100650K .......... .......... .......... .......... .......... 74% 27.1M 1s 2024-04-25T16:34:08.573 INFO:tasks.workunit.client.0.smithi098.stderr:100700K .......... .......... .......... .......... .......... 74% 24.0M 1s 2024-04-25T16:34:08.575 INFO:tasks.workunit.client.0.smithi098.stderr:100750K .......... .......... .......... .......... .......... 74% 28.6M 1s 2024-04-25T16:34:08.576 INFO:tasks.workunit.client.0.smithi098.stderr:100800K .......... .......... .......... .......... .......... 74% 30.2M 1s 2024-04-25T16:34:08.578 INFO:tasks.workunit.client.0.smithi098.stderr:100850K .......... .......... .......... .......... .......... 74% 30.0M 1s 2024-04-25T16:34:08.580 INFO:tasks.workunit.client.0.smithi098.stderr:100900K .......... .......... .......... .......... .......... 74% 25.7M 1s 2024-04-25T16:34:08.582 INFO:tasks.workunit.client.0.smithi098.stderr:100950K .......... .......... .......... .......... .......... 74% 29.6M 1s 2024-04-25T16:34:08.583 INFO:tasks.workunit.client.0.smithi098.stderr:101000K .......... .......... .......... .......... .......... 74% 28.6M 1s 2024-04-25T16:34:08.585 INFO:tasks.workunit.client.0.smithi098.stderr:101050K .......... .......... .......... .......... .......... 74% 29.0M 1s 2024-04-25T16:34:08.587 INFO:tasks.workunit.client.0.smithi098.stderr:101100K .......... .......... .......... .......... .......... 74% 24.1M 1s 2024-04-25T16:34:08.589 INFO:tasks.workunit.client.0.smithi098.stderr:101150K .......... .......... .......... .......... .......... 74% 28.7M 1s 2024-04-25T16:34:08.590 INFO:tasks.workunit.client.0.smithi098.stderr:101200K .......... .......... .......... .......... .......... 74% 29.2M 1s 2024-04-25T16:34:08.592 INFO:tasks.workunit.client.0.smithi098.stderr:101250K .......... .......... .......... .......... .......... 74% 28.6M 1s 2024-04-25T16:34:08.594 INFO:tasks.workunit.client.0.smithi098.stderr:101300K .......... .......... .......... .......... .......... 74% 24.0M 1s 2024-04-25T16:34:08.596 INFO:tasks.workunit.client.0.smithi098.stderr:101350K .......... .......... .......... .......... .......... 74% 27.7M 1s 2024-04-25T16:34:08.598 INFO:tasks.workunit.client.0.smithi098.stderr:101400K .......... .......... .......... .......... .......... 74% 28.3M 1s 2024-04-25T16:34:08.599 INFO:tasks.workunit.client.0.smithi098.stderr:101450K .......... .......... .......... .......... .......... 74% 28.5M 1s 2024-04-25T16:34:08.601 INFO:tasks.workunit.client.0.smithi098.stderr:101500K .......... .......... .......... .......... .......... 74% 24.4M 1s 2024-04-25T16:34:08.603 INFO:tasks.workunit.client.0.smithi098.stderr:101550K .......... .......... .......... .......... .......... 74% 27.8M 1s 2024-04-25T16:34:08.605 INFO:tasks.workunit.client.0.smithi098.stderr:101600K .......... .......... .......... .......... .......... 74% 28.1M 1s 2024-04-25T16:34:08.607 INFO:tasks.workunit.client.0.smithi098.stderr:101650K .......... .......... .......... .......... .......... 74% 28.4M 1s 2024-04-25T16:34:08.609 INFO:tasks.workunit.client.0.smithi098.stderr:101700K .......... .......... .......... .......... .......... 74% 23.3M 1s 2024-04-25T16:34:08.610 INFO:tasks.workunit.client.0.smithi098.stderr:101750K .......... .......... .......... .......... .......... 74% 28.0M 1s 2024-04-25T16:34:08.612 INFO:tasks.workunit.client.0.smithi098.stderr:101800K .......... .......... .......... .......... .......... 75% 27.3M 1s 2024-04-25T16:34:08.614 INFO:tasks.workunit.client.0.smithi098.stderr:101850K .......... .......... .......... .......... .......... 75% 26.9M 1s 2024-04-25T16:34:08.616 INFO:tasks.workunit.client.0.smithi098.stderr:101900K .......... .......... .......... .......... .......... 75% 23.7M 1s 2024-04-25T16:34:08.618 INFO:tasks.workunit.client.0.smithi098.stderr:101950K .......... .......... .......... .......... .......... 75% 27.3M 1s 2024-04-25T16:34:08.619 INFO:tasks.workunit.client.0.smithi098.stderr:102000K .......... .......... .......... .......... .......... 75% 27.5M 1s 2024-04-25T16:34:08.622 INFO:tasks.workunit.client.0.smithi098.stderr:102050K .......... .......... .......... .......... .......... 75% 26.5M 1s 2024-04-25T16:34:08.623 INFO:tasks.workunit.client.0.smithi098.stderr:102100K .......... .......... .......... .......... .......... 75% 23.0M 1s 2024-04-25T16:34:08.625 INFO:tasks.workunit.client.0.smithi098.stderr:102150K .......... .......... .......... .......... .......... 75% 26.8M 1s 2024-04-25T16:34:08.627 INFO:tasks.workunit.client.0.smithi098.stderr:102200K .......... .......... .......... .......... .......... 75% 27.2M 1s 2024-04-25T16:34:08.629 INFO:tasks.workunit.client.0.smithi098.stderr:102250K .......... .......... .......... .......... .......... 75% 26.1M 1s 2024-04-25T16:34:08.631 INFO:tasks.workunit.client.0.smithi098.stderr:102300K .......... .......... .......... .......... .......... 75% 22.9M 1s 2024-04-25T16:34:08.633 INFO:tasks.workunit.client.0.smithi098.stderr:102350K .......... .......... .......... .......... .......... 75% 25.5M 1s 2024-04-25T16:34:08.634 INFO:tasks.workunit.client.0.smithi098.stderr:102400K .......... .......... .......... .......... .......... 75% 31.4M 1s 2024-04-25T16:34:08.636 INFO:tasks.workunit.client.0.smithi098.stderr:102450K .......... .......... .......... .......... .......... 75% 35.1M 1s 2024-04-25T16:34:08.637 INFO:tasks.workunit.client.0.smithi098.stderr:102500K .......... .......... .......... .......... .......... 75% 30.1M 1s 2024-04-25T16:34:08.639 INFO:tasks.workunit.client.0.smithi098.stderr:102550K .......... .......... .......... .......... .......... 75% 38.5M 1s 2024-04-25T16:34:08.640 INFO:tasks.workunit.client.0.smithi098.stderr:102600K .......... .......... .......... .......... .......... 75% 41.1M 1s 2024-04-25T16:34:08.641 INFO:tasks.workunit.client.0.smithi098.stderr:102650K .......... .......... .......... .......... .......... 75% 41.1M 1s 2024-04-25T16:34:08.642 INFO:tasks.workunit.client.0.smithi098.stderr:102700K .......... .......... .......... .......... .......... 75% 33.9M 1s 2024-04-25T16:34:08.644 INFO:tasks.workunit.client.0.smithi098.stderr:102750K .......... .......... .......... .......... .......... 75% 28.0M 1s 2024-04-25T16:34:08.646 INFO:tasks.workunit.client.0.smithi098.stderr:102800K .......... .......... .......... .......... .......... 75% 28.4M 1s 2024-04-25T16:34:08.648 INFO:tasks.workunit.client.0.smithi098.stderr:102850K .......... .......... .......... .......... .......... 75% 32.8M 1s 2024-04-25T16:34:08.649 INFO:tasks.workunit.client.0.smithi098.stderr:102900K .......... .......... .......... .......... .......... 75% 26.9M 1s 2024-04-25T16:34:08.651 INFO:tasks.workunit.client.0.smithi098.stderr:102950K .......... .......... .......... .......... .......... 75% 34.2M 1s 2024-04-25T16:34:08.653 INFO:tasks.workunit.client.0.smithi098.stderr:103000K .......... .......... .......... .......... .......... 75% 26.6M 1s 2024-04-25T16:34:08.654 INFO:tasks.workunit.client.0.smithi098.stderr:103050K .......... .......... .......... .......... .......... 75% 33.2M 1s 2024-04-25T16:34:08.656 INFO:tasks.workunit.client.0.smithi098.stderr:103100K .......... .......... .......... .......... .......... 75% 30.9M 1s 2024-04-25T16:34:08.657 INFO:tasks.workunit.client.0.smithi098.stderr:103150K .......... .......... .......... .......... .......... 76% 37.4M 1s 2024-04-25T16:34:08.658 INFO:tasks.workunit.client.0.smithi098.stderr:103200K .......... .......... .......... .......... .......... 76% 39.2M 1s 2024-04-25T16:34:08.659 INFO:tasks.workunit.client.0.smithi098.stderr:103250K .......... .......... .......... .......... .......... 76% 39.4M 1s 2024-04-25T16:34:08.661 INFO:tasks.workunit.client.0.smithi098.stderr:103300K .......... .......... .......... .......... .......... 76% 33.5M 1s 2024-04-25T16:34:08.662 INFO:tasks.workunit.client.0.smithi098.stderr:103350K .......... .......... .......... .......... .......... 76% 39.1M 1s 2024-04-25T16:34:08.664 INFO:tasks.workunit.client.0.smithi098.stderr:103400K .......... .......... .......... .......... .......... 76% 38.4M 1s 2024-04-25T16:34:08.665 INFO:tasks.workunit.client.0.smithi098.stderr:103450K .......... .......... .......... .......... .......... 76% 38.4M 1s 2024-04-25T16:34:08.666 INFO:tasks.workunit.client.0.smithi098.stderr:103500K .......... .......... .......... .......... .......... 76% 32.8M 1s 2024-04-25T16:34:08.668 INFO:tasks.workunit.client.0.smithi098.stderr:103550K .......... .......... .......... .......... .......... 76% 37.7M 1s 2024-04-25T16:34:08.669 INFO:tasks.workunit.client.0.smithi098.stderr:103600K .......... .......... .......... .......... .......... 76% 32.9M 1s 2024-04-25T16:34:08.671 INFO:tasks.workunit.client.0.smithi098.stderr:103650K .......... .......... .......... .......... .......... 76% 31.8M 1s 2024-04-25T16:34:08.672 INFO:tasks.workunit.client.0.smithi098.stderr:103700K .......... .......... .......... .......... .......... 76% 27.0M 1s 2024-04-25T16:34:08.674 INFO:tasks.workunit.client.0.smithi098.stderr:103750K .......... .......... .......... .......... .......... 76% 31.4M 1s 2024-04-25T16:34:08.676 INFO:tasks.workunit.client.0.smithi098.stderr:103800K .......... .......... .......... .......... .......... 76% 31.0M 1s 2024-04-25T16:34:08.678 INFO:tasks.workunit.client.0.smithi098.stderr:103850K .......... .......... .......... .......... .......... 76% 31.9M 1s 2024-04-25T16:34:08.679 INFO:tasks.workunit.client.0.smithi098.stderr:103900K .......... .......... .......... .......... .......... 76% 26.7M 1s 2024-04-25T16:34:08.681 INFO:tasks.workunit.client.0.smithi098.stderr:103950K .......... .......... .......... .......... .......... 76% 30.7M 1s 2024-04-25T16:34:08.682 INFO:tasks.workunit.client.0.smithi098.stderr:104000K .......... .......... .......... .......... .......... 76% 30.8M 1s 2024-04-25T16:34:08.684 INFO:tasks.workunit.client.0.smithi098.stderr:104050K .......... .......... .......... .......... .......... 76% 29.3M 1s 2024-04-25T16:34:08.686 INFO:tasks.workunit.client.0.smithi098.stderr:104100K .......... .......... .......... .......... .......... 76% 26.4M 1s 2024-04-25T16:34:08.687 INFO:tasks.workunit.client.0.smithi098.stderr:104150K .......... .......... .......... .......... .......... 76% 30.7M 1s 2024-04-25T16:34:08.689 INFO:tasks.workunit.client.0.smithi098.stderr:104200K .......... .......... .......... .......... .......... 76% 30.4M 1s 2024-04-25T16:34:08.690 INFO:tasks.workunit.client.0.smithi098.stderr:104250K .......... .......... .......... .......... .......... 76% 30.8M 1s 2024-04-25T16:34:08.692 INFO:tasks.workunit.client.0.smithi098.stderr:104300K .......... .......... .......... .......... .......... 76% 26.1M 1s 2024-04-25T16:34:08.694 INFO:tasks.workunit.client.0.smithi098.stderr:104350K .......... .......... .......... .......... .......... 76% 28.1M 1s 2024-04-25T16:34:08.696 INFO:tasks.workunit.client.0.smithi098.stderr:104400K .......... .......... .......... .......... .......... 76% 30.1M 1s 2024-04-25T16:34:08.697 INFO:tasks.workunit.client.0.smithi098.stderr:104450K .......... .......... .......... .......... .......... 76% 31.6M 1s 2024-04-25T16:34:08.699 INFO:tasks.workunit.client.0.smithi098.stderr:104500K .......... .......... .......... .......... .......... 77% 27.0M 1s 2024-04-25T16:34:08.700 INFO:tasks.workunit.client.0.smithi098.stderr:104550K .......... .......... .......... .......... .......... 77% 31.4M 1s 2024-04-25T16:34:08.702 INFO:tasks.workunit.client.0.smithi098.stderr:104600K .......... .......... .......... .......... .......... 77% 31.4M 1s 2024-04-25T16:34:08.704 INFO:tasks.workunit.client.0.smithi098.stderr:104650K .......... .......... .......... .......... .......... 77% 31.9M 1s 2024-04-25T16:34:08.705 INFO:tasks.workunit.client.0.smithi098.stderr:104700K .......... .......... .......... .......... .......... 77% 26.9M 1s 2024-04-25T16:34:08.707 INFO:tasks.workunit.client.0.smithi098.stderr:104750K .......... .......... .......... .......... .......... 77% 30.8M 1s 2024-04-25T16:34:08.709 INFO:tasks.workunit.client.0.smithi098.stderr:104800K .......... .......... .......... .......... .......... 77% 31.0M 1s 2024-04-25T16:34:08.710 INFO:tasks.workunit.client.0.smithi098.stderr:104850K .......... .......... .......... .......... .......... 77% 31.5M 1s 2024-04-25T16:34:08.712 INFO:tasks.workunit.client.0.smithi098.stderr:104900K .......... .......... .......... .......... .......... 77% 24.9M 1s 2024-04-25T16:34:08.714 INFO:tasks.workunit.client.0.smithi098.stderr:104950K .......... .......... .......... .......... .......... 77% 27.9M 1s 2024-04-25T16:34:08.715 INFO:tasks.workunit.client.0.smithi098.stderr:105000K .......... .......... .......... .......... .......... 77% 28.9M 1s 2024-04-25T16:34:08.717 INFO:tasks.workunit.client.0.smithi098.stderr:105050K .......... .......... .......... .......... .......... 77% 28.4M 1s 2024-04-25T16:34:08.719 INFO:tasks.workunit.client.0.smithi098.stderr:105100K .......... .......... .......... .......... .......... 77% 25.5M 1s 2024-04-25T16:34:08.721 INFO:tasks.workunit.client.0.smithi098.stderr:105150K .......... .......... .......... .......... .......... 77% 30.4M 1s 2024-04-25T16:34:08.722 INFO:tasks.workunit.client.0.smithi098.stderr:105200K .......... .......... .......... .......... .......... 77% 30.5M 1s 2024-04-25T16:34:08.724 INFO:tasks.workunit.client.0.smithi098.stderr:105250K .......... .......... .......... .......... .......... 77% 29.6M 1s 2024-04-25T16:34:08.726 INFO:tasks.workunit.client.0.smithi098.stderr:105300K .......... .......... .......... .......... .......... 77% 25.7M 1s 2024-04-25T16:34:08.727 INFO:tasks.workunit.client.0.smithi098.stderr:105350K .......... .......... .......... .......... .......... 77% 29.6M 1s 2024-04-25T16:34:08.730 INFO:tasks.workunit.client.0.smithi098.stderr:105400K .......... .......... .......... .......... .......... 77% 30.0M 1s 2024-04-25T16:34:08.731 INFO:tasks.workunit.client.0.smithi098.stderr:105450K .......... .......... .......... .......... .......... 77% 29.7M 1s 2024-04-25T16:34:08.733 INFO:tasks.workunit.client.0.smithi098.stderr:105500K .......... .......... .......... .......... .......... 77% 25.2M 1s 2024-04-25T16:34:08.734 INFO:tasks.workunit.client.0.smithi098.stderr:105550K .......... .......... .......... .......... .......... 77% 28.5M 1s 2024-04-25T16:34:08.736 INFO:tasks.workunit.client.0.smithi098.stderr:105600K .......... .......... .......... .......... .......... 77% 29.3M 1s 2024-04-25T16:34:08.738 INFO:tasks.workunit.client.0.smithi098.stderr:105650K .......... .......... .......... .......... .......... 77% 28.8M 1s 2024-04-25T16:34:08.740 INFO:tasks.workunit.client.0.smithi098.stderr:105700K .......... .......... .......... .......... .......... 77% 24.9M 1s 2024-04-25T16:34:08.741 INFO:tasks.workunit.client.0.smithi098.stderr:105750K .......... .......... .......... .......... .......... 77% 29.9M 1s 2024-04-25T16:34:08.743 INFO:tasks.workunit.client.0.smithi098.stderr:105800K .......... .......... .......... .......... .......... 77% 29.1M 1s 2024-04-25T16:34:08.745 INFO:tasks.workunit.client.0.smithi098.stderr:105850K .......... .......... .......... .......... .......... 78% 28.2M 1s 2024-04-25T16:34:08.747 INFO:tasks.workunit.client.0.smithi098.stderr:105900K .......... .......... .......... .......... .......... 78% 24.3M 1s 2024-04-25T16:34:08.749 INFO:tasks.workunit.client.0.smithi098.stderr:105950K .......... .......... .......... .......... .......... 78% 28.4M 1s 2024-04-25T16:34:08.750 INFO:tasks.workunit.client.0.smithi098.stderr:106000K .......... .......... .......... .......... .......... 78% 27.8M 1s 2024-04-25T16:34:08.752 INFO:tasks.workunit.client.0.smithi098.stderr:106050K .......... .......... .......... .......... .......... 78% 27.5M 1s 2024-04-25T16:34:08.754 INFO:tasks.workunit.client.0.smithi098.stderr:106100K .......... .......... .......... .......... .......... 78% 23.7M 1s 2024-04-25T16:34:08.756 INFO:tasks.workunit.client.0.smithi098.stderr:106150K .......... .......... .......... .......... .......... 78% 27.4M 1s 2024-04-25T16:34:08.758 INFO:tasks.workunit.client.0.smithi098.stderr:106200K .......... .......... .......... .......... .......... 78% 27.5M 1s 2024-04-25T16:34:08.760 INFO:tasks.workunit.client.0.smithi098.stderr:106250K .......... .......... .......... .......... .......... 78% 27.7M 1s 2024-04-25T16:34:08.762 INFO:tasks.workunit.client.0.smithi098.stderr:106300K .......... .......... .......... .......... .......... 78% 23.6M 1s 2024-04-25T16:34:08.763 INFO:tasks.workunit.client.0.smithi098.stderr:106350K .......... .......... .......... .......... .......... 78% 26.7M 1s 2024-04-25T16:34:08.765 INFO:tasks.workunit.client.0.smithi098.stderr:106400K .......... .......... .......... .......... .......... 78% 27.1M 1s 2024-04-25T16:34:08.768 INFO:tasks.workunit.client.0.smithi098.stderr:106450K .......... .......... .......... .......... .......... 78% 27.2M 1s 2024-04-25T16:34:08.769 INFO:tasks.workunit.client.0.smithi098.stderr:106500K .......... .......... .......... .......... .......... 78% 24.9M 1s 2024-04-25T16:34:08.770 INFO:tasks.workunit.client.0.smithi098.stderr:106550K .......... .......... .......... .......... .......... 78% 38.8M 1s 2024-04-25T16:34:08.772 INFO:tasks.workunit.client.0.smithi098.stderr:106600K .......... .......... .......... .......... .......... 78% 33.4M 1s 2024-04-25T16:34:08.773 INFO:tasks.workunit.client.0.smithi098.stderr:106650K .......... .......... .......... .......... .......... 78% 39.1M 1s 2024-04-25T16:34:08.774 INFO:tasks.workunit.client.0.smithi098.stderr:106700K .......... .......... .......... .......... .......... 78% 33.4M 1s 2024-04-25T16:34:08.777 INFO:tasks.workunit.client.0.smithi098.stderr:106750K .......... .......... .......... .......... .......... 78% 21.1M 1s 2024-04-25T16:34:08.779 INFO:tasks.workunit.client.0.smithi098.stderr:106800K .......... .......... .......... .......... .......... 78% 24.3M 1s 2024-04-25T16:34:08.780 INFO:tasks.workunit.client.0.smithi098.stderr:106850K .......... .......... .......... .......... .......... 78% 32.8M 1s 2024-04-25T16:34:08.782 INFO:tasks.workunit.client.0.smithi098.stderr:106900K .......... .......... .......... .......... .......... 78% 24.9M 1s 2024-04-25T16:34:08.784 INFO:tasks.workunit.client.0.smithi098.stderr:106950K .......... .......... .......... .......... .......... 78% 27.3M 1s 2024-04-25T16:34:08.785 INFO:tasks.workunit.client.0.smithi098.stderr:107000K .......... .......... .......... .......... .......... 78% 37.6M 1s 2024-04-25T16:34:08.787 INFO:tasks.workunit.client.0.smithi098.stderr:107050K .......... .......... .......... .......... .......... 78% 35.4M 1s 2024-04-25T16:34:08.788 INFO:tasks.workunit.client.0.smithi098.stderr:107100K .......... .......... .......... .......... .......... 78% 30.8M 1s 2024-04-25T16:34:08.789 INFO:tasks.workunit.client.0.smithi098.stderr:107150K .......... .......... .......... .......... .......... 78% 36.3M 1s 2024-04-25T16:34:08.791 INFO:tasks.workunit.client.0.smithi098.stderr:107200K .......... .......... .......... .......... .......... 79% 36.6M 1s 2024-04-25T16:34:08.792 INFO:tasks.workunit.client.0.smithi098.stderr:107250K .......... .......... .......... .......... .......... 79% 35.9M 1s 2024-04-25T16:34:08.794 INFO:tasks.workunit.client.0.smithi098.stderr:107300K .......... .......... .......... .......... .......... 79% 28.8M 1s 2024-04-25T16:34:08.795 INFO:tasks.workunit.client.0.smithi098.stderr:107350K .......... .......... .......... .......... .......... 79% 35.7M 1s 2024-04-25T16:34:08.797 INFO:tasks.workunit.client.0.smithi098.stderr:107400K .......... .......... .......... .......... .......... 79% 34.8M 1s 2024-04-25T16:34:08.798 INFO:tasks.workunit.client.0.smithi098.stderr:107450K .......... .......... .......... .......... .......... 79% 35.3M 1s 2024-04-25T16:34:08.800 INFO:tasks.workunit.client.0.smithi098.stderr:107500K .......... .......... .......... .......... .......... 79% 30.2M 1s 2024-04-25T16:34:08.801 INFO:tasks.workunit.client.0.smithi098.stderr:107550K .......... .......... .......... .......... .......... 79% 35.2M 1s 2024-04-25T16:34:08.803 INFO:tasks.workunit.client.0.smithi098.stderr:107600K .......... .......... .......... .......... .......... 79% 35.0M 1s 2024-04-25T16:34:08.804 INFO:tasks.workunit.client.0.smithi098.stderr:107650K .......... .......... .......... .......... .......... 79% 34.7M 1s 2024-04-25T16:34:08.806 INFO:tasks.workunit.client.0.smithi098.stderr:107700K .......... .......... .......... .......... .......... 79% 27.7M 1s 2024-04-25T16:34:08.807 INFO:tasks.workunit.client.0.smithi098.stderr:107750K .......... .......... .......... .......... .......... 79% 31.8M 1s 2024-04-25T16:34:08.809 INFO:tasks.workunit.client.0.smithi098.stderr:107800K .......... .......... .......... .......... .......... 79% 32.5M 1s 2024-04-25T16:34:08.810 INFO:tasks.workunit.client.0.smithi098.stderr:107850K .......... .......... .......... .......... .......... 79% 31.9M 1s 2024-04-25T16:34:08.812 INFO:tasks.workunit.client.0.smithi098.stderr:107900K .......... .......... .......... .......... .......... 79% 27.7M 1s 2024-04-25T16:34:08.813 INFO:tasks.workunit.client.0.smithi098.stderr:107950K .......... .......... .......... .......... .......... 79% 32.2M 1s 2024-04-25T16:34:08.815 INFO:tasks.workunit.client.0.smithi098.stderr:108000K .......... .......... .......... .......... .......... 79% 31.5M 1s 2024-04-25T16:34:08.817 INFO:tasks.workunit.client.0.smithi098.stderr:108050K .......... .......... .......... .......... .......... 79% 31.3M 1s 2024-04-25T16:34:08.818 INFO:tasks.workunit.client.0.smithi098.stderr:108100K .......... .......... .......... .......... .......... 79% 27.7M 1s 2024-04-25T16:34:08.820 INFO:tasks.workunit.client.0.smithi098.stderr:108150K .......... .......... .......... .......... .......... 79% 31.6M 1s 2024-04-25T16:34:08.821 INFO:tasks.workunit.client.0.smithi098.stderr:108200K .......... .......... .......... .......... .......... 79% 32.2M 1s 2024-04-25T16:34:08.823 INFO:tasks.workunit.client.0.smithi098.stderr:108250K .......... .......... .......... .......... .......... 79% 31.4M 1s 2024-04-25T16:34:08.825 INFO:tasks.workunit.client.0.smithi098.stderr:108300K .......... .......... .......... .......... .......... 79% 28.7M 1s 2024-04-25T16:34:08.826 INFO:tasks.workunit.client.0.smithi098.stderr:108350K .......... .......... .......... .......... .......... 79% 34.2M 1s 2024-04-25T16:34:08.828 INFO:tasks.workunit.client.0.smithi098.stderr:108400K .......... .......... .......... .......... .......... 79% 33.8M 1s 2024-04-25T16:34:08.829 INFO:tasks.workunit.client.0.smithi098.stderr:108450K .......... .......... .......... .......... .......... 79% 33.5M 1s 2024-04-25T16:34:08.831 INFO:tasks.workunit.client.0.smithi098.stderr:108500K .......... .......... .......... .......... .......... 79% 27.7M 1s 2024-04-25T16:34:08.833 INFO:tasks.workunit.client.0.smithi098.stderr:108550K .......... .......... .......... .......... .......... 79% 30.4M 1s 2024-04-25T16:34:08.834 INFO:tasks.workunit.client.0.smithi098.stderr:108600K .......... .......... .......... .......... .......... 80% 31.6M 1s 2024-04-25T16:34:08.836 INFO:tasks.workunit.client.0.smithi098.stderr:108650K .......... .......... .......... .......... .......... 80% 31.1M 1s 2024-04-25T16:34:08.837 INFO:tasks.workunit.client.0.smithi098.stderr:108700K .......... .......... .......... .......... .......... 80% 27.1M 1s 2024-04-25T16:34:08.839 INFO:tasks.workunit.client.0.smithi098.stderr:108750K .......... .......... .......... .......... .......... 80% 31.1M 1s 2024-04-25T16:34:08.841 INFO:tasks.workunit.client.0.smithi098.stderr:108800K .......... .......... .......... .......... .......... 80% 31.4M 1s 2024-04-25T16:34:08.842 INFO:tasks.workunit.client.0.smithi098.stderr:108850K .......... .......... .......... .......... .......... 80% 30.3M 1s 2024-04-25T16:34:08.844 INFO:tasks.workunit.client.0.smithi098.stderr:108900K .......... .......... .......... .......... .......... 80% 25.4M 1s 2024-04-25T16:34:08.846 INFO:tasks.workunit.client.0.smithi098.stderr:108950K .......... .......... .......... .......... .......... 80% 28.7M 1s 2024-04-25T16:34:08.847 INFO:tasks.workunit.client.0.smithi098.stderr:109000K .......... .......... .......... .......... .......... 80% 30.1M 1s 2024-04-25T16:34:08.849 INFO:tasks.workunit.client.0.smithi098.stderr:109050K .......... .......... .......... .......... .......... 80% 29.3M 1s 2024-04-25T16:34:08.851 INFO:tasks.workunit.client.0.smithi098.stderr:109100K .......... .......... .......... .......... .......... 80% 23.8M 1s 2024-04-25T16:34:08.853 INFO:tasks.workunit.client.0.smithi098.stderr:109150K .......... .......... .......... .......... .......... 80% 29.4M 1s 2024-04-25T16:34:08.854 INFO:tasks.workunit.client.0.smithi098.stderr:109200K .......... .......... .......... .......... .......... 80% 30.1M 1s 2024-04-25T16:34:08.856 INFO:tasks.workunit.client.0.smithi098.stderr:109250K .......... .......... .......... .......... .......... 80% 29.2M 1s 2024-04-25T16:34:08.858 INFO:tasks.workunit.client.0.smithi098.stderr:109300K .......... .......... .......... .......... .......... 80% 24.9M 1s 2024-04-25T16:34:08.860 INFO:tasks.workunit.client.0.smithi098.stderr:109350K .......... .......... .......... .......... .......... 80% 27.6M 1s 2024-04-25T16:34:08.861 INFO:tasks.workunit.client.0.smithi098.stderr:109400K .......... .......... .......... .......... .......... 80% 29.1M 1s 2024-04-25T16:34:08.863 INFO:tasks.workunit.client.0.smithi098.stderr:109450K .......... .......... .......... .......... .......... 80% 28.8M 1s 2024-04-25T16:34:08.865 INFO:tasks.workunit.client.0.smithi098.stderr:109500K .......... .......... .......... .......... .......... 80% 24.8M 1s 2024-04-25T16:34:08.867 INFO:tasks.workunit.client.0.smithi098.stderr:109550K .......... .......... .......... .......... .......... 80% 27.5M 1s 2024-04-25T16:34:08.868 INFO:tasks.workunit.client.0.smithi098.stderr:109600K .......... .......... .......... .......... .......... 80% 28.5M 1s 2024-04-25T16:34:08.870 INFO:tasks.workunit.client.0.smithi098.stderr:109650K .......... .......... .......... .......... .......... 80% 28.4M 1s 2024-04-25T16:34:08.872 INFO:tasks.workunit.client.0.smithi098.stderr:109700K .......... .......... .......... .......... .......... 80% 24.0M 1s 2024-04-25T16:34:08.874 INFO:tasks.workunit.client.0.smithi098.stderr:109750K .......... .......... .......... .......... .......... 80% 27.8M 1s 2024-04-25T16:34:08.876 INFO:tasks.workunit.client.0.smithi098.stderr:109800K .......... .......... .......... .......... .......... 80% 26.9M 1s 2024-04-25T16:34:08.878 INFO:tasks.workunit.client.0.smithi098.stderr:109850K .......... .......... .......... .......... .......... 80% 28.2M 1s 2024-04-25T16:34:08.880 INFO:tasks.workunit.client.0.smithi098.stderr:109900K .......... .......... .......... .......... .......... 80% 24.3M 1s 2024-04-25T16:34:08.881 INFO:tasks.workunit.client.0.smithi098.stderr:109950K .......... .......... .......... .......... .......... 81% 28.2M 1s 2024-04-25T16:34:08.883 INFO:tasks.workunit.client.0.smithi098.stderr:110000K .......... .......... .......... .......... .......... 81% 28.1M 1s 2024-04-25T16:34:08.885 INFO:tasks.workunit.client.0.smithi098.stderr:110050K .......... .......... .......... .......... .......... 81% 28.5M 1s 2024-04-25T16:34:08.887 INFO:tasks.workunit.client.0.smithi098.stderr:110100K .......... .......... .......... .......... .......... 81% 21.5M 1s 2024-04-25T16:34:08.889 INFO:tasks.workunit.client.0.smithi098.stderr:110150K .......... .......... .......... .......... .......... 81% 26.8M 1s 2024-04-25T16:34:08.891 INFO:tasks.workunit.client.0.smithi098.stderr:110200K .......... .......... .......... .......... .......... 81% 27.0M 1s 2024-04-25T16:34:08.893 INFO:tasks.workunit.client.0.smithi098.stderr:110250K .......... .......... .......... .......... .......... 81% 26.6M 1s 2024-04-25T16:34:08.895 INFO:tasks.workunit.client.0.smithi098.stderr:110300K .......... .......... .......... .......... .......... 81% 20.5M 1s 2024-04-25T16:34:08.897 INFO:tasks.workunit.client.0.smithi098.stderr:110350K .......... .......... .......... .......... .......... 81% 24.2M 1s 2024-04-25T16:34:08.899 INFO:tasks.workunit.client.0.smithi098.stderr:110400K .......... .......... .......... .......... .......... 81% 22.6M 1s 2024-04-25T16:34:08.901 INFO:tasks.workunit.client.0.smithi098.stderr:110450K .......... .......... .......... .......... .......... 81% 26.1M 1s 2024-04-25T16:34:08.903 INFO:tasks.workunit.client.0.smithi098.stderr:110500K .......... .......... .......... .......... .......... 81% 22.9M 1s 2024-04-25T16:34:08.905 INFO:tasks.workunit.client.0.smithi098.stderr:110550K .......... .......... .......... .......... .......... 81% 27.2M 1s 2024-04-25T16:34:08.907 INFO:tasks.workunit.client.0.smithi098.stderr:110600K .......... .......... .......... .......... .......... 81% 32.0M 1s 2024-04-25T16:34:08.908 INFO:tasks.workunit.client.0.smithi098.stderr:110650K .......... .......... .......... .......... .......... 81% 31.4M 1s 2024-04-25T16:34:08.910 INFO:tasks.workunit.client.0.smithi098.stderr:110700K .......... .......... .......... .......... .......... 81% 33.2M 1s 2024-04-25T16:34:08.911 INFO:tasks.workunit.client.0.smithi098.stderr:110750K .......... .......... .......... .......... .......... 81% 39.9M 1s 2024-04-25T16:34:08.912 INFO:tasks.workunit.client.0.smithi098.stderr:110800K .......... .......... .......... .......... .......... 81% 33.8M 1s 2024-04-25T16:34:08.914 INFO:tasks.workunit.client.0.smithi098.stderr:110850K .......... .......... .......... .......... .......... 81% 30.4M 1s 2024-04-25T16:34:08.916 INFO:tasks.workunit.client.0.smithi098.stderr:110900K .......... .......... .......... .......... .......... 81% 26.6M 1s 2024-04-25T16:34:08.917 INFO:tasks.workunit.client.0.smithi098.stderr:110950K .......... .......... .......... .......... .......... 81% 33.5M 1s 2024-04-25T16:34:08.918 INFO:tasks.workunit.client.0.smithi098.stderr:111000K .......... .......... .......... .......... .......... 81% 36.4M 1s 2024-04-25T16:34:08.920 INFO:tasks.workunit.client.0.smithi098.stderr:111050K .......... .......... .......... .......... .......... 81% 34.9M 1s 2024-04-25T16:34:08.921 INFO:tasks.workunit.client.0.smithi098.stderr:111100K .......... .......... .......... .......... .......... 81% 31.7M 1s 2024-04-25T16:34:08.923 INFO:tasks.workunit.client.0.smithi098.stderr:111150K .......... .......... .......... .......... .......... 81% 35.2M 1s 2024-04-25T16:34:08.924 INFO:tasks.workunit.client.0.smithi098.stderr:111200K .......... .......... .......... .......... .......... 81% 27.7M 1s 2024-04-25T16:34:08.926 INFO:tasks.workunit.client.0.smithi098.stderr:111250K .......... .......... .......... .......... .......... 81% 31.0M 1s 2024-04-25T16:34:08.928 INFO:tasks.workunit.client.0.smithi098.stderr:111300K .......... .......... .......... .......... .......... 82% 30.1M 1s 2024-04-25T16:34:08.929 INFO:tasks.workunit.client.0.smithi098.stderr:111350K .......... .......... .......... .......... .......... 82% 36.8M 1s 2024-04-25T16:34:08.930 INFO:tasks.workunit.client.0.smithi098.stderr:111400K .......... .......... .......... .......... .......... 82% 36.2M 1s 2024-04-25T16:34:08.932 INFO:tasks.workunit.client.0.smithi098.stderr:111450K .......... .......... .......... .......... .......... 82% 36.8M 1s 2024-04-25T16:34:08.933 INFO:tasks.workunit.client.0.smithi098.stderr:111500K .......... .......... .......... .......... .......... 82% 30.0M 1s 2024-04-25T16:34:08.935 INFO:tasks.workunit.client.0.smithi098.stderr:111550K .......... .......... .......... .......... .......... 82% 31.9M 1s 2024-04-25T16:34:08.936 INFO:tasks.workunit.client.0.smithi098.stderr:111600K .......... .......... .......... .......... .......... 82% 32.2M 1s 2024-04-25T16:34:08.938 INFO:tasks.workunit.client.0.smithi098.stderr:111650K .......... .......... .......... .......... .......... 82% 32.9M 1s 2024-04-25T16:34:08.940 INFO:tasks.workunit.client.0.smithi098.stderr:111700K .......... .......... .......... .......... .......... 82% 26.9M 1s 2024-04-25T16:34:08.941 INFO:tasks.workunit.client.0.smithi098.stderr:111750K .......... .......... .......... .......... .......... 82% 32.2M 1s 2024-04-25T16:34:08.943 INFO:tasks.workunit.client.0.smithi098.stderr:111800K .......... .......... .......... .......... .......... 82% 32.1M 1s 2024-04-25T16:34:08.944 INFO:tasks.workunit.client.0.smithi098.stderr:111850K .......... .......... .......... .......... .......... 82% 32.5M 1s 2024-04-25T16:34:08.946 INFO:tasks.workunit.client.0.smithi098.stderr:111900K .......... .......... .......... .......... .......... 82% 27.6M 1s 2024-04-25T16:34:08.948 INFO:tasks.workunit.client.0.smithi098.stderr:111950K .......... .......... .......... .......... .......... 82% 32.5M 1s 2024-04-25T16:34:08.949 INFO:tasks.workunit.client.0.smithi098.stderr:112000K .......... .......... .......... .......... .......... 82% 32.5M 1s 2024-04-25T16:34:08.951 INFO:tasks.workunit.client.0.smithi098.stderr:112050K .......... .......... .......... .......... .......... 82% 32.0M 1s 2024-04-25T16:34:08.952 INFO:tasks.workunit.client.0.smithi098.stderr:112100K .......... .......... .......... .......... .......... 82% 26.9M 1s 2024-04-25T16:34:08.954 INFO:tasks.workunit.client.0.smithi098.stderr:112150K .......... .......... .......... .......... .......... 82% 31.2M 1s 2024-04-25T16:34:08.955 INFO:tasks.workunit.client.0.smithi098.stderr:112200K .......... .......... .......... .......... .......... 82% 30.8M 1s 2024-04-25T16:34:08.957 INFO:tasks.workunit.client.0.smithi098.stderr:112250K .......... .......... .......... .......... .......... 82% 31.3M 1s 2024-04-25T16:34:08.959 INFO:tasks.workunit.client.0.smithi098.stderr:112300K .......... .......... .......... .......... .......... 82% 26.6M 1s 2024-04-25T16:34:08.960 INFO:tasks.workunit.client.0.smithi098.stderr:112350K .......... .......... .......... .......... .......... 82% 31.2M 1s 2024-04-25T16:34:08.962 INFO:tasks.workunit.client.0.smithi098.stderr:112400K .......... .......... .......... .......... .......... 82% 31.7M 1s 2024-04-25T16:34:08.964 INFO:tasks.workunit.client.0.smithi098.stderr:112450K .......... .......... .......... .......... .......... 82% 30.8M 1s 2024-04-25T16:34:08.965 INFO:tasks.workunit.client.0.smithi098.stderr:112500K .......... .......... .......... .......... .......... 82% 26.4M 1s 2024-04-25T16:34:08.967 INFO:tasks.workunit.client.0.smithi098.stderr:112550K .......... .......... .......... .......... .......... 82% 30.4M 1s 2024-04-25T16:34:08.969 INFO:tasks.workunit.client.0.smithi098.stderr:112600K .......... .......... .......... .......... .......... 82% 30.8M 1s 2024-04-25T16:34:08.970 INFO:tasks.workunit.client.0.smithi098.stderr:112650K .......... .......... .......... .......... .......... 83% 32.2M 1s 2024-04-25T16:34:08.972 INFO:tasks.workunit.client.0.smithi098.stderr:112700K .......... .......... .......... .......... .......... 83% 27.9M 1s 2024-04-25T16:34:08.973 INFO:tasks.workunit.client.0.smithi098.stderr:112750K .......... .......... .......... .......... .......... 83% 32.9M 1s 2024-04-25T16:34:08.975 INFO:tasks.workunit.client.0.smithi098.stderr:112800K .......... .......... .......... .......... .......... 83% 32.3M 1s 2024-04-25T16:34:08.976 INFO:tasks.workunit.client.0.smithi098.stderr:112850K .......... .......... .......... .......... .......... 83% 33.2M 1s 2024-04-25T16:34:08.978 INFO:tasks.workunit.client.0.smithi098.stderr:112900K .......... .......... .......... .......... .......... 83% 27.8M 1s 2024-04-25T16:34:08.980 INFO:tasks.workunit.client.0.smithi098.stderr:112950K .......... .......... .......... .......... .......... 83% 32.1M 1s 2024-04-25T16:34:08.981 INFO:tasks.workunit.client.0.smithi098.stderr:113000K .......... .......... .......... .......... .......... 83% 32.5M 1s 2024-04-25T16:34:08.983 INFO:tasks.workunit.client.0.smithi098.stderr:113050K .......... .......... .......... .......... .......... 83% 29.1M 1s 2024-04-25T16:34:08.985 INFO:tasks.workunit.client.0.smithi098.stderr:113100K .......... .......... .......... .......... .......... 83% 27.1M 1s 2024-04-25T16:34:08.986 INFO:tasks.workunit.client.0.smithi098.stderr:113150K .......... .......... .......... .......... .......... 83% 32.0M 1s 2024-04-25T16:34:08.988 INFO:tasks.workunit.client.0.smithi098.stderr:113200K .......... .......... .......... .......... .......... 83% 25.6M 1s 2024-04-25T16:34:08.990 INFO:tasks.workunit.client.0.smithi098.stderr:113250K .......... .......... .......... .......... .......... 83% 22.7M 1s 2024-04-25T16:34:08.992 INFO:tasks.workunit.client.0.smithi098.stderr:113300K .......... .......... .......... .......... .......... 83% 23.1M 1s 2024-04-25T16:34:08.994 INFO:tasks.workunit.client.0.smithi098.stderr:113350K .......... .......... .......... .......... .......... 83% 28.0M 1s 2024-04-25T16:34:08.996 INFO:tasks.workunit.client.0.smithi098.stderr:113400K .......... .......... .......... .......... .......... 83% 26.7M 1s 2024-04-25T16:34:08.998 INFO:tasks.workunit.client.0.smithi098.stderr:113450K .......... .......... .......... .......... .......... 83% 25.3M 1s 2024-04-25T16:34:09.000 INFO:tasks.workunit.client.0.smithi098.stderr:113500K .......... .......... .......... .......... .......... 83% 24.6M 1s 2024-04-25T16:34:09.002 INFO:tasks.workunit.client.0.smithi098.stderr:113550K .......... .......... .......... .......... .......... 83% 25.1M 1s 2024-04-25T16:34:09.003 INFO:tasks.workunit.client.0.smithi098.stderr:113600K .......... .......... .......... .......... .......... 83% 29.1M 1s 2024-04-25T16:34:09.005 INFO:tasks.workunit.client.0.smithi098.stderr:113650K .......... .......... .......... .......... .......... 83% 28.7M 1s 2024-04-25T16:34:09.007 INFO:tasks.workunit.client.0.smithi098.stderr:113700K .......... .......... .......... .......... .......... 83% 24.7M 1s 2024-04-25T16:34:09.009 INFO:tasks.workunit.client.0.smithi098.stderr:113750K .......... .......... .......... .......... .......... 83% 25.5M 1s 2024-04-25T16:34:09.011 INFO:tasks.workunit.client.0.smithi098.stderr:113800K .......... .......... .......... .......... .......... 83% 26.9M 1s 2024-04-25T16:34:09.013 INFO:tasks.workunit.client.0.smithi098.stderr:113850K .......... .......... .......... .......... .......... 83% 27.4M 1s 2024-04-25T16:34:09.014 INFO:tasks.workunit.client.0.smithi098.stderr:113900K .......... .......... .......... .......... .......... 83% 22.8M 1s 2024-04-25T16:34:09.016 INFO:tasks.workunit.client.0.smithi098.stderr:113950K .......... .......... .......... .......... .......... 83% 26.6M 1s 2024-04-25T16:34:09.018 INFO:tasks.workunit.client.0.smithi098.stderr:114000K .......... .......... .......... .......... .......... 84% 27.3M 1s 2024-04-25T16:34:09.020 INFO:tasks.workunit.client.0.smithi098.stderr:114050K .......... .......... .......... .......... .......... 84% 25.6M 1s 2024-04-25T16:34:09.023 INFO:tasks.workunit.client.0.smithi098.stderr:114100K .......... .......... .......... .......... .......... 84% 22.7M 1s 2024-04-25T16:34:09.024 INFO:tasks.workunit.client.0.smithi098.stderr:114150K .......... .......... .......... .......... .......... 84% 26.1M 1s 2024-04-25T16:34:09.026 INFO:tasks.workunit.client.0.smithi098.stderr:114200K .......... .......... .......... .......... .......... 84% 25.5M 1s 2024-04-25T16:34:09.028 INFO:tasks.workunit.client.0.smithi098.stderr:114250K .......... .......... .......... .......... .......... 84% 25.4M 1s 2024-04-25T16:34:09.031 INFO:tasks.workunit.client.0.smithi098.stderr:114300K .......... .......... .......... .......... .......... 84% 21.5M 1s 2024-04-25T16:34:09.032 INFO:tasks.workunit.client.0.smithi098.stderr:114350K .......... .......... .......... .......... .......... 84% 25.4M 1s 2024-04-25T16:34:09.034 INFO:tasks.workunit.client.0.smithi098.stderr:114400K .......... .......... .......... .......... .......... 84% 27.7M 1s 2024-04-25T16:34:09.036 INFO:tasks.workunit.client.0.smithi098.stderr:114450K .......... .......... .......... .......... .......... 84% 27.7M 1s 2024-04-25T16:34:09.038 INFO:tasks.workunit.client.0.smithi098.stderr:114500K .......... .......... .......... .......... .......... 84% 22.6M 1s 2024-04-25T16:34:09.040 INFO:tasks.workunit.client.0.smithi098.stderr:114550K .......... .......... .......... .......... .......... 84% 26.8M 1s 2024-04-25T16:34:09.042 INFO:tasks.workunit.client.0.smithi098.stderr:114600K .......... .......... .......... .......... .......... 84% 26.4M 1s 2024-04-25T16:34:09.044 INFO:tasks.workunit.client.0.smithi098.stderr:114650K .......... .......... .......... .......... .......... 84% 27.3M 1s 2024-04-25T16:34:09.045 INFO:tasks.workunit.client.0.smithi098.stderr:114700K .......... .......... .......... .......... .......... 84% 27.1M 1s 2024-04-25T16:34:09.047 INFO:tasks.workunit.client.0.smithi098.stderr:114750K .......... .......... .......... .......... .......... 84% 33.9M 1s 2024-04-25T16:34:09.048 INFO:tasks.workunit.client.0.smithi098.stderr:114800K .......... .......... .......... .......... .......... 84% 34.2M 1s 2024-04-25T16:34:09.049 INFO:tasks.workunit.client.0.smithi098.stderr:114850K .......... .......... .......... .......... .......... 84% 36.3M 1s 2024-04-25T16:34:09.051 INFO:tasks.workunit.client.0.smithi098.stderr:114900K .......... .......... .......... .......... .......... 84% 28.7M 1s 2024-04-25T16:34:09.053 INFO:tasks.workunit.client.0.smithi098.stderr:114950K .......... .......... .......... .......... .......... 84% 28.8M 1s 2024-04-25T16:34:09.055 INFO:tasks.workunit.client.0.smithi098.stderr:115000K .......... .......... .......... .......... .......... 84% 28.1M 1s 2024-04-25T16:34:09.057 INFO:tasks.workunit.client.0.smithi098.stderr:115050K .......... .......... .......... .......... .......... 84% 24.7M 1s 2024-04-25T16:34:09.058 INFO:tasks.workunit.client.0.smithi098.stderr:115100K .......... .......... .......... .......... .......... 84% 27.3M 1s 2024-04-25T16:34:09.060 INFO:tasks.workunit.client.0.smithi098.stderr:115150K .......... .......... .......... .......... .......... 84% 25.2M 1s 2024-04-25T16:34:09.062 INFO:tasks.workunit.client.0.smithi098.stderr:115200K .......... .......... .......... .......... .......... 84% 27.1M 1s 2024-04-25T16:34:09.064 INFO:tasks.workunit.client.0.smithi098.stderr:115250K .......... .......... .......... .......... .......... 84% 32.3M 1s 2024-04-25T16:34:09.065 INFO:tasks.workunit.client.0.smithi098.stderr:115300K .......... .......... .......... .......... .......... 84% 28.8M 1s 2024-04-25T16:34:09.067 INFO:tasks.workunit.client.0.smithi098.stderr:115350K .......... .......... .......... .......... .......... 85% 34.3M 1s 2024-04-25T16:34:09.068 INFO:tasks.workunit.client.0.smithi098.stderr:115400K .......... .......... .......... .......... .......... 85% 35.1M 1s 2024-04-25T16:34:09.070 INFO:tasks.workunit.client.0.smithi098.stderr:115450K .......... .......... .......... .......... .......... 85% 34.0M 1s 2024-04-25T16:34:09.071 INFO:tasks.workunit.client.0.smithi098.stderr:115500K .......... .......... .......... .......... .......... 85% 32.1M 1s 2024-04-25T16:34:09.072 INFO:tasks.workunit.client.0.smithi098.stderr:115550K .......... .......... .......... .......... .......... 85% 37.0M 1s 2024-04-25T16:34:09.074 INFO:tasks.workunit.client.0.smithi098.stderr:115600K .......... .......... .......... .......... .......... 85% 36.8M 1s 2024-04-25T16:34:09.075 INFO:tasks.workunit.client.0.smithi098.stderr:115650K .......... .......... .......... .......... .......... 85% 37.2M 1s 2024-04-25T16:34:09.077 INFO:tasks.workunit.client.0.smithi098.stderr:115700K .......... .......... .......... .......... .......... 85% 31.7M 1s 2024-04-25T16:34:09.078 INFO:tasks.workunit.client.0.smithi098.stderr:115750K .......... .......... .......... .......... .......... 85% 36.6M 1s 2024-04-25T16:34:09.079 INFO:tasks.workunit.client.0.smithi098.stderr:115800K .......... .......... .......... .......... .......... 85% 36.1M 1s 2024-04-25T16:34:09.081 INFO:tasks.workunit.client.0.smithi098.stderr:115850K .......... .......... .......... .......... .......... 85% 34.9M 1s 2024-04-25T16:34:09.082 INFO:tasks.workunit.client.0.smithi098.stderr:115900K .......... .......... .......... .......... .......... 85% 30.7M 1s 2024-04-25T16:34:09.084 INFO:tasks.workunit.client.0.smithi098.stderr:115950K .......... .......... .......... .......... .......... 85% 35.1M 1s 2024-04-25T16:34:09.085 INFO:tasks.workunit.client.0.smithi098.stderr:116000K .......... .......... .......... .......... .......... 85% 35.8M 1s 2024-04-25T16:34:09.086 INFO:tasks.workunit.client.0.smithi098.stderr:116050K .......... .......... .......... .......... .......... 85% 35.7M 1s 2024-04-25T16:34:09.088 INFO:tasks.workunit.client.0.smithi098.stderr:116100K .......... .......... .......... .......... .......... 85% 29.5M 1s 2024-04-25T16:34:09.089 INFO:tasks.workunit.client.0.smithi098.stderr:116150K .......... .......... .......... .......... .......... 85% 35.2M 1s 2024-04-25T16:34:09.091 INFO:tasks.workunit.client.0.smithi098.stderr:116200K .......... .......... .......... .......... .......... 85% 35.3M 1s 2024-04-25T16:34:09.092 INFO:tasks.workunit.client.0.smithi098.stderr:116250K .......... .......... .......... .......... .......... 85% 35.1M 1s 2024-04-25T16:34:09.094 INFO:tasks.workunit.client.0.smithi098.stderr:116300K .......... .......... .......... .......... .......... 85% 29.2M 1s 2024-04-25T16:34:09.095 INFO:tasks.workunit.client.0.smithi098.stderr:116350K .......... .......... .......... .......... .......... 85% 33.8M 1s 2024-04-25T16:34:09.097 INFO:tasks.workunit.client.0.smithi098.stderr:116400K .......... .......... .......... .......... .......... 85% 34.1M 1s 2024-04-25T16:34:09.098 INFO:tasks.workunit.client.0.smithi098.stderr:116450K .......... .......... .......... .......... .......... 85% 34.2M 1s 2024-04-25T16:34:09.100 INFO:tasks.workunit.client.0.smithi098.stderr:116500K .......... .......... .......... .......... .......... 85% 28.6M 1s 2024-04-25T16:34:09.102 INFO:tasks.workunit.client.0.smithi098.stderr:116550K .......... .......... .......... .......... .......... 85% 33.1M 1s 2024-04-25T16:34:09.103 INFO:tasks.workunit.client.0.smithi098.stderr:116600K .......... .......... .......... .......... .......... 85% 33.1M 1s 2024-04-25T16:34:09.104 INFO:tasks.workunit.client.0.smithi098.stderr:116650K .......... .......... .......... .......... .......... 85% 33.3M 1s 2024-04-25T16:34:09.106 INFO:tasks.workunit.client.0.smithi098.stderr:116700K .......... .......... .......... .......... .......... 86% 28.7M 1s 2024-04-25T16:34:09.107 INFO:tasks.workunit.client.0.smithi098.stderr:116750K .......... .......... .......... .......... .......... 86% 32.5M 1s 2024-04-25T16:34:09.109 INFO:tasks.workunit.client.0.smithi098.stderr:116800K .......... .......... .......... .......... .......... 86% 31.8M 1s 2024-04-25T16:34:09.111 INFO:tasks.workunit.client.0.smithi098.stderr:116850K .......... .......... .......... .......... .......... 86% 32.3M 1s 2024-04-25T16:34:09.112 INFO:tasks.workunit.client.0.smithi098.stderr:116900K .......... .......... .......... .......... .......... 86% 27.7M 1s 2024-04-25T16:34:09.114 INFO:tasks.workunit.client.0.smithi098.stderr:116950K .......... .......... .......... .......... .......... 86% 31.1M 1s 2024-04-25T16:34:09.115 INFO:tasks.workunit.client.0.smithi098.stderr:117000K .......... .......... .......... .......... .......... 86% 32.0M 1s 2024-04-25T16:34:09.117 INFO:tasks.workunit.client.0.smithi098.stderr:117050K .......... .......... .......... .......... .......... 86% 31.7M 1s 2024-04-25T16:34:09.119 INFO:tasks.workunit.client.0.smithi098.stderr:117100K .......... .......... .......... .......... .......... 86% 27.2M 1s 2024-04-25T16:34:09.120 INFO:tasks.workunit.client.0.smithi098.stderr:117150K .......... .......... .......... .......... .......... 86% 31.2M 1s 2024-04-25T16:34:09.122 INFO:tasks.workunit.client.0.smithi098.stderr:117200K .......... .......... .......... .......... .......... 86% 29.3M 1s 2024-04-25T16:34:09.124 INFO:tasks.workunit.client.0.smithi098.stderr:117250K .......... .......... .......... .......... .......... 86% 28.8M 1s 2024-04-25T16:34:09.125 INFO:tasks.workunit.client.0.smithi098.stderr:117300K .......... .......... .......... .......... .......... 86% 27.4M 1s 2024-04-25T16:34:09.127 INFO:tasks.workunit.client.0.smithi098.stderr:117350K .......... .......... .......... .......... .......... 86% 31.6M 1s 2024-04-25T16:34:09.128 INFO:tasks.workunit.client.0.smithi098.stderr:117400K .......... .......... .......... .......... .......... 86% 32.1M 1s 2024-04-25T16:34:09.130 INFO:tasks.workunit.client.0.smithi098.stderr:117450K .......... .......... .......... .......... .......... 86% 31.0M 1s 2024-04-25T16:34:09.132 INFO:tasks.workunit.client.0.smithi098.stderr:117500K .......... .......... .......... .......... .......... 86% 25.9M 1s 2024-04-25T16:34:09.134 INFO:tasks.workunit.client.0.smithi098.stderr:117550K .......... .......... .......... .......... .......... 86% 30.6M 1s 2024-04-25T16:34:09.135 INFO:tasks.workunit.client.0.smithi098.stderr:117600K .......... .......... .......... .......... .......... 86% 30.3M 1s 2024-04-25T16:34:09.137 INFO:tasks.workunit.client.0.smithi098.stderr:117650K .......... .......... .......... .......... .......... 86% 29.6M 1s 2024-04-25T16:34:09.139 INFO:tasks.workunit.client.0.smithi098.stderr:117700K .......... .......... .......... .......... .......... 86% 25.5M 1s 2024-04-25T16:34:09.140 INFO:tasks.workunit.client.0.smithi098.stderr:117750K .......... .......... .......... .......... .......... 86% 28.8M 1s 2024-04-25T16:34:09.142 INFO:tasks.workunit.client.0.smithi098.stderr:117800K .......... .......... .......... .......... .......... 86% 29.2M 1s 2024-04-25T16:34:09.144 INFO:tasks.workunit.client.0.smithi098.stderr:117850K .......... .......... .......... .......... .......... 86% 29.1M 1s 2024-04-25T16:34:09.146 INFO:tasks.workunit.client.0.smithi098.stderr:117900K .......... .......... .......... .......... .......... 86% 23.2M 1s 2024-04-25T16:34:09.148 INFO:tasks.workunit.client.0.smithi098.stderr:117950K .......... .......... .......... .......... .......... 86% 26.3M 1s 2024-04-25T16:34:09.149 INFO:tasks.workunit.client.0.smithi098.stderr:118000K .......... .......... .......... .......... .......... 86% 26.0M 1s 2024-04-25T16:34:09.152 INFO:tasks.workunit.client.0.smithi098.stderr:118050K .......... .......... .......... .......... .......... 86% 26.0M 1s 2024-04-25T16:34:09.154 INFO:tasks.workunit.client.0.smithi098.stderr:118100K .......... .......... .......... .......... .......... 87% 22.0M 1s 2024-04-25T16:34:09.155 INFO:tasks.workunit.client.0.smithi098.stderr:118150K .......... .......... .......... .......... .......... 87% 25.7M 1s 2024-04-25T16:34:09.157 INFO:tasks.workunit.client.0.smithi098.stderr:118200K .......... .......... .......... .......... .......... 87% 25.5M 1s 2024-04-25T16:34:09.160 INFO:tasks.workunit.client.0.smithi098.stderr:118250K .......... .......... .......... .......... .......... 87% 25.5M 1s 2024-04-25T16:34:09.162 INFO:tasks.workunit.client.0.smithi098.stderr:118300K .......... .......... .......... .......... .......... 87% 23.3M 1s 2024-04-25T16:34:09.163 INFO:tasks.workunit.client.0.smithi098.stderr:118350K .......... .......... .......... .......... .......... 87% 27.5M 1s 2024-04-25T16:34:09.165 INFO:tasks.workunit.client.0.smithi098.stderr:118400K .......... .......... .......... .......... .......... 87% 27.7M 1s 2024-04-25T16:34:09.167 INFO:tasks.workunit.client.0.smithi098.stderr:118450K .......... .......... .......... .......... .......... 87% 26.8M 1s 2024-04-25T16:34:09.169 INFO:tasks.workunit.client.0.smithi098.stderr:118500K .......... .......... .......... .......... .......... 87% 23.7M 1s 2024-04-25T16:34:09.171 INFO:tasks.workunit.client.0.smithi098.stderr:118550K .......... .......... .......... .......... .......... 87% 27.2M 1s 2024-04-25T16:34:09.172 INFO:tasks.workunit.client.0.smithi098.stderr:118600K .......... .......... .......... .......... .......... 87% 27.2M 1s 2024-04-25T16:34:09.175 INFO:tasks.workunit.client.0.smithi098.stderr:118650K .......... .......... .......... .......... .......... 87% 26.6M 1s 2024-04-25T16:34:09.176 INFO:tasks.workunit.client.0.smithi098.stderr:118700K .......... .......... .......... .......... .......... 87% 23.2M 1s 2024-04-25T16:34:09.179 INFO:tasks.workunit.client.0.smithi098.stderr:118750K .......... .......... .......... .......... .......... 87% 24.9M 1s 2024-04-25T16:34:09.180 INFO:tasks.workunit.client.0.smithi098.stderr:118800K .......... .......... .......... .......... .......... 87% 37.2M 1s 2024-04-25T16:34:09.181 INFO:tasks.workunit.client.0.smithi098.stderr:118850K .......... .......... .......... .......... .......... 87% 38.8M 1s 2024-04-25T16:34:09.183 INFO:tasks.workunit.client.0.smithi098.stderr:118900K .......... .......... .......... .......... .......... 87% 32.3M 1s 2024-04-25T16:34:09.184 INFO:tasks.workunit.client.0.smithi098.stderr:118950K .......... .......... .......... .......... .......... 87% 39.5M 1s 2024-04-25T16:34:09.185 INFO:tasks.workunit.client.0.smithi098.stderr:119000K .......... .......... .......... .......... .......... 87% 38.2M 1s 2024-04-25T16:34:09.186 INFO:tasks.workunit.client.0.smithi098.stderr:119050K .......... .......... .......... .......... .......... 87% 38.0M 1s 2024-04-25T16:34:09.188 INFO:tasks.workunit.client.0.smithi098.stderr:119100K .......... .......... .......... .......... .......... 87% 27.3M 1s 2024-04-25T16:34:09.189 INFO:tasks.workunit.client.0.smithi098.stderr:119150K .......... .......... .......... .......... .......... 87% 35.6M 1s 2024-04-25T16:34:09.191 INFO:tasks.workunit.client.0.smithi098.stderr:119200K .......... .......... .......... .......... .......... 87% 30.7M 1s 2024-04-25T16:34:09.193 INFO:tasks.workunit.client.0.smithi098.stderr:119250K .......... .......... .......... .......... .......... 87% 30.6M 1s 2024-04-25T16:34:09.195 INFO:tasks.workunit.client.0.smithi098.stderr:119300K .......... .......... .......... .......... .......... 87% 24.1M 1s 2024-04-25T16:34:09.196 INFO:tasks.workunit.client.0.smithi098.stderr:119350K .......... .......... .......... .......... .......... 87% 33.7M 1s 2024-04-25T16:34:09.198 INFO:tasks.workunit.client.0.smithi098.stderr:119400K .......... .......... .......... .......... .......... 87% 34.4M 1s 2024-04-25T16:34:09.199 INFO:tasks.workunit.client.0.smithi098.stderr:119450K .......... .......... .......... .......... .......... 88% 33.2M 1s 2024-04-25T16:34:09.201 INFO:tasks.workunit.client.0.smithi098.stderr:119500K .......... .......... .......... .......... .......... 88% 29.6M 1s 2024-04-25T16:34:09.202 INFO:tasks.workunit.client.0.smithi098.stderr:119550K .......... .......... .......... .......... .......... 88% 37.8M 1s 2024-04-25T16:34:09.203 INFO:tasks.workunit.client.0.smithi098.stderr:119600K .......... .......... .......... .......... .......... 88% 37.0M 1s 2024-04-25T16:34:09.205 INFO:tasks.workunit.client.0.smithi098.stderr:119650K .......... .......... .......... .......... .......... 88% 37.1M 1s 2024-04-25T16:34:09.206 INFO:tasks.workunit.client.0.smithi098.stderr:119700K .......... .......... .......... .......... .......... 88% 32.1M 1s 2024-04-25T16:34:09.208 INFO:tasks.workunit.client.0.smithi098.stderr:119750K .......... .......... .......... .......... .......... 88% 33.3M 1s 2024-04-25T16:34:09.209 INFO:tasks.workunit.client.0.smithi098.stderr:119800K .......... .......... .......... .......... .......... 88% 35.2M 1s 2024-04-25T16:34:09.211 INFO:tasks.workunit.client.0.smithi098.stderr:119850K .......... .......... .......... .......... .......... 88% 30.8M 1s 2024-04-25T16:34:09.212 INFO:tasks.workunit.client.0.smithi098.stderr:119900K .......... .......... .......... .......... .......... 88% 26.7M 1s 2024-04-25T16:34:09.214 INFO:tasks.workunit.client.0.smithi098.stderr:119950K .......... .......... .......... .......... .......... 88% 30.6M 1s 2024-04-25T16:34:09.216 INFO:tasks.workunit.client.0.smithi098.stderr:120000K .......... .......... .......... .......... .......... 88% 31.2M 1s 2024-04-25T16:34:09.217 INFO:tasks.workunit.client.0.smithi098.stderr:120050K .......... .......... .......... .......... .......... 88% 30.9M 1s 2024-04-25T16:34:09.219 INFO:tasks.workunit.client.0.smithi098.stderr:120100K .......... .......... .......... .......... .......... 88% 26.4M 1s 2024-04-25T16:34:09.220 INFO:tasks.workunit.client.0.smithi098.stderr:120150K .......... .......... .......... .......... .......... 88% 33.6M 1s 2024-04-25T16:34:09.222 INFO:tasks.workunit.client.0.smithi098.stderr:120200K .......... .......... .......... .......... .......... 88% 32.7M 1s 2024-04-25T16:34:09.223 INFO:tasks.workunit.client.0.smithi098.stderr:120250K .......... .......... .......... .......... .......... 88% 33.4M 1s 2024-04-25T16:34:09.225 INFO:tasks.workunit.client.0.smithi098.stderr:120300K .......... .......... .......... .......... .......... 88% 28.0M 1s 2024-04-25T16:34:09.227 INFO:tasks.workunit.client.0.smithi098.stderr:120350K .......... .......... .......... .......... .......... 88% 32.4M 1s 2024-04-25T16:34:09.228 INFO:tasks.workunit.client.0.smithi098.stderr:120400K .......... .......... .......... .......... .......... 88% 32.5M 1s 2024-04-25T16:34:09.230 INFO:tasks.workunit.client.0.smithi098.stderr:120450K .......... .......... .......... .......... .......... 88% 31.7M 1s 2024-04-25T16:34:09.232 INFO:tasks.workunit.client.0.smithi098.stderr:120500K .......... .......... .......... .......... .......... 88% 24.7M 1s 2024-04-25T16:34:09.233 INFO:tasks.workunit.client.0.smithi098.stderr:120550K .......... .......... .......... .......... .......... 88% 32.4M 1s 2024-04-25T16:34:09.235 INFO:tasks.workunit.client.0.smithi098.stderr:120600K .......... .......... .......... .......... .......... 88% 32.2M 1s 2024-04-25T16:34:09.236 INFO:tasks.workunit.client.0.smithi098.stderr:120650K .......... .......... .......... .......... .......... 88% 32.4M 1s 2024-04-25T16:34:09.238 INFO:tasks.workunit.client.0.smithi098.stderr:120700K .......... .......... .......... .......... .......... 88% 27.1M 1s 2024-04-25T16:34:09.239 INFO:tasks.workunit.client.0.smithi098.stderr:120750K .......... .......... .......... .......... .......... 88% 32.6M 1s 2024-04-25T16:34:09.241 INFO:tasks.workunit.client.0.smithi098.stderr:120800K .......... .......... .......... .......... .......... 89% 30.9M 1s 2024-04-25T16:34:09.243 INFO:tasks.workunit.client.0.smithi098.stderr:120850K .......... .......... .......... .......... .......... 89% 31.5M 1s 2024-04-25T16:34:09.245 INFO:tasks.workunit.client.0.smithi098.stderr:120900K .......... .......... .......... .......... .......... 89% 25.3M 1s 2024-04-25T16:34:09.246 INFO:tasks.workunit.client.0.smithi098.stderr:120950K .......... .......... .......... .......... .......... 89% 30.7M 1s 2024-04-25T16:34:09.247 INFO:tasks.workunit.client.0.smithi098.stderr:121000K .......... .......... .......... .......... .......... 89% 31.3M 1s 2024-04-25T16:34:09.249 INFO:tasks.workunit.client.0.smithi098.stderr:121050K .......... .......... .......... .......... .......... 89% 31.0M 1s 2024-04-25T16:34:09.251 INFO:tasks.workunit.client.0.smithi098.stderr:121100K .......... .......... .......... .......... .......... 89% 26.2M 1s 2024-04-25T16:34:09.253 INFO:tasks.workunit.client.0.smithi098.stderr:121150K .......... .......... .......... .......... .......... 89% 30.3M 1s 2024-04-25T16:34:09.254 INFO:tasks.workunit.client.0.smithi098.stderr:121200K .......... .......... .......... .......... .......... 89% 30.7M 1s 2024-04-25T16:34:09.256 INFO:tasks.workunit.client.0.smithi098.stderr:121250K .......... .......... .......... .......... .......... 89% 29.6M 1s 2024-04-25T16:34:09.258 INFO:tasks.workunit.client.0.smithi098.stderr:121300K .......... .......... .......... .......... .......... 89% 26.1M 1s 2024-04-25T16:34:09.259 INFO:tasks.workunit.client.0.smithi098.stderr:121350K .......... .......... .......... .......... .......... 89% 29.8M 1s 2024-04-25T16:34:09.261 INFO:tasks.workunit.client.0.smithi098.stderr:121400K .......... .......... .......... .......... .......... 89% 28.5M 0s 2024-04-25T16:34:09.263 INFO:tasks.workunit.client.0.smithi098.stderr:121450K .......... .......... .......... .......... .......... 89% 28.7M 0s 2024-04-25T16:34:09.265 INFO:tasks.workunit.client.0.smithi098.stderr:121500K .......... .......... .......... .......... .......... 89% 24.6M 0s 2024-04-25T16:34:09.267 INFO:tasks.workunit.client.0.smithi098.stderr:121550K .......... .......... .......... .......... .......... 89% 26.9M 0s 2024-04-25T16:34:09.268 INFO:tasks.workunit.client.0.smithi098.stderr:121600K .......... .......... .......... .......... .......... 89% 27.6M 0s 2024-04-25T16:34:09.270 INFO:tasks.workunit.client.0.smithi098.stderr:121650K .......... .......... .......... .......... .......... 89% 27.4M 0s 2024-04-25T16:34:09.272 INFO:tasks.workunit.client.0.smithi098.stderr:121700K .......... .......... .......... .......... .......... 89% 23.8M 0s 2024-04-25T16:34:09.274 INFO:tasks.workunit.client.0.smithi098.stderr:121750K .......... .......... .......... .......... .......... 89% 28.0M 0s 2024-04-25T16:34:09.276 INFO:tasks.workunit.client.0.smithi098.stderr:121800K .......... .......... .......... .......... .......... 89% 26.6M 0s 2024-04-25T16:34:09.278 INFO:tasks.workunit.client.0.smithi098.stderr:121850K .......... .......... .......... .......... .......... 89% 27.1M 0s 2024-04-25T16:34:09.280 INFO:tasks.workunit.client.0.smithi098.stderr:121900K .......... .......... .......... .......... .......... 89% 22.9M 0s 2024-04-25T16:34:09.282 INFO:tasks.workunit.client.0.smithi098.stderr:121950K .......... .......... .......... .......... .......... 89% 27.4M 0s 2024-04-25T16:34:09.283 INFO:tasks.workunit.client.0.smithi098.stderr:122000K .......... .......... .......... .......... .......... 89% 27.0M 0s 2024-04-25T16:34:09.285 INFO:tasks.workunit.client.0.smithi098.stderr:122050K .......... .......... .......... .......... .......... 89% 28.3M 0s 2024-04-25T16:34:09.287 INFO:tasks.workunit.client.0.smithi098.stderr:122100K .......... .......... .......... .......... .......... 89% 23.2M 0s 2024-04-25T16:34:09.289 INFO:tasks.workunit.client.0.smithi098.stderr:122150K .......... .......... .......... .......... .......... 90% 26.8M 0s 2024-04-25T16:34:09.291 INFO:tasks.workunit.client.0.smithi098.stderr:122200K .......... .......... .......... .......... .......... 90% 27.0M 0s 2024-04-25T16:34:09.293 INFO:tasks.workunit.client.0.smithi098.stderr:122250K .......... .......... .......... .......... .......... 90% 27.8M 0s 2024-04-25T16:34:09.295 INFO:tasks.workunit.client.0.smithi098.stderr:122300K .......... .......... .......... .......... .......... 90% 23.6M 0s 2024-04-25T16:34:09.296 INFO:tasks.workunit.client.0.smithi098.stderr:122350K .......... .......... .......... .......... .......... 90% 27.8M 0s 2024-04-25T16:34:09.298 INFO:tasks.workunit.client.0.smithi098.stderr:122400K .......... .......... .......... .......... .......... 90% 27.7M 0s 2024-04-25T16:34:09.300 INFO:tasks.workunit.client.0.smithi098.stderr:122450K .......... .......... .......... .......... .......... 90% 27.1M 0s 2024-04-25T16:34:09.302 INFO:tasks.workunit.client.0.smithi098.stderr:122500K .......... .......... .......... .......... .......... 90% 23.0M 0s 2024-04-25T16:34:09.304 INFO:tasks.workunit.client.0.smithi098.stderr:122550K .......... .......... .......... .......... .......... 90% 27.5M 0s 2024-04-25T16:34:09.306 INFO:tasks.workunit.client.0.smithi098.stderr:122600K .......... .......... .......... .......... .......... 90% 27.5M 0s 2024-04-25T16:34:09.308 INFO:tasks.workunit.client.0.smithi098.stderr:122650K .......... .......... .......... .......... .......... 90% 27.7M 0s 2024-04-25T16:34:09.310 INFO:tasks.workunit.client.0.smithi098.stderr:122700K .......... .......... .......... .......... .......... 90% 22.2M 0s 2024-04-25T16:34:09.312 INFO:tasks.workunit.client.0.smithi098.stderr:122750K .......... .......... .......... .......... .......... 90% 25.6M 0s 2024-04-25T16:34:09.313 INFO:tasks.workunit.client.0.smithi098.stderr:122800K .......... .......... .......... .......... .......... 90% 25.4M 0s 2024-04-25T16:34:09.316 INFO:tasks.workunit.client.0.smithi098.stderr:122850K .......... .......... .......... .......... .......... 90% 23.9M 0s 2024-04-25T16:34:09.317 INFO:tasks.workunit.client.0.smithi098.stderr:122900K .......... .......... .......... .......... .......... 90% 26.8M 0s 2024-04-25T16:34:09.319 INFO:tasks.workunit.client.0.smithi098.stderr:122950K .......... .......... .......... .......... .......... 90% 30.2M 0s 2024-04-25T16:34:09.321 INFO:tasks.workunit.client.0.smithi098.stderr:123000K .......... .......... .......... .......... .......... 90% 25.5M 0s 2024-04-25T16:34:09.323 INFO:tasks.workunit.client.0.smithi098.stderr:123050K .......... .......... .......... .......... .......... 90% 30.8M 0s 2024-04-25T16:34:09.326 INFO:tasks.workunit.client.0.smithi098.stderr:123100K .......... .......... .......... .......... .......... 90% 19.9M 0s 2024-04-25T16:34:09.327 INFO:tasks.workunit.client.0.smithi098.stderr:123150K .......... .......... .......... .......... .......... 90% 29.9M 0s 2024-04-25T16:34:09.328 INFO:tasks.workunit.client.0.smithi098.stderr:123200K .......... .......... .......... .......... .......... 90% 34.7M 0s 2024-04-25T16:34:09.329 INFO:tasks.workunit.client.0.smithi098.stderr:123250K .......... .......... .......... .......... .......... 90% 37.3M 0s 2024-04-25T16:34:09.331 INFO:tasks.workunit.client.0.smithi098.stderr:123300K .......... .......... .......... .......... .......... 90% 31.9M 0s 2024-04-25T16:34:09.332 INFO:tasks.workunit.client.0.smithi098.stderr:123350K .......... .......... .......... .......... .......... 90% 36.1M 0s 2024-04-25T16:34:09.333 INFO:tasks.workunit.client.0.smithi098.stderr:123400K .......... .......... .......... .......... .......... 90% 39.5M 0s 2024-04-25T16:34:09.335 INFO:tasks.workunit.client.0.smithi098.stderr:123450K .......... .......... .......... .......... .......... 90% 35.3M 0s 2024-04-25T16:34:09.337 INFO:tasks.workunit.client.0.smithi098.stderr:123500K .......... .......... .......... .......... .......... 91% 26.2M 0s 2024-04-25T16:34:09.338 INFO:tasks.workunit.client.0.smithi098.stderr:123550K .......... .......... .......... .......... .......... 91% 34.9M 0s 2024-04-25T16:34:09.343 INFO:tasks.workunit.client.0.smithi098.stderr:123600K .......... .......... .......... .......... .......... 91% 35.6M 0s 2024-04-25T16:34:09.344 INFO:tasks.workunit.client.0.smithi098.stderr:123650K .......... .......... .......... .......... .......... 91% 34.8M 0s 2024-04-25T16:34:09.344 INFO:tasks.workunit.client.0.smithi098.stderr:123700K .......... .......... .......... .......... .......... 91% 29.5M 0s 2024-04-25T16:34:09.344 INFO:tasks.workunit.client.0.smithi098.stderr:123750K .......... .......... .......... .......... .......... 91% 34.2M 0s 2024-04-25T16:34:09.345 INFO:tasks.workunit.client.0.smithi098.stderr:123800K .......... .......... .......... .......... .......... 91% 34.1M 0s 2024-04-25T16:34:09.347 INFO:tasks.workunit.client.0.smithi098.stderr:123850K .......... .......... .......... .......... .......... 91% 33.5M 0s 2024-04-25T16:34:09.348 INFO:tasks.workunit.client.0.smithi098.stderr:123900K .......... .......... .......... .......... .......... 91% 29.3M 0s 2024-04-25T16:34:09.350 INFO:tasks.workunit.client.0.smithi098.stderr:123950K .......... .......... .......... .......... .......... 91% 33.0M 0s 2024-04-25T16:34:09.351 INFO:tasks.workunit.client.0.smithi098.stderr:124000K .......... .......... .......... .......... .......... 91% 33.5M 0s 2024-04-25T16:34:09.353 INFO:tasks.workunit.client.0.smithi098.stderr:124050K .......... .......... .......... .......... .......... 91% 33.1M 0s 2024-04-25T16:34:09.354 INFO:tasks.workunit.client.0.smithi098.stderr:124100K .......... .......... .......... .......... .......... 91% 28.7M 0s 2024-04-25T16:34:09.356 INFO:tasks.workunit.client.0.smithi098.stderr:124150K .......... .......... .......... .......... .......... 91% 32.0M 0s 2024-04-25T16:34:09.358 INFO:tasks.workunit.client.0.smithi098.stderr:124200K .......... .......... .......... .......... .......... 91% 32.8M 0s 2024-04-25T16:34:09.359 INFO:tasks.workunit.client.0.smithi098.stderr:124250K .......... .......... .......... .......... .......... 91% 33.0M 0s 2024-04-25T16:34:09.361 INFO:tasks.workunit.client.0.smithi098.stderr:124300K .......... .......... .......... .......... .......... 91% 28.3M 0s 2024-04-25T16:34:09.362 INFO:tasks.workunit.client.0.smithi098.stderr:124350K .......... .......... .......... .......... .......... 91% 32.8M 0s 2024-04-25T16:34:09.364 INFO:tasks.workunit.client.0.smithi098.stderr:124400K .......... .......... .......... .......... .......... 91% 31.9M 0s 2024-04-25T16:34:09.366 INFO:tasks.workunit.client.0.smithi098.stderr:124450K .......... .......... .......... .......... .......... 91% 28.6M 0s 2024-04-25T16:34:09.367 INFO:tasks.workunit.client.0.smithi098.stderr:124500K .......... .......... .......... .......... .......... 91% 27.2M 0s 2024-04-25T16:34:09.369 INFO:tasks.workunit.client.0.smithi098.stderr:124550K .......... .......... .......... .......... .......... 91% 32.5M 0s 2024-04-25T16:34:09.370 INFO:tasks.workunit.client.0.smithi098.stderr:124600K .......... .......... .......... .......... .......... 91% 32.4M 0s 2024-04-25T16:34:09.372 INFO:tasks.workunit.client.0.smithi098.stderr:124650K .......... .......... .......... .......... .......... 91% 30.6M 0s 2024-04-25T16:34:09.374 INFO:tasks.workunit.client.0.smithi098.stderr:124700K .......... .......... .......... .......... .......... 91% 26.5M 0s 2024-04-25T16:34:09.376 INFO:tasks.workunit.client.0.smithi098.stderr:124750K .......... .......... .......... .......... .......... 91% 26.4M 0s 2024-04-25T16:34:09.377 INFO:tasks.workunit.client.0.smithi098.stderr:124800K .......... .......... .......... .......... .......... 91% 26.3M 0s 2024-04-25T16:34:09.379 INFO:tasks.workunit.client.0.smithi098.stderr:124850K .......... .......... .......... .......... .......... 92% 27.2M 0s 2024-04-25T16:34:09.381 INFO:tasks.workunit.client.0.smithi098.stderr:124900K .......... .......... .......... .......... .......... 92% 24.3M 0s 2024-04-25T16:34:09.383 INFO:tasks.workunit.client.0.smithi098.stderr:124950K .......... .......... .......... .......... .......... 92% 26.2M 0s 2024-04-25T16:34:09.385 INFO:tasks.workunit.client.0.smithi098.stderr:125000K .......... .......... .......... .......... .......... 92% 25.9M 0s 2024-04-25T16:34:09.387 INFO:tasks.workunit.client.0.smithi098.stderr:125050K .......... .......... .......... .......... .......... 92% 29.1M 0s 2024-04-25T16:34:09.389 INFO:tasks.workunit.client.0.smithi098.stderr:125100K .......... .......... .......... .......... .......... 92% 22.0M 0s 2024-04-25T16:34:09.391 INFO:tasks.workunit.client.0.smithi098.stderr:125150K .......... .......... .......... .......... .......... 92% 27.6M 0s 2024-04-25T16:34:09.392 INFO:tasks.workunit.client.0.smithi098.stderr:125200K .......... .......... .......... .......... .......... 92% 29.1M 0s 2024-04-25T16:34:09.394 INFO:tasks.workunit.client.0.smithi098.stderr:125250K .......... .......... .......... .......... .......... 92% 27.6M 0s 2024-04-25T16:34:09.396 INFO:tasks.workunit.client.0.smithi098.stderr:125300K .......... .......... .......... .......... .......... 92% 25.0M 0s 2024-04-25T16:34:09.398 INFO:tasks.workunit.client.0.smithi098.stderr:125350K .......... .......... .......... .......... .......... 92% 27.6M 0s 2024-04-25T16:34:09.399 INFO:tasks.workunit.client.0.smithi098.stderr:125400K .......... .......... .......... .......... .......... 92% 31.1M 0s 2024-04-25T16:34:09.401 INFO:tasks.workunit.client.0.smithi098.stderr:125450K .......... .......... .......... .......... .......... 92% 31.2M 0s 2024-04-25T16:34:09.403 INFO:tasks.workunit.client.0.smithi098.stderr:125500K .......... .......... .......... .......... .......... 92% 25.6M 0s 2024-04-25T16:34:09.405 INFO:tasks.workunit.client.0.smithi098.stderr:125550K .......... .......... .......... .......... .......... 92% 30.8M 0s 2024-04-25T16:34:09.406 INFO:tasks.workunit.client.0.smithi098.stderr:125600K .......... .......... .......... .......... .......... 92% 31.1M 0s 2024-04-25T16:34:09.408 INFO:tasks.workunit.client.0.smithi098.stderr:125650K .......... .......... .......... .......... .......... 92% 30.1M 0s 2024-04-25T16:34:09.410 INFO:tasks.workunit.client.0.smithi098.stderr:125700K .......... .......... .......... .......... .......... 92% 26.9M 0s 2024-04-25T16:34:09.411 INFO:tasks.workunit.client.0.smithi098.stderr:125750K .......... .......... .......... .......... .......... 92% 31.5M 0s 2024-04-25T16:34:09.413 INFO:tasks.workunit.client.0.smithi098.stderr:125800K .......... .......... .......... .......... .......... 92% 31.7M 0s 2024-04-25T16:34:09.414 INFO:tasks.workunit.client.0.smithi098.stderr:125850K .......... .......... .......... .......... .......... 92% 32.1M 0s 2024-04-25T16:34:09.416 INFO:tasks.workunit.client.0.smithi098.stderr:125900K .......... .......... .......... .......... .......... 92% 26.4M 0s 2024-04-25T16:34:09.418 INFO:tasks.workunit.client.0.smithi098.stderr:125950K .......... .......... .......... .......... .......... 92% 30.9M 0s 2024-04-25T16:34:09.419 INFO:tasks.workunit.client.0.smithi098.stderr:126000K .......... .......... .......... .......... .......... 92% 31.2M 0s 2024-04-25T16:34:09.421 INFO:tasks.workunit.client.0.smithi098.stderr:126050K .......... .......... .......... .......... .......... 92% 30.2M 0s 2024-04-25T16:34:09.423 INFO:tasks.workunit.client.0.smithi098.stderr:126100K .......... .......... .......... .......... .......... 92% 25.3M 0s 2024-04-25T16:34:09.424 INFO:tasks.workunit.client.0.smithi098.stderr:126150K .......... .......... .......... .......... .......... 92% 29.7M 0s 2024-04-25T16:34:09.426 INFO:tasks.workunit.client.0.smithi098.stderr:126200K .......... .......... .......... .......... .......... 92% 26.4M 0s 2024-04-25T16:34:09.428 INFO:tasks.workunit.client.0.smithi098.stderr:126250K .......... .......... .......... .......... .......... 93% 28.8M 0s 2024-04-25T16:34:09.430 INFO:tasks.workunit.client.0.smithi098.stderr:126300K .......... .......... .......... .......... .......... 93% 26.0M 0s 2024-04-25T16:34:09.432 INFO:tasks.workunit.client.0.smithi098.stderr:126350K .......... .......... .......... .......... .......... 93% 26.1M 0s 2024-04-25T16:34:09.433 INFO:tasks.workunit.client.0.smithi098.stderr:126400K .......... .......... .......... .......... .......... 93% 25.3M 0s 2024-04-25T16:34:09.436 INFO:tasks.workunit.client.0.smithi098.stderr:126450K .......... .......... .......... .......... .......... 93% 25.5M 0s 2024-04-25T16:34:09.438 INFO:tasks.workunit.client.0.smithi098.stderr:126500K .......... .......... .......... .......... .......... 93% 21.4M 0s 2024-04-25T16:34:09.440 INFO:tasks.workunit.client.0.smithi098.stderr:126550K .......... .......... .......... .......... .......... 93% 26.1M 0s 2024-04-25T16:34:09.441 INFO:tasks.workunit.client.0.smithi098.stderr:126600K .......... .......... .......... .......... .......... 93% 25.2M 0s 2024-04-25T16:34:09.444 INFO:tasks.workunit.client.0.smithi098.stderr:126650K .......... .......... .......... .......... .......... 93% 26.7M 0s 2024-04-25T16:34:09.446 INFO:tasks.workunit.client.0.smithi098.stderr:126700K .......... .......... .......... .......... .......... 93% 20.9M 0s 2024-04-25T16:34:09.448 INFO:tasks.workunit.client.0.smithi098.stderr:126750K .......... .......... .......... .......... .......... 93% 22.6M 0s 2024-04-25T16:34:09.450 INFO:tasks.workunit.client.0.smithi098.stderr:126800K .......... .......... .......... .......... .......... 93% 22.8M 0s 2024-04-25T16:34:09.452 INFO:tasks.workunit.client.0.smithi098.stderr:126850K .......... .......... .......... .......... .......... 93% 22.2M 0s 2024-04-25T16:34:09.455 INFO:tasks.workunit.client.0.smithi098.stderr:126900K .......... .......... .......... .......... .......... 93% 20.6M 0s 2024-04-25T16:34:09.457 INFO:tasks.workunit.client.0.smithi098.stderr:126950K .......... .......... .......... .......... .......... 93% 22.1M 0s 2024-04-25T16:34:09.458 INFO:tasks.workunit.client.0.smithi098.stderr:127000K .......... .......... .......... .......... .......... 93% 30.0M 0s 2024-04-25T16:34:09.460 INFO:tasks.workunit.client.0.smithi098.stderr:127050K .......... .......... .......... .......... .......... 93% 28.3M 0s 2024-04-25T16:34:09.462 INFO:tasks.workunit.client.0.smithi098.stderr:127100K .......... .......... .......... .......... .......... 93% 26.1M 0s 2024-04-25T16:34:09.465 INFO:tasks.workunit.client.0.smithi098.stderr:127150K .......... .......... .......... .......... .......... 93% 24.0M 0s 2024-04-25T16:34:09.467 INFO:tasks.workunit.client.0.smithi098.stderr:127200K .......... .......... .......... .......... .......... 93% 15.0M 0s 2024-04-25T16:34:09.469 INFO:tasks.workunit.client.0.smithi098.stderr:127250K .......... .......... .......... .......... .......... 93% 33.0M 0s 2024-04-25T16:34:09.471 INFO:tasks.workunit.client.0.smithi098.stderr:127300K .......... .......... .......... .......... .......... 93% 22.3M 0s 2024-04-25T16:34:09.472 INFO:tasks.workunit.client.0.smithi098.stderr:127350K .......... .......... .......... .......... .......... 93% 33.9M 0s 2024-04-25T16:34:09.474 INFO:tasks.workunit.client.0.smithi098.stderr:127400K .......... .......... .......... .......... .......... 93% 35.6M 0s 2024-04-25T16:34:09.475 INFO:tasks.workunit.client.0.smithi098.stderr:127450K .......... .......... .......... .......... .......... 93% 36.5M 0s 2024-04-25T16:34:09.477 INFO:tasks.workunit.client.0.smithi098.stderr:127500K .......... .......... .......... .......... .......... 93% 30.7M 0s 2024-04-25T16:34:09.478 INFO:tasks.workunit.client.0.smithi098.stderr:127550K .......... .......... .......... .......... .......... 93% 35.4M 0s 2024-04-25T16:34:09.479 INFO:tasks.workunit.client.0.smithi098.stderr:127600K .......... .......... .......... .......... .......... 94% 34.7M 0s 2024-04-25T16:34:09.481 INFO:tasks.workunit.client.0.smithi098.stderr:127650K .......... .......... .......... .......... .......... 94% 33.3M 0s 2024-04-25T16:34:09.483 INFO:tasks.workunit.client.0.smithi098.stderr:127700K .......... .......... .......... .......... .......... 94% 28.7M 0s 2024-04-25T16:34:09.484 INFO:tasks.workunit.client.0.smithi098.stderr:127750K .......... .......... .......... .......... .......... 94% 35.6M 0s 2024-04-25T16:34:09.485 INFO:tasks.workunit.client.0.smithi098.stderr:127800K .......... .......... .......... .......... .......... 94% 33.9M 0s 2024-04-25T16:34:09.487 INFO:tasks.workunit.client.0.smithi098.stderr:127850K .......... .......... .......... .......... .......... 94% 33.0M 0s 2024-04-25T16:34:09.489 INFO:tasks.workunit.client.0.smithi098.stderr:127900K .......... .......... .......... .......... .......... 94% 29.4M 0s 2024-04-25T16:34:09.490 INFO:tasks.workunit.client.0.smithi098.stderr:127950K .......... .......... .......... .......... .......... 94% 34.1M 0s 2024-04-25T16:34:09.492 INFO:tasks.workunit.client.0.smithi098.stderr:128000K .......... .......... .......... .......... .......... 94% 34.2M 0s 2024-04-25T16:34:09.493 INFO:tasks.workunit.client.0.smithi098.stderr:128050K .......... .......... .......... .......... .......... 94% 30.9M 0s 2024-04-25T16:34:09.495 INFO:tasks.workunit.client.0.smithi098.stderr:128100K .......... .......... .......... .......... .......... 94% 27.3M 0s 2024-04-25T16:34:09.496 INFO:tasks.workunit.client.0.smithi098.stderr:128150K .......... .......... .......... .......... .......... 94% 32.4M 0s 2024-04-25T16:34:09.498 INFO:tasks.workunit.client.0.smithi098.stderr:128200K .......... .......... .......... .......... .......... 94% 30.6M 0s 2024-04-25T16:34:09.500 INFO:tasks.workunit.client.0.smithi098.stderr:128250K .......... .......... .......... .......... .......... 94% 32.0M 0s 2024-04-25T16:34:09.501 INFO:tasks.workunit.client.0.smithi098.stderr:128300K .......... .......... .......... .......... .......... 94% 26.2M 0s 2024-04-25T16:34:09.503 INFO:tasks.workunit.client.0.smithi098.stderr:128350K .......... .......... .......... .......... .......... 94% 30.8M 0s 2024-04-25T16:34:09.504 INFO:tasks.workunit.client.0.smithi098.stderr:128400K .......... .......... .......... .......... .......... 94% 31.8M 0s 2024-04-25T16:34:09.506 INFO:tasks.workunit.client.0.smithi098.stderr:128450K .......... .......... .......... .......... .......... 94% 31.3M 0s 2024-04-25T16:34:09.508 INFO:tasks.workunit.client.0.smithi098.stderr:128500K .......... .......... .......... .......... .......... 94% 26.3M 0s 2024-04-25T16:34:09.509 INFO:tasks.workunit.client.0.smithi098.stderr:128550K .......... .......... .......... .......... .......... 94% 29.0M 0s 2024-04-25T16:34:09.511 INFO:tasks.workunit.client.0.smithi098.stderr:128600K .......... .......... .......... .......... .......... 94% 29.4M 0s 2024-04-25T16:34:09.513 INFO:tasks.workunit.client.0.smithi098.stderr:128650K .......... .......... .......... .......... .......... 94% 32.1M 0s 2024-04-25T16:34:09.514 INFO:tasks.workunit.client.0.smithi098.stderr:128700K .......... .......... .......... .......... .......... 94% 29.2M 0s 2024-04-25T16:34:09.516 INFO:tasks.workunit.client.0.smithi098.stderr:128750K .......... .......... .......... .......... .......... 94% 33.2M 0s 2024-04-25T16:34:09.517 INFO:tasks.workunit.client.0.smithi098.stderr:128800K .......... .......... .......... .......... .......... 94% 33.9M 0s 2024-04-25T16:34:09.519 INFO:tasks.workunit.client.0.smithi098.stderr:128850K .......... .......... .......... .......... .......... 94% 33.4M 0s 2024-04-25T16:34:09.520 INFO:tasks.workunit.client.0.smithi098.stderr:128900K .......... .......... .......... .......... .......... 94% 29.0M 0s 2024-04-25T16:34:09.522 INFO:tasks.workunit.client.0.smithi098.stderr:128950K .......... .......... .......... .......... .......... 95% 32.8M 0s 2024-04-25T16:34:09.523 INFO:tasks.workunit.client.0.smithi098.stderr:129000K .......... .......... .......... .......... .......... 95% 33.0M 0s 2024-04-25T16:34:09.525 INFO:tasks.workunit.client.0.smithi098.stderr:129050K .......... .......... .......... .......... .......... 95% 32.9M 0s 2024-04-25T16:34:09.527 INFO:tasks.workunit.client.0.smithi098.stderr:129100K .......... .......... .......... .......... .......... 95% 28.0M 0s 2024-04-25T16:34:09.528 INFO:tasks.workunit.client.0.smithi098.stderr:129150K .......... .......... .......... .......... .......... 95% 32.8M 0s 2024-04-25T16:34:09.530 INFO:tasks.workunit.client.0.smithi098.stderr:129200K .......... .......... .......... .......... .......... 95% 29.4M 0s 2024-04-25T16:34:09.532 INFO:tasks.workunit.client.0.smithi098.stderr:129250K .......... .......... .......... .......... .......... 95% 28.2M 0s 2024-04-25T16:34:09.533 INFO:tasks.workunit.client.0.smithi098.stderr:129300K .......... .......... .......... .......... .......... 95% 25.7M 0s 2024-04-25T16:34:09.535 INFO:tasks.workunit.client.0.smithi098.stderr:129350K .......... .......... .......... .......... .......... 95% 31.7M 0s 2024-04-25T16:34:09.537 INFO:tasks.workunit.client.0.smithi098.stderr:129400K .......... .......... .......... .......... .......... 95% 31.7M 0s 2024-04-25T16:34:09.538 INFO:tasks.workunit.client.0.smithi098.stderr:129450K .......... .......... .......... .......... .......... 95% 31.8M 0s 2024-04-25T16:34:09.540 INFO:tasks.workunit.client.0.smithi098.stderr:129500K .......... .......... .......... .......... .......... 95% 27.3M 0s 2024-04-25T16:34:09.541 INFO:tasks.workunit.client.0.smithi098.stderr:129550K .......... .......... .......... .......... .......... 95% 31.2M 0s 2024-04-25T16:34:09.548 INFO:tasks.workunit.client.0.smithi098.stderr:129600K .......... .......... .......... .......... .......... 95% 31.1M 0s 2024-04-25T16:34:09.548 INFO:tasks.workunit.client.0.smithi098.stderr:129650K .......... .......... .......... .......... .......... 95% 31.0M 0s 2024-04-25T16:34:09.548 INFO:tasks.workunit.client.0.smithi098.stderr:129700K .......... .......... .......... .......... .......... 95% 26.4M 0s 2024-04-25T16:34:09.548 INFO:tasks.workunit.client.0.smithi098.stderr:129750K .......... .......... .......... .......... .......... 95% 30.7M 0s 2024-04-25T16:34:09.550 INFO:tasks.workunit.client.0.smithi098.stderr:129800K .......... .......... .......... .......... .......... 95% 30.4M 0s 2024-04-25T16:34:09.552 INFO:tasks.workunit.client.0.smithi098.stderr:129850K .......... .......... .......... .......... .......... 95% 30.5M 0s 2024-04-25T16:34:09.553 INFO:tasks.workunit.client.0.smithi098.stderr:129900K .......... .......... .......... .......... .......... 95% 25.8M 0s 2024-04-25T16:34:09.555 INFO:tasks.workunit.client.0.smithi098.stderr:129950K .......... .......... .......... .......... .......... 95% 30.0M 0s 2024-04-25T16:34:09.556 INFO:tasks.workunit.client.0.smithi098.stderr:130000K .......... .......... .......... .......... .......... 95% 30.4M 0s 2024-04-25T16:34:09.558 INFO:tasks.workunit.client.0.smithi098.stderr:130050K .......... .......... .......... .......... .......... 95% 28.2M 0s 2024-04-25T16:34:09.560 INFO:tasks.workunit.client.0.smithi098.stderr:130100K .......... .......... .......... .......... .......... 95% 23.8M 0s 2024-04-25T16:34:09.562 INFO:tasks.workunit.client.0.smithi098.stderr:130150K .......... .......... .......... .......... .......... 95% 28.2M 0s 2024-04-25T16:34:09.564 INFO:tasks.workunit.client.0.smithi098.stderr:130200K .......... .......... .......... .......... .......... 95% 26.2M 0s 2024-04-25T16:34:09.566 INFO:tasks.workunit.client.0.smithi098.stderr:130250K .......... .......... .......... .......... .......... 95% 25.6M 0s 2024-04-25T16:34:09.568 INFO:tasks.workunit.client.0.smithi098.stderr:130300K .......... .......... .......... .......... .......... 96% 23.4M 0s 2024-04-25T16:34:09.570 INFO:tasks.workunit.client.0.smithi098.stderr:130350K .......... .......... .......... .......... .......... 96% 26.5M 0s 2024-04-25T16:34:09.572 INFO:tasks.workunit.client.0.smithi098.stderr:130400K .......... .......... .......... .......... .......... 96% 27.0M 0s 2024-04-25T16:34:09.573 INFO:tasks.workunit.client.0.smithi098.stderr:130450K .......... .......... .......... .......... .......... 96% 26.6M 0s 2024-04-25T16:34:09.575 INFO:tasks.workunit.client.0.smithi098.stderr:130500K .......... .......... .......... .......... .......... 96% 22.4M 0s 2024-04-25T16:34:09.577 INFO:tasks.workunit.client.0.smithi098.stderr:130550K .......... .......... .......... .......... .......... 96% 26.6M 0s 2024-04-25T16:34:09.579 INFO:tasks.workunit.client.0.smithi098.stderr:130600K .......... .......... .......... .......... .......... 96% 26.3M 0s 2024-04-25T16:34:09.581 INFO:tasks.workunit.client.0.smithi098.stderr:130650K .......... .......... .......... .......... .......... 96% 26.0M 0s 2024-04-25T16:34:09.583 INFO:tasks.workunit.client.0.smithi098.stderr:130700K .......... .......... .......... .......... .......... 96% 22.7M 0s 2024-04-25T16:34:09.585 INFO:tasks.workunit.client.0.smithi098.stderr:130750K .......... .......... .......... .......... .......... 96% 26.7M 0s 2024-04-25T16:34:09.587 INFO:tasks.workunit.client.0.smithi098.stderr:130800K .......... .......... .......... .......... .......... 96% 25.7M 0s 2024-04-25T16:34:09.589 INFO:tasks.workunit.client.0.smithi098.stderr:130850K .......... .......... .......... .......... .......... 96% 25.6M 0s 2024-04-25T16:34:09.591 INFO:tasks.workunit.client.0.smithi098.stderr:130900K .......... .......... .......... .......... .......... 96% 22.0M 0s 2024-04-25T16:34:09.593 INFO:tasks.workunit.client.0.smithi098.stderr:130950K .......... .......... .......... .......... .......... 96% 25.5M 0s 2024-04-25T16:34:09.595 INFO:tasks.workunit.client.0.smithi098.stderr:131000K .......... .......... .......... .......... .......... 96% 24.7M 0s 2024-04-25T16:34:09.597 INFO:tasks.workunit.client.0.smithi098.stderr:131050K .......... .......... .......... .......... .......... 96% 25.7M 0s 2024-04-25T16:34:09.598 INFO:tasks.workunit.client.0.smithi098.stderr:131100K .......... .......... .......... .......... .......... 96% 36.0M 0s 2024-04-25T16:34:09.600 INFO:tasks.workunit.client.0.smithi098.stderr:131150K .......... .......... .......... .......... .......... 96% 32.6M 0s 2024-04-25T16:34:09.601 INFO:tasks.workunit.client.0.smithi098.stderr:131200K .......... .......... .......... .......... .......... 96% 34.6M 0s 2024-04-25T16:34:09.602 INFO:tasks.workunit.client.0.smithi098.stderr:131250K .......... .......... .......... .......... .......... 96% 37.3M 0s 2024-04-25T16:34:09.604 INFO:tasks.workunit.client.0.smithi098.stderr:131300K .......... .......... .......... .......... .......... 96% 32.7M 0s 2024-04-25T16:34:09.605 INFO:tasks.workunit.client.0.smithi098.stderr:131350K .......... .......... .......... .......... .......... 96% 39.3M 0s 2024-04-25T16:34:09.606 INFO:tasks.workunit.client.0.smithi098.stderr:131400K .......... .......... .......... .......... .......... 96% 39.7M 0s 2024-04-25T16:34:09.608 INFO:tasks.workunit.client.0.smithi098.stderr:131450K .......... .......... .......... .......... .......... 96% 34.1M 0s 2024-04-25T16:34:09.610 INFO:tasks.workunit.client.0.smithi098.stderr:131500K .......... .......... .......... .......... .......... 96% 25.0M 0s 2024-04-25T16:34:09.611 INFO:tasks.workunit.client.0.smithi098.stderr:131550K .......... .......... .......... .......... .......... 96% 33.8M 0s 2024-04-25T16:34:09.613 INFO:tasks.workunit.client.0.smithi098.stderr:131600K .......... .......... .......... .......... .......... 96% 35.5M 0s 2024-04-25T16:34:09.614 INFO:tasks.workunit.client.0.smithi098.stderr:131650K .......... .......... .......... .......... .......... 97% 38.0M 0s 2024-04-25T16:34:09.616 INFO:tasks.workunit.client.0.smithi098.stderr:131700K .......... .......... .......... .......... .......... 97% 26.8M 0s 2024-04-25T16:34:09.617 INFO:tasks.workunit.client.0.smithi098.stderr:131750K .......... .......... .......... .......... .......... 97% 37.5M 0s 2024-04-25T16:34:09.618 INFO:tasks.workunit.client.0.smithi098.stderr:131800K .......... .......... .......... .......... .......... 97% 37.2M 0s 2024-04-25T16:34:09.619 INFO:tasks.workunit.client.0.smithi098.stderr:131850K .......... .......... .......... .......... .......... 97% 39.0M 0s 2024-04-25T16:34:09.621 INFO:tasks.workunit.client.0.smithi098.stderr:131900K .......... .......... .......... .......... .......... 97% 31.0M 0s 2024-04-25T16:34:09.622 INFO:tasks.workunit.client.0.smithi098.stderr:131950K .......... .......... .......... .......... .......... 97% 37.2M 0s 2024-04-25T16:34:09.624 INFO:tasks.workunit.client.0.smithi098.stderr:132000K .......... .......... .......... .......... .......... 97% 35.4M 0s 2024-04-25T16:34:09.625 INFO:tasks.workunit.client.0.smithi098.stderr:132050K .......... .......... .......... .......... .......... 97% 36.9M 0s 2024-04-25T16:34:09.627 INFO:tasks.workunit.client.0.smithi098.stderr:132100K .......... .......... .......... .......... .......... 97% 31.8M 0s 2024-04-25T16:34:09.628 INFO:tasks.workunit.client.0.smithi098.stderr:132150K .......... .......... .......... .......... .......... 97% 37.4M 0s 2024-04-25T16:34:09.629 INFO:tasks.workunit.client.0.smithi098.stderr:132200K .......... .......... .......... .......... .......... 97% 37.3M 0s 2024-04-25T16:34:09.631 INFO:tasks.workunit.client.0.smithi098.stderr:132250K .......... .......... .......... .......... .......... 97% 36.1M 0s 2024-04-25T16:34:09.632 INFO:tasks.workunit.client.0.smithi098.stderr:132300K .......... .......... .......... .......... .......... 97% 31.1M 0s 2024-04-25T16:34:09.634 INFO:tasks.workunit.client.0.smithi098.stderr:132350K .......... .......... .......... .......... .......... 97% 33.9M 0s 2024-04-25T16:34:09.635 INFO:tasks.workunit.client.0.smithi098.stderr:132400K .......... .......... .......... .......... .......... 97% 34.3M 0s 2024-04-25T16:34:09.636 INFO:tasks.workunit.client.0.smithi098.stderr:132450K .......... .......... .......... .......... .......... 97% 35.2M 0s 2024-04-25T16:34:09.638 INFO:tasks.workunit.client.0.smithi098.stderr:132500K .......... .......... .......... .......... .......... 97% 30.2M 0s 2024-04-25T16:34:09.639 INFO:tasks.workunit.client.0.smithi098.stderr:132550K .......... .......... .......... .......... .......... 97% 34.3M 0s 2024-04-25T16:34:09.641 INFO:tasks.workunit.client.0.smithi098.stderr:132600K .......... .......... .......... .......... .......... 97% 34.3M 0s 2024-04-25T16:34:09.642 INFO:tasks.workunit.client.0.smithi098.stderr:132650K .......... .......... .......... .......... .......... 97% 34.6M 0s 2024-04-25T16:34:09.644 INFO:tasks.workunit.client.0.smithi098.stderr:132700K .......... .......... .......... .......... .......... 97% 28.9M 0s 2024-04-25T16:34:09.645 INFO:tasks.workunit.client.0.smithi098.stderr:132750K .......... .......... .......... .......... .......... 97% 34.3M 0s 2024-04-25T16:34:09.647 INFO:tasks.workunit.client.0.smithi098.stderr:132800K .......... .......... .......... .......... .......... 97% 33.9M 0s 2024-04-25T16:34:09.648 INFO:tasks.workunit.client.0.smithi098.stderr:132850K .......... .......... .......... .......... .......... 97% 34.3M 0s 2024-04-25T16:34:09.650 INFO:tasks.workunit.client.0.smithi098.stderr:132900K .......... .......... .......... .......... .......... 97% 30.1M 0s 2024-04-25T16:34:09.651 INFO:tasks.workunit.client.0.smithi098.stderr:132950K .......... .......... .......... .......... .......... 97% 35.0M 0s 2024-04-25T16:34:09.653 INFO:tasks.workunit.client.0.smithi098.stderr:133000K .......... .......... .......... .......... .......... 98% 34.6M 0s 2024-04-25T16:34:09.654 INFO:tasks.workunit.client.0.smithi098.stderr:133050K .......... .......... .......... .......... .......... 98% 33.9M 0s 2024-04-25T16:34:09.656 INFO:tasks.workunit.client.0.smithi098.stderr:133100K .......... .......... .......... .......... .......... 98% 28.8M 0s 2024-04-25T16:34:09.657 INFO:tasks.workunit.client.0.smithi098.stderr:133150K .......... .......... .......... .......... .......... 98% 33.6M 0s 2024-04-25T16:34:09.659 INFO:tasks.workunit.client.0.smithi098.stderr:133200K .......... .......... .......... .......... .......... 98% 33.8M 0s 2024-04-25T16:34:09.660 INFO:tasks.workunit.client.0.smithi098.stderr:133250K .......... .......... .......... .......... .......... 98% 33.3M 0s 2024-04-25T16:34:09.662 INFO:tasks.workunit.client.0.smithi098.stderr:133300K .......... .......... .......... .......... .......... 98% 28.8M 0s 2024-04-25T16:34:09.663 INFO:tasks.workunit.client.0.smithi098.stderr:133350K .......... .......... .......... .......... .......... 98% 33.4M 0s 2024-04-25T16:34:09.665 INFO:tasks.workunit.client.0.smithi098.stderr:133400K .......... .......... .......... .......... .......... 98% 32.8M 0s 2024-04-25T16:34:09.666 INFO:tasks.workunit.client.0.smithi098.stderr:133450K .......... .......... .......... .......... .......... 98% 34.4M 0s 2024-04-25T16:34:09.668 INFO:tasks.workunit.client.0.smithi098.stderr:133500K .......... .......... .......... .......... .......... 98% 28.7M 0s 2024-04-25T16:34:09.669 INFO:tasks.workunit.client.0.smithi098.stderr:133550K .......... .......... .......... .......... .......... 98% 32.7M 0s 2024-04-25T16:34:09.671 INFO:tasks.workunit.client.0.smithi098.stderr:133600K .......... .......... .......... .......... .......... 98% 31.4M 0s 2024-04-25T16:34:09.672 INFO:tasks.workunit.client.0.smithi098.stderr:133650K .......... .......... .......... .......... .......... 98% 33.3M 0s 2024-04-25T16:34:09.674 INFO:tasks.workunit.client.0.smithi098.stderr:133700K .......... .......... .......... .......... .......... 98% 28.4M 0s 2024-04-25T16:34:09.676 INFO:tasks.workunit.client.0.smithi098.stderr:133750K .......... .......... .......... .......... .......... 98% 32.9M 0s 2024-04-25T16:34:09.677 INFO:tasks.workunit.client.0.smithi098.stderr:133800K .......... .......... .......... .......... .......... 98% 33.2M 0s 2024-04-25T16:34:09.679 INFO:tasks.workunit.client.0.smithi098.stderr:133850K .......... .......... .......... .......... .......... 98% 32.5M 0s 2024-04-25T16:34:09.681 INFO:tasks.workunit.client.0.smithi098.stderr:133900K .......... .......... .......... .......... .......... 98% 23.3M 0s 2024-04-25T16:34:09.683 INFO:tasks.workunit.client.0.smithi098.stderr:133950K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-04-25T16:34:09.685 INFO:tasks.workunit.client.0.smithi098.stderr:134000K .......... .......... .......... .......... .......... 98% 28.4M 0s 2024-04-25T16:34:09.686 INFO:tasks.workunit.client.0.smithi098.stderr:134050K .......... .......... .......... .......... .......... 98% 27.4M 0s 2024-04-25T16:34:09.688 INFO:tasks.workunit.client.0.smithi098.stderr:134100K .......... .......... .......... .......... .......... 98% 23.7M 0s 2024-04-25T16:34:09.690 INFO:tasks.workunit.client.0.smithi098.stderr:134150K .......... .......... .......... .......... .......... 98% 26.9M 0s 2024-04-25T16:34:09.692 INFO:tasks.workunit.client.0.smithi098.stderr:134200K .......... .......... .......... .......... .......... 98% 27.3M 0s 2024-04-25T16:34:09.694 INFO:tasks.workunit.client.0.smithi098.stderr:134250K .......... .......... .......... .......... .......... 98% 27.3M 0s 2024-04-25T16:34:09.696 INFO:tasks.workunit.client.0.smithi098.stderr:134300K .......... .......... .......... .......... .......... 98% 23.3M 0s 2024-04-25T16:34:09.698 INFO:tasks.workunit.client.0.smithi098.stderr:134350K .......... .......... .......... .......... .......... 99% 27.3M 0s 2024-04-25T16:34:09.699 INFO:tasks.workunit.client.0.smithi098.stderr:134400K .......... .......... .......... .......... .......... 99% 26.9M 0s 2024-04-25T16:34:09.701 INFO:tasks.workunit.client.0.smithi098.stderr:134450K .......... .......... .......... .......... .......... 99% 27.3M 0s 2024-04-25T16:34:09.703 INFO:tasks.workunit.client.0.smithi098.stderr:134500K .......... .......... .......... .......... .......... 99% 22.8M 0s 2024-04-25T16:34:09.705 INFO:tasks.workunit.client.0.smithi098.stderr:134550K .......... .......... .......... .......... .......... 99% 27.0M 0s 2024-04-25T16:34:09.707 INFO:tasks.workunit.client.0.smithi098.stderr:134600K .......... .......... .......... .......... .......... 99% 27.0M 0s 2024-04-25T16:34:09.709 INFO:tasks.workunit.client.0.smithi098.stderr:134650K .......... .......... .......... .......... .......... 99% 25.7M 0s 2024-04-25T16:34:09.711 INFO:tasks.workunit.client.0.smithi098.stderr:134700K .......... .......... .......... .......... .......... 99% 22.2M 0s 2024-04-25T16:34:09.713 INFO:tasks.workunit.client.0.smithi098.stderr:134750K .......... .......... .......... .......... .......... 99% 27.1M 0s 2024-04-25T16:34:09.715 INFO:tasks.workunit.client.0.smithi098.stderr:134800K .......... .......... .......... .......... .......... 99% 24.8M 0s 2024-04-25T16:34:09.717 INFO:tasks.workunit.client.0.smithi098.stderr:134850K .......... .......... .......... .......... .......... 99% 24.2M 0s 2024-04-25T16:34:09.719 INFO:tasks.workunit.client.0.smithi098.stderr:134900K .......... .......... .......... .......... .......... 99% 20.9M 0s 2024-04-25T16:34:09.721 INFO:tasks.workunit.client.0.smithi098.stderr:134950K .......... .......... .......... .......... .......... 99% 22.8M 0s 2024-04-25T16:34:09.723 INFO:tasks.workunit.client.0.smithi098.stderr:135000K .......... .......... .......... .......... .......... 99% 25.3M 0s 2024-04-25T16:34:09.725 INFO:tasks.workunit.client.0.smithi098.stderr:135050K .......... .......... .......... .......... .......... 99% 25.9M 0s 2024-04-25T16:34:09.727 INFO:tasks.workunit.client.0.smithi098.stderr:135100K .......... .......... .......... .......... .......... 99% 22.3M 0s 2024-04-25T16:34:09.729 INFO:tasks.workunit.client.0.smithi098.stderr:135150K .......... .......... .......... .......... .......... 99% 25.8M 0s 2024-04-25T16:34:09.731 INFO:tasks.workunit.client.0.smithi098.stderr:135200K .......... .......... .......... .......... .......... 99% 26.5M 0s 2024-04-25T16:34:09.735 INFO:tasks.workunit.client.0.smithi098.stderr:135250K .......... .......... .......... .......... .......... 99% 24.7M 0s 2024-04-25T16:34:09.736 INFO:tasks.workunit.client.0.smithi098.stderr:135300K .......... .......... .......... .......... .......... 99% 20.4M 0s 2024-04-25T16:34:09.738 INFO:tasks.workunit.client.0.smithi098.stderr:135350K .......... .......... .......... .......... .......... 99% 22.9M 0s 2024-04-25T16:34:09.739 INFO:tasks.workunit.client.0.smithi098.stderr:135400K .......... .......... .......... .......... .......... 99% 27.8M 0s 2024-04-25T16:34:09.741 INFO:tasks.workunit.client.0.smithi098.stderr:135450K .......... .......... .......... .......... .......... 99% 32.4M 0s 2024-04-25T16:34:09.743 INFO:tasks.workunit.client.0.smithi098.stderr:135500K .......... .......... .......... .......... .......... 99% 23.7M 0s 2024-04-25T16:34:09.744 INFO:tasks.workunit.client.0.smithi098.stderr:135550K .......... .......... .......... .......... .......... 99% 34.7M 0s 2024-04-25T16:34:09.746 INFO:tasks.workunit.client.0.smithi098.stderr:135600K .......... .......... .......... .......... .......... 99% 35.2M 0s 2024-04-25T16:34:09.747 INFO:tasks.workunit.client.0.smithi098.stderr:135650K .......... .......... .......... .......... .......... 99% 36.2M 0s 2024-04-25T16:34:09.749 INFO:tasks.workunit.client.0.smithi098.stderr:135700K .......... .......... .......... .......... .......... 99% 30.3M 0s 2024-04-25T16:34:09.749 INFO:tasks.workunit.client.0.smithi098.stderr:135750K .. 100% 5.30T=4.7s 2024-04-25T16:34:09.749 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:09.749 INFO:tasks.workunit.client.0.smithi098.stderr:2024-04-25 16:34:09 (28.1 MB/s) - ‘linux.tar.xz’ saved [139010916/139010916] 2024-04-25T16:34:09.749 INFO:tasks.workunit.client.0.smithi098.stderr: 2024-04-25T16:34:09.749 INFO:tasks.workunit.client.0.smithi098.stderr:+ mkdir t 2024-04-25T16:34:09.752 INFO:tasks.workunit.client.0.smithi098.stderr:+ cd t 2024-04-25T16:34:09.754 INFO:tasks.workunit.client.0.smithi098.stderr:+ tar xJf ../linux.tar.xz 2024-04-25T16:34:10.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:10 smithi203 ceph-mon[33115]: pgmap v406: 129 pgs: 129 active+clean; 93 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.7 MiB/s wr, 12 op/s 2024-04-25T16:34:10.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:10 smithi098 ceph-mon[29162]: pgmap v406: 129 pgs: 129 active+clean; 93 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.7 MiB/s wr, 12 op/s 2024-04-25T16:34:10.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:10 smithi173 ceph-mon[32556]: pgmap v406: 129 pgs: 129 active+clean; 93 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 7.7 MiB/s wr, 12 op/s 2024-04-25T16:34:12.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:12 smithi203 ceph-mon[33115]: pgmap v407: 129 pgs: 129 active+clean; 97 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.0 MiB/s wr, 14 op/s 2024-04-25T16:34:12.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:12 smithi098 ceph-mon[29162]: pgmap v407: 129 pgs: 129 active+clean; 97 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.0 MiB/s wr, 14 op/s 2024-04-25T16:34:12.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:12 smithi173 ceph-mon[32556]: pgmap v407: 129 pgs: 129 active+clean; 97 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 8.0 MiB/s wr, 14 op/s 2024-04-25T16:34:14.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:14 smithi203 ceph-mon[33115]: pgmap v408: 129 pgs: 129 active+clean; 134 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 11 MiB/s wr, 31 op/s 2024-04-25T16:34:14.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:14 smithi098 ceph-mon[29162]: pgmap v408: 129 pgs: 129 active+clean; 134 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 11 MiB/s wr, 31 op/s 2024-04-25T16:34:14.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:14 smithi173 ceph-mon[32556]: pgmap v408: 129 pgs: 129 active+clean; 134 MiB data, 648 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 11 MiB/s wr, 31 op/s 2024-04-25T16:34:16.405 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:34:16.405 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:34:16.429 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:16 smithi098 ceph-mon[29162]: pgmap v409: 129 pgs: 129 active+clean; 134 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 11 MiB/s wr, 33 op/s 2024-04-25T16:34:16.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:16 smithi203 ceph-mon[33115]: pgmap v409: 129 pgs: 129 active+clean; 134 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 11 MiB/s wr, 33 op/s 2024-04-25T16:34:16.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:16 smithi173 ceph-mon[32556]: pgmap v409: 129 pgs: 129 active+clean; 134 MiB data, 657 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 11 MiB/s wr, 33 op/s 2024-04-25T16:34:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.542+0000 7f3617cca640 1 -- 172.21.15.98:0/2042049108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 msgr2=0x7f3610106a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.542+0000 7f3617cca640 1 --2- 172.21.15.98:0/2042049108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 0x7f3610106a00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f360c009900 tx=0x7f360c02a410 comp rx=0 tx=0).stop 2024-04-25T16:34:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 -- 172.21.15.98:0/2042049108 shutdown_connections 2024-04-25T16:34:16.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 --2- 172.21.15.98:0/2042049108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36101098a0 0x7f361010bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 --2- 172.21.15.98:0/2042049108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3610106f40 0x7f3610109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 --2- 172.21.15.98:0/2042049108 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 0x7f3610106a00 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 -- 172.21.15.98:0/2042049108 >> 172.21.15.98:0/2042049108 conn(0x7f36100fdbe0 msgr2=0x7f3610100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:16.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 -- 172.21.15.98:0/2042049108 shutdown_connections 2024-04-25T16:34:16.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 -- 172.21.15.98:0/2042049108 wait complete. 2024-04-25T16:34:16.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 Processor -- start 2024-04-25T16:34:16.543 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.543+0000 7f3617cca640 1 -- start start 2024-04-25T16:34:16.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3617cca640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 0x7f361019a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3617cca640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3610106f40 0x7f361019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3617cca640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36101098a0 0x7f36101a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3617cca640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f361010f4a0 con 0x7f36101045e0 2024-04-25T16:34:16.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3617cca640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f361010f320 con 0x7f36101098a0 2024-04-25T16:34:16.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3617cca640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f361010f620 con 0x7f3610106f40 2024-04-25T16:34:16.545 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615a66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 0x7f361019a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615a66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 0x7f361019a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37308/0 (socket says 172.21.15.98:37308) 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615a66640 1 -- 172.21.15.98:0/1854588166 learned_addr learned my addr 172.21.15.98:0/1854588166 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3616267640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36101098a0 0x7f36101a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615265640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3610106f40 0x7f361019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615a66640 1 -- 172.21.15.98:0/1854588166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3610106f40 msgr2=0x7f361019a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615a66640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3610106f40 0x7f361019a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615a66640 1 -- 172.21.15.98:0/1854588166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36101098a0 msgr2=0x7f36101a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615a66640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36101098a0 0x7f36101a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615a66640 1 -- 172.21.15.98:0/1854588166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36101a1d30 con 0x7f36101045e0 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3616267640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36101098a0 0x7f36101a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.544+0000 7f3615265640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3610106f40 0x7f361019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.545+0000 7f3615a66640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 0x7f361019a1f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f360c005ab0 tx=0x7f360c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:16.546 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.545+0000 7f3606ffd640 1 -- 172.21.15.98:0/1854588166 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c03f030 con 0x7f36101045e0 2024-04-25T16:34:16.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.545+0000 7f3606ffd640 1 -- 172.21.15.98:0/1854588166 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f360c032e10 con 0x7f36101045e0 2024-04-25T16:34:16.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.545+0000 7f3617cca640 1 -- 172.21.15.98:0/1854588166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36101a1fc0 con 0x7f36101045e0 2024-04-25T16:34:16.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.545+0000 7f3606ffd640 1 -- 172.21.15.98:0/1854588166 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c035070 con 0x7f36101045e0 2024-04-25T16:34:16.547 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.545+0000 7f3617cca640 1 -- 172.21.15.98:0/1854588166 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36101a2420 con 0x7f36101045e0 2024-04-25T16:34:16.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.548+0000 7f361eb7a740 1 Processor -- start 2024-04-25T16:34:16.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.548+0000 7f361eb7a740 1 -- start start 2024-04-25T16:34:16.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.548+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.548+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dcffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.548+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.548+0000 7f361eb7a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5594e8d34d80 con 0x5594e8a054d0 2024-04-25T16:34:16.549 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.548+0000 7f361eb7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5594e8cee8b0 con 0x5594e8a2e390 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.548+0000 7f361eb7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5594e8d30d40 con 0x5594e8a0d6b0 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3616267640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615265640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3616267640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcfa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:51964/0 (socket says 172.21.15.98:51964) 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615265640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45800/0 (socket says 172.21.15.98:45800) 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3616267640 1 -- 172.21.15.98:0/3232227749 learned_addr learned my addr 172.21.15.98:0/3232227749 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615265640 1 -- 172.21.15.98:0/3232227749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 msgr2=0x5594e8dcfa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615265640 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcfa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615265640 1 -- 172.21.15.98:0/3232227749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 msgr2=0x5594e8dcffe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3606ffd640 1 -- 172.21.15.98:0/1854588166 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f360c042030 con 0x7f36101045e0 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615a66640 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dcffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615265640 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dcffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615265640 1 -- 172.21.15.98:0/3232227749 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594e8cbfa40 con 0x5594e8a0d6b0 2024-04-25T16:34:16.550 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.549+0000 7f3615a66640 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dcffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:34:16.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.550+0000 7f3615265640 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd0c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3600009fa0 tx=0x7f36000118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6952e9fb5fb5097b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:16.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.550+0000 7f35fbfff640 1 -- 172.21.15.98:0/3232227749 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36000040f0 con 0x5594e8a0d6b0 2024-04-25T16:34:16.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.550+0000 7f35fbfff640 1 -- 172.21.15.98:0/3232227749 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36000054b0 con 0x5594e8a0d6b0 2024-04-25T16:34:16.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.550+0000 7f35fbfff640 1 -- 172.21.15.98:0/3232227749 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3600004ca0 con 0x5594e8a0d6b0 2024-04-25T16:34:16.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.550+0000 7f3606ffd640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35fc0afbf0 0x7f35fc0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.550+0000 7f3616267640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35fc0afbf0 0x7f35fc0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.550+0000 7f3606ffd640 1 -- 172.21.15.98:0/1854588166 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f360c0ff320 con 0x7f36101045e0 2024-04-25T16:34:16.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 -- 172.21.15.98:0/3232227749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 msgr2=0x5594e8dd0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f3616267640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35fc0afbf0 0x7f35fc0b2100 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x5594e8dd0ab0 tx=0x7f3608009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:16.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd0c70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3600009fa0 tx=0x7f36000118d0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.554 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 -- 172.21.15.98:0/3232227749 shutdown_connections 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd0c70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dcffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3232227749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcfa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 -- 172.21.15.98:0/3232227749 >> 172.21.15.98:0/3232227749 conn(0x5594e8a224e0 msgr2=0x5594e8dcb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 -- 172.21.15.98:0/3232227749 shutdown_connections 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.551+0000 7f361eb7a740 1 -- 172.21.15.98:0/3232227749 wait complete. 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 1 -- start start 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 1 Processor -- start 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a054d0 0x5594e8dd9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8de0e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5594e8dd1780 con 0x5594e8a0d6b0 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5594e8cbbd10 con 0x5594e8a2e390 2024-04-25T16:34:16.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5594e8dd1900 con 0x5594e8a054d0 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.552+0000 7f361eb7a740 10 client.0 _ll_register_callbacks cb 0x5594e8ce8210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3615265640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3615265640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37328/0 (socket says 172.21.15.98:37328) 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3615265640 1 -- 172.21.15.98:0/3058137027 learned_addr learned my addr 172.21.15.98:0/3058137027 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3615a66640 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a054d0 0x5594e8dd9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3616267640 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8de0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3616267640 1 -- 172.21.15.98:0/3058137027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a054d0 msgr2=0x5594e8dd9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3616267640 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a054d0 0x5594e8dd9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3616267640 1 -- 172.21.15.98:0/3058137027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 msgr2=0x5594e8dd9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3616267640 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3616267640 1 -- 172.21.15.98:0/3058137027 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594e8b107d0 con 0x5594e8a2e390 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3615265640 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f3616267640 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8de0e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3608009fd0 tx=0x7f3608004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a4eb49e612fbb24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f35f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:16.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f35f97fa640 1 -- 172.21.15.98:0/3058137027 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360802d060 con 0x5594e8a2e390 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f361eb7a740 1 -- 172.21.15.98:0/3058137027 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36000054b0 con 0x5594e8a2e390 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.553+0000 7f35f97fa640 1 -- 172.21.15.98:0/3058137027 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f3608017030 con 0x5594e8a2e390 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f35f97fa640 1 -- 172.21.15.98:0/3058137027 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3608034050 con 0x5594e8a2e390 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f361eb7a740 10 client.25544 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f361eb7a740 1 -- 172.21.15.98:0/3058137027 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5594e8a463f0 con 0x5594e8a2e390 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f35f97fa640 1 -- 172.21.15.98:0/3058137027 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f36000054b0 con 0x5594e8a2e390 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f35f97fa640 1 -- 172.21.15.98:0/3058137027 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f36080286b0 con 0x5594e8a2e390 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f361eb7a740 10 client.25544 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f361eb7a740 10 client.25544 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f361eb7a740 20 client.25544 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5594e89ecf60 0x5594e8de73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f361eb7a740 4 client.25544 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.554+0000 7f361eb7a740 1 -- 172.21.15.98:0/3058137027 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5594e8d42990 con 0x5594e89ecf60 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.555+0000 7f3615a66640 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5594e89ecf60 0x5594e8de73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.555+0000 7f35f97fa640 10 client.25544 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:16.558 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.555+0000 7f3615a66640 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5594e89ecf60 0x5594e8de73b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d16e4c6d7b018777 server_cookie=c19bb39ba3edd488 in_seq=0 out_seq=0 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f35f97fa640 1 -- 172.21.15.98:0/3058137027 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5594e8d42990 con 0x5594e89ecf60 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f35f97fa640 10 client.25544 handle_command_reply: tid=0 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 client.25544 shutdown 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 -- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 msgr2=0x5594e8de0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8de0e00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3608009fd0 tx=0x7f3608004ab0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 -- 172.21.15.98:0/3058137027 shutdown_connections 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5594e89ecf60 0x5594e8de73b0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8de0e00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f35f97fa640 0 client.25544 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dd9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 --2- 172.21.15.98:0/3058137027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a054d0 0x5594e8dd9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.559 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 -- 172.21.15.98:0/3058137027 >> 172.21.15.98:0/3058137027 conn(0x5594e8a224e0 msgr2=0x5594e8dca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:16.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 -- 172.21.15.98:0/3058137027 shutdown_connections 2024-04-25T16:34:16.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 1 -- 172.21.15.98:0/3058137027 wait complete. 2024-04-25T16:34:16.560 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.556+0000 7f361eb7a740 20 client.25544 trim_cache size 0 max 16384 2024-04-25T16:34:16.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f361eb7a740 1 Processor -- start 2024-04-25T16:34:16.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f361eb7a740 1 -- start start 2024-04-25T16:34:16.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dd89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.579 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dce780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f361eb7a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36101ac910 con 0x5594e8a054d0 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f361eb7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f36101ae450 con 0x5594e8a2e390 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f361eb7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36101acfe0 con 0x5594e8a0d6b0 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f3615265640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f3615a66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dd89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f3616267640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f3615a66640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dd89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37344/0 (socket says 172.21.15.98:37344) 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f3615a66640 1 -- 172.21.15.98:0/2772911390 learned_addr learned my addr 172.21.15.98:0/2772911390 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.579+0000 7f3615265640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45816/0 (socket says 172.21.15.98:45816) 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f3615a66640 1 -- 172.21.15.98:0/2772911390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 msgr2=0x5594e8dce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f3615a66640 1 --2- 172.21.15.98:0/2772911390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dce780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.580 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f3615a66640 1 -- 172.21.15.98:0/2772911390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 msgr2=0x5594e8dcecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:16.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f3615a66640 1 --2- 172.21.15.98:0/2772911390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f3615a66640 1 -- 172.21.15.98:0/2772911390 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594e8a060f0 con 0x5594e8a054d0 2024-04-25T16:34:16.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f3615a66640 1 --2- 172.21.15.98:0/2772911390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dd89a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f360c0c71a0 tx=0x7f360c0b0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9cf92498ddf453e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:16.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f35fb7fe640 1 -- 172.21.15.98:0/2772911390 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c049040 con 0x5594e8a054d0 2024-04-25T16:34:16.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f35fb7fe640 1 -- 172.21.15.98:0/2772911390 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f360c0c74a0 con 0x5594e8a054d0 2024-04-25T16:34:16.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.580+0000 7f35fb7fe640 1 -- 172.21.15.98:0/2772911390 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c0c3730 con 0x5594e8a054d0 2024-04-25T16:34:16.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 -- 172.21.15.98:0/2772911390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 msgr2=0x5594e8dd89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 --2- 172.21.15.98:0/2772911390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dd89a0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f360c0c71a0 tx=0x7f360c0b0040 comp rx=0 tx=0).stop 2024-04-25T16:34:16.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 -- 172.21.15.98:0/2772911390 shutdown_connections 2024-04-25T16:34:16.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 --2- 172.21.15.98:0/2772911390 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a2e390 0x5594e8dcecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 --2- 172.21.15.98:0/2772911390 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a0d6b0 0x5594e8dce780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.585 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 --2- 172.21.15.98:0/2772911390 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a054d0 0x5594e8dd89a0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 -- 172.21.15.98:0/2772911390 >> 172.21.15.98:0/2772911390 conn(0x5594e8a224e0 msgr2=0x5594e8dce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 -- 172.21.15.98:0/2772911390 shutdown_connections 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.581+0000 7f361eb7a740 1 -- 172.21.15.98:0/2772911390 wait complete. 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 1 -- start start 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 1 Processor -- start 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a054d0 0x5594e8de11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dddd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a2e390 0x5594e8dde2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f36101add80 con 0x5594e8a0d6b0 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f36101ad6b0 con 0x5594e8a054d0 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f36101aa8f0 con 0x5594e8a2e390 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f3615a66640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a054d0 0x5594e8de11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f361eb7a740 10 client.0 _ll_register_callbacks cb 0x7f3608003bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f3615a66640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a054d0 0x5594e8de11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52004/0 (socket says 172.21.15.98:52004) 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f3615a66640 1 -- 172.21.15.98:0/1510055141 learned_addr learned my addr 172.21.15.98:0/1510055141 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f3616267640 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a2e390 0x5594e8dde2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.582+0000 7f3615265640 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dddd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f3615a66640 1 -- 172.21.15.98:0/1510055141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a2e390 msgr2=0x5594e8dde2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f3615a66640 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a2e390 0x5594e8dde2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f3615a66640 1 -- 172.21.15.98:0/1510055141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 msgr2=0x5594e8dddd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f3615a66640 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dddd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f3615a66640 1 -- 172.21.15.98:0/1510055141 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594e8cf54b0 con 0x5594e8a054d0 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f3616267640 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a2e390 0x5594e8dde2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f3615a66640 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a054d0 0x5594e8de11f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f360c0c3520 tx=0x7f360c0fca20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cc85cfc598282da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f35f97fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f35f97fa640 1 -- 172.21.15.98:0/1510055141 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f360c0e0070 con 0x5594e8a054d0 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f35f97fa640 1 -- 172.21.15.98:0/1510055141 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f360c0c7700 con 0x5594e8a054d0 2024-04-25T16:34:16.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.583+0000 7f361eb7a740 1 -- 172.21.15.98:0/1510055141 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f360c0c74a0 con 0x5594e8a054d0 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f35f97fa640 1 -- 172.21.15.98:0/1510055141 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f360c049070 con 0x5594e8a054d0 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f361eb7a740 10 client.25550 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f361eb7a740 1 -- 172.21.15.98:0/1510055141 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5594e8e4e4b0 con 0x5594e8a054d0 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f35f97fa640 1 -- 172.21.15.98:0/1510055141 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f360c0c74a0 con 0x5594e8a054d0 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f35f97fa640 1 -- 172.21.15.98:0/1510055141 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f360c0e1ce0 con 0x5594e8a054d0 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f361eb7a740 10 client.25550 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f361eb7a740 10 client.25550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f361eb7a740 20 client.25550 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f361eb7a740 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5594e89c23a0 0x5594e8e50e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f361eb7a740 4 client.25550 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f361eb7a740 1 -- 172.21.15.98:0/1510055141 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5594e8d3c800 con 0x5594e89c23a0 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.584+0000 7f3615265640 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5594e89c23a0 0x5594e8e50e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.585+0000 7f35f97fa640 10 client.25550 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.585+0000 7f3615265640 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5594e89c23a0 0x5594e8e50e70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc2b066828bac609 server_cookie=63b131cefdb70157 in_seq=0 out_seq=0 2024-04-25T16:34:16.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.585+0000 7f35f97fa640 1 -- 172.21.15.98:0/1510055141 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5594e8d3c800 con 0x5594e89c23a0 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.585+0000 7f35f97fa640 10 client.25550 handle_command_reply: tid=0 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.585+0000 7f361eb7a740 1 client.25550 shutdown 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.585+0000 7f361eb7a740 1 -- 172.21.15.98:0/1510055141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a054d0 msgr2=0x5594e8de11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.585+0000 7f361eb7a740 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a054d0 0x5594e8de11f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f360c0c3520 tx=0x7f360c0fca20 comp rx=0 tx=0).stop 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 1 -- 172.21.15.98:0/1510055141 shutdown_connections 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x5594e89c23a0 0x5594e8e50e70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5594e8a2e390 0x5594e8dde2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5594e8a0d6b0 0x5594e8dddd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f35f97fa640 0 client.25550 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 1 --2- 172.21.15.98:0/1510055141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5594e8a054d0 0x5594e8de11f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 1 -- 172.21.15.98:0/1510055141 >> 172.21.15.98:0/1510055141 conn(0x5594e8a224e0 msgr2=0x5594e8dceaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 1 -- 172.21.15.98:0/1510055141 shutdown_connections 2024-04-25T16:34:16.589 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 1 -- 172.21.15.98:0/1510055141 wait complete. 2024-04-25T16:34:16.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f361eb7a740 20 client.25550 trim_cache size 0 max 16384 2024-04-25T16:34:16.590 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:34:16.590 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-04-25T16:34:16.590 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-04-25T16:34:16.590 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:34:16.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f3617cca640 1 -- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35fc0afbf0 msgr2=0x7f35fc0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f3617cca640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35fc0afbf0 0x7f35fc0b2100 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x5594e8dd0ab0 tx=0x7f3608009500 comp rx=0 tx=0).stop 2024-04-25T16:34:16.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f3617cca640 1 -- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 msgr2=0x7f361019a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.586+0000 7f3617cca640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 0x7f361019a1f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f360c005ab0 tx=0x7f360c0042f0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.587+0000 7f3617cca640 1 -- 172.21.15.98:0/1854588166 shutdown_connections 2024-04-25T16:34:16.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.587+0000 7f3617cca640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f35fc0afbf0 0x7f35fc0b2100 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.587+0000 7f3617cca640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36101098a0 0x7f36101a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.587+0000 7f3617cca640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3610106f40 0x7f361019a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.587+0000 7f3617cca640 1 --2- 172.21.15.98:0/1854588166 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f36101045e0 0x7f361019a1f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.587+0000 7f3617cca640 1 -- 172.21.15.98:0/1854588166 >> 172.21.15.98:0/1854588166 conn(0x7f36100fdbe0 msgr2=0x7f36101066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:16.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.587+0000 7f3617cca640 1 -- 172.21.15.98:0/1854588166 shutdown_connections 2024-04-25T16:34:16.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.587+0000 7f3617cca640 1 -- 172.21.15.98:0/1854588166 wait complete. 2024-04-25T16:34:16.599 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:34:16.600 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 -- 172.21.15.98:0/2351178437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac1029c0 msgr2=0x7f44ac1036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 --2- 172.21.15.98:0/2351178437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac1029c0 0x7f44ac1036c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f449c009900 tx=0x7f449c02a410 comp rx=0 tx=0).stop 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 -- 172.21.15.98:0/2351178437 shutdown_connections 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 --2- 172.21.15.98:0/2351178437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44ac108900 0x7f44ac10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 --2- 172.21.15.98:0/2351178437 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44ac103c00 0x7f44ac104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 --2- 172.21.15.98:0/2351178437 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac1029c0 0x7f44ac1036c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 -- 172.21.15.98:0/2351178437 >> 172.21.15.98:0/2351178437 conn(0x7f44ac0fdbd0 msgr2=0x7f44ac100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 -- 172.21.15.98:0/2351178437 shutdown_connections 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.779+0000 7f44b22f9640 1 -- 172.21.15.98:0/2351178437 wait complete. 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b22f9640 1 Processor -- start 2024-04-25T16:34:16.779 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b22f9640 1 -- start start 2024-04-25T16:34:16.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b22f9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44ac1029c0 0x7f44ac19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b22f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac103c00 0x7f44ac19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b22f9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44ac108900 0x7f44ac1a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b22f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac10f4c0 con 0x7f44ac103c00 2024-04-25T16:34:16.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b22f9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac10f340 con 0x7f44ac108900 2024-04-25T16:34:16.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b22f9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac10f640 con 0x7f44ac1029c0 2024-04-25T16:34:16.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44ab7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac103c00 0x7f44ac19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44ab7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac103c00 0x7f44ac19a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37384/0 (socket says 172.21.15.98:37384) 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44abfff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44ac1029c0 0x7f44ac19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44b0896640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44ac108900 0x7f44ac1a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.780+0000 7f44ab7fe640 1 -- 172.21.15.98:0/725236323 learned_addr learned my addr 172.21.15.98:0/725236323 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44ab7fe640 1 -- 172.21.15.98:0/725236323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44ac1029c0 msgr2=0x7f44ac19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44ab7fe640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44ac1029c0 0x7f44ac19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44ab7fe640 1 -- 172.21.15.98:0/725236323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44ac108900 msgr2=0x7f44ac1a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44ab7fe640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44ac108900 0x7f44ac1a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44ab7fe640 1 -- 172.21.15.98:0/725236323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ac1a1c80 con 0x7f44ac103c00 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44b0896640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44ac108900 0x7f44ac1a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44ab7fe640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac103c00 0x7f44ac19a740 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f44a000d950 tx=0x7f44a000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44a97fa640 1 -- 172.21.15.98:0/725236323 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a0017030 con 0x7f44ac103c00 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44a97fa640 1 -- 172.21.15.98:0/725236323 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44a0004020 con 0x7f44ac103c00 2024-04-25T16:34:16.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ac1a1f70 con 0x7f44ac103c00 2024-04-25T16:34:16.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44a97fa640 1 -- 172.21.15.98:0/725236323 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a00055d0 con 0x7f44ac103c00 2024-04-25T16:34:16.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.781+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ac1a23d0 con 0x7f44ac103c00 2024-04-25T16:34:16.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.783+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44ac107710 con 0x7f44ac103c00 2024-04-25T16:34:16.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.785+0000 7f44a97fa640 1 -- 172.21.15.98:0/725236323 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f44a0009a30 con 0x7f44ac103c00 2024-04-25T16:34:16.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.785+0000 7f44a97fa640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44940afaf0 0x7f44940b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:16.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.786+0000 7f44abfff640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44940afaf0 0x7f44940b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:16.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.786+0000 7f44a97fa640 1 -- 172.21.15.98:0/725236323 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f44a00defd0 con 0x7f44ac103c00 2024-04-25T16:34:16.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.786+0000 7f44abfff640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44940afaf0 0x7f44940b2000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f449c02a930 tx=0x7f449c031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:16.788 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:16.790+0000 7f44a97fa640 1 -- 172.21.15.98:0/725236323 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f44a00a7800 con 0x7f44ac103c00 2024-04-25T16:34:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.033+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44ac02db50 con 0x7f44ac103c00 2024-04-25T16:34:17.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.033+0000 7f44a97fa640 1 -- 172.21.15.98:0/725236323 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11935 (secure 0 0 0) 0x7f44a00a7100 con 0x7f44ac103c00 2024-04-25T16:34:17.033 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:34:17.033 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:32:32.844737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24494,24473,24457,24484,24539]},"id":1}]} 2024-04-25T16:34:17.033 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 29 2024-04-25T16:34:17.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44940afaf0 msgr2=0x7f44940b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44940afaf0 0x7f44940b2000 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f449c02a930 tx=0x7f449c031400 comp rx=0 tx=0).stop 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac103c00 msgr2=0x7f44ac19a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac103c00 0x7f44ac19a740 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f44a000d950 tx=0x7f44a000de10 comp rx=0 tx=0).stop 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 shutdown_connections 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f44940afaf0 0x7f44940b2000 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44ac108900 0x7f44ac1a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f44ac103c00 0x7f44ac19a740 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 --2- 172.21.15.98:0/725236323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44ac1029c0 0x7f44ac19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.035+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 >> 172.21.15.98:0/725236323 conn(0x7f44ac0fdbd0 msgr2=0x7f44ac1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.036+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 shutdown_connections 2024-04-25T16:34:17.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.036+0000 7f44b22f9640 1 -- 172.21.15.98:0/725236323 wait complete. 2024-04-25T16:34:17.046 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 -- 172.21.15.98:0/2553431623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78103c20 msgr2=0x7fba78104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 --2- 172.21.15.98:0/2553431623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78103c20 0x7fba78104030 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fba68009900 tx=0x7fba6802a410 comp rx=0 tx=0).stop 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 -- 172.21.15.98:0/2553431623 shutdown_connections 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 --2- 172.21.15.98:0/2553431623 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba78108920 0x7fba7810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 --2- 172.21.15.98:0/2553431623 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78103c20 0x7fba78104030 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 --2- 172.21.15.98:0/2553431623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba781029e0 0x7fba781036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 -- 172.21.15.98:0/2553431623 >> 172.21.15.98:0/2553431623 conn(0x7fba780fdbd0 msgr2=0x7fba78100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 -- 172.21.15.98:0/2553431623 shutdown_connections 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.161+0000 7fba7f69a640 1 -- 172.21.15.98:0/2553431623 wait complete. 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7f69a640 1 Processor -- start 2024-04-25T16:34:17.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7f69a640 1 -- start start 2024-04-25T16:34:17.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7f69a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba781029e0 0x7fba7810d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7f69a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba78103c20 0x7fba7810b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7f69a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78108920 0x7fba7810be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7f69a640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810f4c0 con 0x7fba78108920 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7f69a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810f340 con 0x7fba781029e0 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7f69a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba7810f640 con 0x7fba78103c20 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7dc37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78108920 0x7fba7810be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7d436640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba781029e0 0x7fba7810d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7dc37640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78108920 0x7fba7810be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37404/0 (socket says 172.21.15.98:37404) 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7d436640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba781029e0 0x7fba7810d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52054/0 (socket says 172.21.15.98:52054) 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7dc37640 1 -- 172.21.15.98:0/4006821779 learned_addr learned my addr 172.21.15.98:0/4006821779 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.162+0000 7fba7cc35640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba78103c20 0x7fba7810b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7dc37640 1 -- 172.21.15.98:0/4006821779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba78103c20 msgr2=0x7fba7810b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7dc37640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba78103c20 0x7fba7810b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7dc37640 1 -- 172.21.15.98:0/4006821779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba781029e0 msgr2=0x7fba7810d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7dc37640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba781029e0 0x7fba7810d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7dc37640 1 -- 172.21.15.98:0/4006821779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba7810c5c0 con 0x7fba78108920 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7d436640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba781029e0 0x7fba7810d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7dc37640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78108920 0x7fba7810be90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fba7000cad0 tx=0x7fba7000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba6e7fc640 1 -- 172.21.15.98:0/4006821779 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7000ae20 con 0x7fba78108920 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7f69a640 1 -- 172.21.15.98:0/4006821779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba781a4570 con 0x7fba78108920 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba6e7fc640 1 -- 172.21.15.98:0/4006821779 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba70005020 con 0x7fba78108920 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba6e7fc640 1 -- 172.21.15.98:0/4006821779 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7000f6b0 con 0x7fba78108920 2024-04-25T16:34:17.165 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.163+0000 7fba7f69a640 1 -- 172.21.15.98:0/4006821779 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba781a4a30 con 0x7fba78108920 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba6e7fc640 1 -- 172.21.15.98:0/4006821779 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fba700051d0 con 0x7fba78108920 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba8661f740 1 Processor -- start 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba8661f740 1 -- start start 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba8661f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99c6390 0x559bb9d68810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba8661f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb999d4d0 0x559bb9d68d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba8661f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d69a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba8661f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559bb9cccd80 con 0x559bb99a56b0 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba8661f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559bb9c86a20 con 0x559bb999d4d0 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba8661f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559bb9cc8d40 con 0x559bb99c6390 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.166+0000 7fba6e7fc640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba640afaf0 0x7fba640b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7cc35640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99c6390 0x559bb9d68810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7cc35640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99c6390 0x559bb9d68810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45864/0 (socket says 172.21.15.98:45864) 2024-04-25T16:34:17.171 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7cc35640 1 -- 172.21.15.98:0/2495811707 learned_addr learned my addr 172.21.15.98:0/2495811707 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7d436640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb999d4d0 0x559bb9d68d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7cc35640 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d69a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba6e7fc640 1 -- 172.21.15.98:0/4006821779 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fba700df300 con 0x7fba78108920 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7dc37640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba640afaf0 0x7fba640b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7d436640 1 -- 172.21.15.98:0/2495811707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99c6390 msgr2=0x559bb9d68810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7d436640 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99c6390 0x559bb9d68810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7d436640 1 -- 172.21.15.98:0/2495811707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 msgr2=0x559bb9d69a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7d436640 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d69a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7d436640 1 -- 172.21.15.98:0/2495811707 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bb9c91c30 con 0x559bb999d4d0 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7cc35640 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99c6390 0x559bb9d68810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7dc37640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba640afaf0 0x7fba640b2000 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fba700db810 tx=0x7fba70090040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba7d436640 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb999d4d0 0x559bb9d68d90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba781022b0 tx=0x7fba740118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f0896179d856a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba637fe640 1 -- 172.21.15.98:0/2495811707 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74004310 con 0x559bb999d4d0 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba637fe640 1 -- 172.21.15.98:0/2495811707 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba740044b0 con 0x559bb999d4d0 2024-04-25T16:34:17.172 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.167+0000 7fba637fe640 1 -- 172.21.15.98:0/2495811707 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba74004df0 con 0x559bb999d4d0 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 -- 172.21.15.98:0/2495811707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb999d4d0 msgr2=0x559bb9d68d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb999d4d0 0x559bb9d68d90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fba781022b0 tx=0x7fba740118d0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 -- 172.21.15.98:0/2495811707 shutdown_connections 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d69a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb999d4d0 0x559bb9d68d90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 --2- 172.21.15.98:0/2495811707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99c6390 0x559bb9d68810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 -- 172.21.15.98:0/2495811707 >> 172.21.15.98:0/2495811707 conn(0x559bb99ba4e0 msgr2=0x559bb9d64020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 -- 172.21.15.98:0/2495811707 shutdown_connections 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.168+0000 7fba8661f740 1 -- 172.21.15.98:0/2495811707 wait complete. 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 1 -- start start 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 1 Processor -- start 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 0x559bb9d72500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99a56b0 0x559bb9d72a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99c6390 0x559bb9d799a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.173 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x559bb9d6a530 con 0x559bb99c6390 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x559bb9c53dc0 con 0x559bb99a56b0 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x559bb9d6a6b0 con 0x559bb999d4d0 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba8661f740 10 client.0 _ll_register_callbacks cb 0x559bb9c6f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba7cc35640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99c6390 0x559bb9d799a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba7cc35640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99c6390 0x559bb9d799a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37410/0 (socket says 172.21.15.98:37410) 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba7d436640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99a56b0 0x559bb9d72a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba7cc35640 1 -- 172.21.15.98:0/2063792539 learned_addr learned my addr 172.21.15.98:0/2063792539 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba7cc35640 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 0x559bb9d72500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba7cc35640 1 -- 172.21.15.98:0/2063792539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 msgr2=0x559bb9d72500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.169+0000 7fba7cc35640 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 0x559bb9d72500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba7cc35640 1 -- 172.21.15.98:0/2063792539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99a56b0 msgr2=0x559bb9d72a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba7cc35640 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99a56b0 0x559bb9d72a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba7cc35640 1 -- 172.21.15.98:0/2063792539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bb99de3f0 con 0x559bb99c6390 2024-04-25T16:34:17.174 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba7cc35640 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99c6390 0x559bb9d799a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fba68009230 tx=0x7fba6802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ec3d9f0dd4f6cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba60ff9640 1 -- 172.21.15.98:0/2063792539 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba68035040 con 0x559bb99c6390 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba8661f740 1 -- 172.21.15.98:0/2063792539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba740044b0 con 0x559bb99c6390 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba60ff9640 1 -- 172.21.15.98:0/2063792539 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fba68004450 con 0x559bb99c6390 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba60ff9640 1 -- 172.21.15.98:0/2063792539 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba680035f0 con 0x559bb99c6390 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba8661f740 10 client.16047 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba8661f740 1 -- 172.21.15.98:0/2063792539 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559bb9aa87d0 con 0x559bb99c6390 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.170+0000 7fba60ff9640 1 -- 172.21.15.98:0/2063792539 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba740044b0 con 0x559bb99c6390 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.171+0000 7fba60ff9640 1 -- 172.21.15.98:0/2063792539 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fba68047020 con 0x559bb99c6390 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.171+0000 7fba8661f740 10 client.16047 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.171+0000 7fba8661f740 10 client.16047 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.171+0000 7fba8661f740 20 client.16047 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.171+0000 7fba8661f740 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x559bb9984f60 0x559bb9d80220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.171+0000 7fba8661f740 4 client.16047 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:17.175 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.171+0000 7fba8661f740 1 -- 172.21.15.98:0/2063792539 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559bb9cda990 con 0x559bb9984f60 2024-04-25T16:34:17.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.171+0000 7fba7d436640 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x559bb9984f60 0x559bb9d80220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.172+0000 7fba7d436640 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x559bb9984f60 0x559bb9d80220 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bb5d6d9ff81d5da1 server_cookie=1c6ecd9b6b3e6044 in_seq=0 out_seq=0 2024-04-25T16:34:17.176 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.172+0000 7fba60ff9640 10 client.16047 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.172+0000 7fba60ff9640 1 -- 172.21.15.98:0/2063792539 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559bb9cda990 con 0x559bb9984f60 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.172+0000 7fba60ff9640 10 client.16047 handle_command_reply: tid=0 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.172+0000 7fba8661f740 1 client.16047 shutdown 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.172+0000 7fba8661f740 1 -- 172.21.15.98:0/2063792539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99c6390 msgr2=0x559bb9d799a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.172+0000 7fba8661f740 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99c6390 0x559bb9d799a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fba68009230 tx=0x7fba6802a410 comp rx=0 tx=0).stop 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 1 -- 172.21.15.98:0/2063792539 shutdown_connections 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x559bb9984f60 0x559bb9d80220 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99c6390 0x559bb9d799a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba60ff9640 0 client.16047 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:34:17.177 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99a56b0 0x559bb9d72a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 1 --2- 172.21.15.98:0/2063792539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 0x559bb9d72500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 1 -- 172.21.15.98:0/2063792539 >> 172.21.15.98:0/2063792539 conn(0x559bb99ba4e0 msgr2=0x559bb9d66f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 1 -- 172.21.15.98:0/2063792539 shutdown_connections 2024-04-25T16:34:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 1 -- 172.21.15.98:0/2063792539 wait complete. 2024-04-25T16:34:17.178 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.173+0000 7fba8661f740 20 client.16047 trim_cache size 0 max 16384 2024-04-25T16:34:17.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba8661f740 1 Processor -- start 2024-04-25T16:34:17.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba8661f740 1 -- start start 2024-04-25T16:34:17.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba8661f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb999d4d0 0x559bb9d70d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.190 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba8661f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99a56b0 0x559bb9d71260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba8661f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 0x559bb9d63680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba8661f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba781aec40 con 0x559bb999d4d0 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba8661f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fba781b0780 con 0x559bb99c6390 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba8661f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba781af310 con 0x559bb99a56b0 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba7cc35640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb999d4d0 0x559bb9d70d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba7cc35640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb999d4d0 0x559bb9d70d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37420/0 (socket says 172.21.15.98:37420) 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba7cc35640 1 -- 172.21.15.98:0/3953879771 learned_addr learned my addr 172.21.15.98:0/3953879771 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba7d436640 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99a56b0 0x559bb9d71260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.190+0000 7fba7cc35640 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 0x559bb9d63680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba7d436640 1 -- 172.21.15.98:0/3953879771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 msgr2=0x559bb9d63680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba7d436640 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 0x559bb9d63680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba7d436640 1 -- 172.21.15.98:0/3953879771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb999d4d0 msgr2=0x559bb9d70d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba7d436640 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb999d4d0 0x559bb9d70d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba7d436640 1 -- 172.21.15.98:0/3953879771 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559bb9d73640 con 0x559bb99a56b0 2024-04-25T16:34:17.191 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba7d436640 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99a56b0 0x559bb9d71260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba7401b040 tx=0x7fba74009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b39c8b8706cd999 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba62ffd640 1 -- 172.21.15.98:0/3953879771 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7402e5b0 con 0x559bb99a56b0 2024-04-25T16:34:17.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba62ffd640 1 -- 172.21.15.98:0/3953879771 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba740035f0 con 0x559bb99a56b0 2024-04-25T16:34:17.192 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.191+0000 7fba62ffd640 1 -- 172.21.15.98:0/3953879771 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7400f6e0 con 0x559bb99a56b0 2024-04-25T16:34:17.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.192+0000 7fba8661f740 1 -- 172.21.15.98:0/3953879771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99a56b0 msgr2=0x559bb9d71260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.192+0000 7fba8661f740 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99a56b0 0x559bb9d71260 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba7401b040 tx=0x7fba74009310 comp rx=0 tx=0).stop 2024-04-25T16:34:17.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.192+0000 7fba8661f740 1 -- 172.21.15.98:0/3953879771 shutdown_connections 2024-04-25T16:34:17.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.192+0000 7fba8661f740 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 0x559bb9d63680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.192+0000 7fba8661f740 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb99a56b0 0x559bb9d71260 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.192+0000 7fba8661f740 1 --2- 172.21.15.98:0/3953879771 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb999d4d0 0x559bb9d70d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.196 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.192+0000 7fba8661f740 1 -- 172.21.15.98:0/3953879771 >> 172.21.15.98:0/3953879771 conn(0x559bb99ba4e0 msgr2=0x559bb9d6d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.193+0000 7fba8661f740 1 -- 172.21.15.98:0/3953879771 shutdown_connections 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.193+0000 7fba8661f740 1 -- 172.21.15.98:0/3953879771 wait complete. 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.193+0000 7fba8661f740 1 -- start start 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.193+0000 7fba8661f740 1 Processor -- start 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba8661f740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 0x559bb9d75fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba8661f740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d764f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba8661f740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 0x559bb9de6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba8661f740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fba781b00b0 con 0x559bb99a56b0 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba8661f740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fba781af9e0 con 0x559bb99c6390 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba8661f740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fba781acc20 con 0x559bb999d4d0 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba8661f740 10 client.0 _ll_register_callbacks cb 0x7fba68031500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d764f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d764f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37428/0 (socket says 172.21.15.98:37428) 2024-04-25T16:34:17.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 -- 172.21.15.98:0/1665735530 learned_addr learned my addr 172.21.15.98:0/1665735530 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7cc35640 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 0x559bb9d75fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7cc35640 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 0x559bb9de6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 -- 172.21.15.98:0/1665735530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 msgr2=0x559bb9d75fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 0x559bb9d75fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 -- 172.21.15.98:0/1665735530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 msgr2=0x559bb9de6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 0x559bb9de6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 -- 172.21.15.98:0/1665735530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559bb9de6ac0 con 0x559bb99a56b0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba7d436640 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d764f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fba74004310 tx=0x7fba74009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6079deccc03eb7d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba60ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.194+0000 7fba60ff9640 1 -- 172.21.15.98:0/1665735530 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7403c020 con 0x559bb99a56b0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba60ff9640 1 -- 172.21.15.98:0/1665735530 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fba740037a0 con 0x559bb99a56b0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 1 -- 172.21.15.98:0/1665735530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba740035f0 con 0x559bb99a56b0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba60ff9640 1 -- 172.21.15.98:0/1665735530 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba74033040 con 0x559bb99a56b0 2024-04-25T16:34:17.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 10 client.16053 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 1 -- 172.21.15.98:0/1665735530 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559bb9de7c60 con 0x559bb99a56b0 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba60ff9640 1 -- 172.21.15.98:0/1665735530 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fba740035f0 con 0x559bb99a56b0 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba60ff9640 1 -- 172.21.15.98:0/1665735530 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fba7400f480 con 0x559bb99a56b0 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 10 client.16053 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 10 client.16053 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 20 client.16053 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x559bb995a3a0 0x559bb9dea500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 4 client.16053 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba8661f740 1 -- 172.21.15.98:0/1665735530 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559bb9cd4800 con 0x559bb995a3a0 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.195+0000 7fba7cc35640 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x559bb995a3a0 0x559bb9dea500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba60ff9640 10 client.16053 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba7cc35640 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x559bb995a3a0 0x559bb9dea500 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d61b63c567fc53b3 server_cookie=1c11003f500aef4c in_seq=0 out_seq=0 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba60ff9640 1 -- 172.21.15.98:0/1665735530 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559bb9cd4800 con 0x559bb995a3a0 2024-04-25T16:34:17.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba60ff9640 10 client.16053 handle_command_reply: tid=0 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 client.16053 shutdown 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 -- 172.21.15.98:0/1665735530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 msgr2=0x559bb9d764f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d764f0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fba74004310 tx=0x7fba74009c60 comp rx=0 tx=0).stop 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 -- 172.21.15.98:0/1665735530 shutdown_connections 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x559bb995a3a0 0x559bb9dea500 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x559bb99c6390 0x559bb9de6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x559bb99a56b0 0x559bb9d764f0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba60ff9640 0 client.16053 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 --2- 172.21.15.98:0/1665735530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x559bb999d4d0 0x559bb9d75fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 -- 172.21.15.98:0/1665735530 >> 172.21.15.98:0/1665735530 conn(0x559bb99ba4e0 msgr2=0x559bb9d70d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.196+0000 7fba8661f740 1 -- 172.21.15.98:0/1665735530 shutdown_connections 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba8661f740 1 -- 172.21.15.98:0/1665735530 wait complete. 2024-04-25T16:34:17.200 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba8661f740 20 client.16053 trim_cache size 0 max 16384 2024-04-25T16:34:17.201 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:17.201 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 -- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba640afaf0 msgr2=0x7fba640b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba640afaf0 0x7fba640b2000 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fba700db810 tx=0x7fba70090040 comp rx=0 tx=0).stop 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 -- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78108920 msgr2=0x7fba7810be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78108920 0x7fba7810be90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fba7000cad0 tx=0x7fba7000cf90 comp rx=0 tx=0).stop 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 -- 172.21.15.98:0/4006821779 shutdown_connections 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fba640afaf0 0x7fba640b2000 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fba78108920 0x7fba7810be90 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba78103c20 0x7fba7810b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 --2- 172.21.15.98:0/4006821779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba781029e0 0x7fba7810d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 -- 172.21.15.98:0/4006821779 >> 172.21.15.98:0/4006821779 conn(0x7fba780fdbd0 msgr2=0x7fba78104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 -- 172.21.15.98:0/4006821779 shutdown_connections 2024-04-25T16:34:17.202 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.197+0000 7fba7f69a640 1 -- 172.21.15.98:0/4006821779 wait complete. 2024-04-25T16:34:17.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 -- 172.21.15.98:0/1334257314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 msgr2=0x7f684010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 --2- 172.21.15.98:0/1334257314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 0x7f684010ad40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6838009ca0 tx=0x7f683802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 -- 172.21.15.98:0/1334257314 shutdown_connections 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 --2- 172.21.15.98:0/1334257314 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 0x7f684010ad40 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 --2- 172.21.15.98:0/1334257314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6840103c20 0x7f6840104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 --2- 172.21.15.98:0/1334257314 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68401029e0 0x7f68401036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 -- 172.21.15.98:0/1334257314 >> 172.21.15.98:0/1334257314 conn(0x7f68400fdbd0 msgr2=0x7f6840100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 -- 172.21.15.98:0/1334257314 shutdown_connections 2024-04-25T16:34:17.343 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.342+0000 7f684851f640 1 -- 172.21.15.98:0/1334257314 wait complete. 2024-04-25T16:34:17.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f684851f640 1 Processor -- start 2024-04-25T16:34:17.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f684851f640 1 -- start start 2024-04-25T16:34:17.345 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f684851f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68401029e0 0x7f684019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f684851f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6840103c20 0x7f684019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f684851f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 0x7f68401a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f684851f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f684010f520 con 0x7f6840108920 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f684851f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f684010f3a0 con 0x7f6840103c20 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f684851f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f684010f6a0 con 0x7f68401029e0 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f68462bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68401029e0 0x7f684019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f6846abc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 0x7f68401a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f68462bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68401029e0 0x7f684019a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45904/0 (socket says 172.21.15.98:45904) 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f6846abc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 0x7f68401a18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37446/0 (socket says 172.21.15.98:37446) 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f6845aba640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6840103c20 0x7f684019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.343+0000 7f68462bb640 1 -- 172.21.15.98:0/648475176 learned_addr learned my addr 172.21.15.98:0/648475176 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68462bb640 1 -- 172.21.15.98:0/648475176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6840103c20 msgr2=0x7f684019a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68462bb640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6840103c20 0x7f684019a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.346 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68462bb640 1 -- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 msgr2=0x7f68401a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68462bb640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 0x7f68401a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68462bb640 1 -- 172.21.15.98:0/648475176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68401a1f20 con 0x7f68401029e0 2024-04-25T16:34:17.347 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68462bb640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68401029e0 0x7f684019a3e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f683c00e9f0 tx=0x7f683c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68377fe640 1 -- 172.21.15.98:0/648475176 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c00ccc0 con 0x7f68401029e0 2024-04-25T16:34:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f684851f640 1 -- 172.21.15.98:0/648475176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68401a2210 con 0x7f68401029e0 2024-04-25T16:34:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f684851f640 1 -- 172.21.15.98:0/648475176 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68401a27b0 con 0x7f68401029e0 2024-04-25T16:34:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68377fe640 1 -- 172.21.15.98:0/648475176 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f683c0044b0 con 0x7f68401029e0 2024-04-25T16:34:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.344+0000 7f68377fe640 1 -- 172.21.15.98:0/648475176 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c010430 con 0x7f68401029e0 2024-04-25T16:34:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.347+0000 7f684f3cd740 1 Processor -- start 2024-04-25T16:34:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f68377fe640 1 -- 172.21.15.98:0/648475176 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f683c0105d0 con 0x7f68401029e0 2024-04-25T16:34:17.352 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f684f3cd740 1 -- start start 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055377390 0x55a05571dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05571e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05571ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f684f3cd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a05567dd80 con 0x55a055377390 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f684f3cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a055637a20 con 0x55a0553566b0 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f684f3cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a055679d40 con 0x55a05534e4d0 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f68462bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05571e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f6846abc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055377390 0x55a05571dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f6846abc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055377390 0x55a05571dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37454/0 (socket says 172.21.15.98:37454) 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f6845aba640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05571ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f6846abc640 1 -- 172.21.15.98:0/2996371776 learned_addr learned my addr 172.21.15.98:0/2996371776 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.348+0000 7f68462bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05571e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45906/0 (socket says 172.21.15.98:45906) 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6845aba640 1 -- 172.21.15.98:0/2996371776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 msgr2=0x55a05571e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6845aba640 1 --2- 172.21.15.98:0/2996371776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05571e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6845aba640 1 -- 172.21.15.98:0/2996371776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055377390 msgr2=0x55a05571dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6845aba640 1 --2- 172.21.15.98:0/2996371776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055377390 0x55a05571dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6845aba640 1 -- 172.21.15.98:0/2996371776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a055642c30 con 0x55a0553566b0 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f68377fe640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f682c0afb70 0x7f682c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6846abc640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f682c0afb70 0x7f682c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6845aba640 1 --2- 172.21.15.98:0/2996371776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05571ee30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6830009bf0 tx=0x7f6830011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14126182b5368f1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f68377fe640 1 -- 172.21.15.98:0/648475176 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f683c0df300 con 0x7f68401029e0 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6834bfe640 1 -- 172.21.15.98:0/2996371776 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830004050 con 0x55a0553566b0 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6834bfe640 1 -- 172.21.15.98:0/2996371776 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6830005020 con 0x55a0553566b0 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.349+0000 7f6834bfe640 1 -- 172.21.15.98:0/2996371776 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830004cf0 con 0x55a0553566b0 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f6846abc640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f682c0afb70 0x7f682c0b2080 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f68380094e0 tx=0x7f6838031000 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 -- 172.21.15.98:0/2996371776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 msgr2=0x55a05571ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2996371776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05571ee30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6830009bf0 tx=0x7f6830011810 comp rx=0 tx=0).stop 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 -- 172.21.15.98:0/2996371776 shutdown_connections 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2996371776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05571ee30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2996371776 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05571e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2996371776 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055377390 0x55a05571dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 -- 172.21.15.98:0/2996371776 >> 172.21.15.98:0/2996371776 conn(0x55a05536b4e0 msgr2=0x55a055715020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 -- 172.21.15.98:0/2996371776 shutdown_connections 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.350+0000 7f684f3cd740 1 -- 172.21.15.98:0/2996371776 wait complete. 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 1 -- start start 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 1 Processor -- start 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a05534e4d0 0x55a055727920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0553566b0 0x55a055727e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a055377390 0x55a05572edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a05571f940 con 0x55a0553566b0 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f6845aba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0553566b0 0x55a055727e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.355 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f68462bb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a05534e4d0 0x55a055727920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f6845aba640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0553566b0 0x55a055727e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37462/0 (socket says 172.21.15.98:37462) 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f6845aba640 1 -- 172.21.15.98:0/4195684757 learned_addr learned my addr 172.21.15.98:0/4195684757 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a055604dc0 con 0x55a05534e4d0 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a05571fac0 con 0x55a055377390 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f684f3cd740 10 client.0 _ll_register_callbacks cb 0x55a055620600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.351+0000 7f6846abc640 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a055377390 0x55a05572edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f68462bb640 1 -- 172.21.15.98:0/4195684757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a055377390 msgr2=0x55a05572edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f68462bb640 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a055377390 0x55a05572edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f68462bb640 1 -- 172.21.15.98:0/4195684757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0553566b0 msgr2=0x55a055727e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f68462bb640 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0553566b0 0x55a055727e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f68462bb640 1 -- 172.21.15.98:0/4195684757 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a05538f3f0 con 0x55a05534e4d0 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f6845aba640 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0553566b0 0x55a055727e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.356 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f68462bb640 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a05534e4d0 0x55a055727920 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55a05571d940 tx=0x7f683c0a7980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d45b7dca33fa9c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f6829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f6829ffb640 1 -- 172.21.15.98:0/4195684757 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c0a71a0 con 0x55a05534e4d0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f6829ffb640 1 -- 172.21.15.98:0/4195684757 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f683c0a6840 con 0x55a05534e4d0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6830005020 con 0x55a05534e4d0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f6829ffb640 1 -- 172.21.15.98:0/4195684757 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f683c00ce60 con 0x55a05534e4d0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f684f3cd740 10 client.25586 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.352+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a0554597d0 con 0x55a05534e4d0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f6829ffb640 1 -- 172.21.15.98:0/4195684757 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6830005020 con 0x55a05534e4d0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f6829ffb640 1 -- 172.21.15.98:0/4195684757 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f683c0a2d40 con 0x55a05534e4d0 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f684f3cd740 10 client.25586 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f684f3cd740 10 client.25586 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f684f3cd740 20 client.25586 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f684f3cd740 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55a055335f60 0x55a0557355c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f684f3cd740 4 client.25586 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:17.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a05568b990 con 0x55a055335f60 2024-04-25T16:34:17.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.353+0000 7f6845aba640 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55a055335f60 0x55a0557355c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.354+0000 7f6829ffb640 10 client.25586 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.354+0000 7f6845aba640 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55a055335f60 0x55a0557355c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=84ec23ecc2a6bbc0 server_cookie=97748d096bdf0825 in_seq=0 out_seq=0 2024-04-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.354+0000 7f6829ffb640 1 -- 172.21.15.98:0/4195684757 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a05568b990 con 0x55a055335f60 2024-04-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.354+0000 7f6829ffb640 10 client.25586 handle_command_reply: tid=0 2024-04-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 client.25586 shutdown 2024-04-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a05534e4d0 msgr2=0x55a055727920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a05534e4d0 0x55a055727920 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55a05571d940 tx=0x7f683c0a7980 comp rx=0 tx=0).stop 2024-04-25T16:34:17.359 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 shutdown_connections 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55a055335f60 0x55a0557355c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a055377390 0x55a05572edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f6829ffb640 0 client.25586 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a0553566b0 0x55a055727e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 --2- 172.21.15.98:0/4195684757 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a05534e4d0 0x55a055727920 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 >> 172.21.15.98:0/4195684757 conn(0x55a05536b4e0 msgr2=0x55a055714040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 shutdown_connections 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 1 -- 172.21.15.98:0/4195684757 wait complete. 2024-04-25T16:34:17.360 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.355+0000 7f684f3cd740 20 client.25586 trim_cache size 0 max 16384 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.372+0000 7f684f3cd740 1 Processor -- start 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.372+0000 7f684f3cd740 1 -- start start 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.372+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05570fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a055727c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f68462bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05570fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f68462bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05570fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37476/0 (socket says 172.21.15.98:37476) 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f68462bb640 1 -- 172.21.15.98:0/757068387 learned_addr learned my addr 172.21.15.98:0/757068387 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.373 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f684f3cd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f682400fc00 con 0x55a055335f60 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f684f3cd740 1 -- 172.21.15.98:0/757068387 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6824011740 con 0x55a0553566b0 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f6845aba640 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f684f3cd740 1 -- 172.21.15.98:0/757068387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f68240102d0 con 0x55a05534e4d0 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f6846abc640 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a055727c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f6845aba640 1 -- 172.21.15.98:0/757068387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 msgr2=0x55a055727c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f6845aba640 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a055727c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f6845aba640 1 -- 172.21.15.98:0/757068387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 msgr2=0x55a05570fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f6845aba640 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05570fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.373+0000 7f6845aba640 1 -- 172.21.15.98:0/757068387 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a05570f2b0 con 0x55a05534e4d0 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.374+0000 7f6845aba640 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572ed50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6830005e00 tx=0x7f6830009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=881ef49db021a532 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.374+0000 7f682bfff640 1 -- 172.21.15.98:0/757068387 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68300037f0 con 0x55a05534e4d0 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.374+0000 7f682bfff640 1 -- 172.21.15.98:0/757068387 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6830003e20 con 0x55a05534e4d0 2024-04-25T16:34:17.374 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.374+0000 7f682bfff640 1 -- 172.21.15.98:0/757068387 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683002e980 con 0x55a05534e4d0 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 -- 172.21.15.98:0/757068387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 msgr2=0x55a05572ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572ed50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6830005e00 tx=0x7f6830009f70 comp rx=0 tx=0).stop 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 -- 172.21.15.98:0/757068387 shutdown_connections 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a055727c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572ed50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 --2- 172.21.15.98:0/757068387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05570fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 -- 172.21.15.98:0/757068387 >> 172.21.15.98:0/757068387 conn(0x55a05536b4e0 msgr2=0x55a0557274f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 -- 172.21.15.98:0/757068387 shutdown_connections 2024-04-25T16:34:17.379 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 -- 172.21.15.98:0/757068387 wait complete. 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 -- start start 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.375+0000 7f684f3cd740 1 Processor -- start 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05572b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f684f3cd740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05579b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f684f3cd740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6824011070 con 0x55a055335f60 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f684f3cd740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f68240109a0 con 0x55a0553566b0 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f684f3cd740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f682400dbe0 con 0x55a05534e4d0 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f684f3cd740 10 client.0 _ll_register_callbacks cb 0x7f683c0046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f68462bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05572b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6845aba640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f68462bb640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05572b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37490/0 (socket says 172.21.15.98:37490) 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f68462bb640 1 -- 172.21.15.98:0/2038342461 learned_addr learned my addr 172.21.15.98:0/2038342461 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.380 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6846abc640 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05579b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6845aba640 1 -- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 msgr2=0x55a05579b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6845aba640 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05579b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6845aba640 1 -- 172.21.15.98:0/2038342461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 msgr2=0x55a05572b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6845aba640 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05572b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6845aba640 1 -- 172.21.15.98:0/2038342461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a05572edf0 con 0x55a05534e4d0 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f68462bb640 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05572b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6846abc640 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05579b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6845aba640 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572be30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6830009e40 tx=0x7f6830011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa1447630a670744 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6829ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.376+0000 7f6829ffb640 1 -- 172.21.15.98:0/2038342461 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6830005bf0 con 0x55a05534e4d0 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f6829ffb640 1 -- 172.21.15.98:0/2038342461 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f683002e880 con 0x55a05534e4d0 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 1 -- 172.21.15.98:0/2038342461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6830003e20 con 0x55a05534e4d0 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f6829ffb640 1 -- 172.21.15.98:0/2038342461 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f683002ea80 con 0x55a05534e4d0 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 10 client.25519 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.381 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 1 -- 172.21.15.98:0/2038342461 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a05579cc80 con 0x55a05534e4d0 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f6829ffb640 1 -- 172.21.15.98:0/2038342461 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6830003e20 con 0x55a05534e4d0 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f6829ffb640 1 -- 172.21.15.98:0/2038342461 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f6830019780 con 0x55a05534e4d0 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 10 client.25519 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 10 client.25519 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 20 client.25519 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55a055377390 0x55a05579f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 4 client.25519 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.377+0000 7f684f3cd740 1 -- 172.21.15.98:0/2038342461 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a055685800 con 0x55a055377390 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f68462bb640 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55a055377390 0x55a05579f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f6829ffb640 10 client.25519 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f68462bb640 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55a055377390 0x55a05579f3e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=55f07b376692634b server_cookie=174c0af5a59766a9 in_seq=0 out_seq=0 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f6829ffb640 1 -- 172.21.15.98:0/2038342461 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a055685800 con 0x55a055377390 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f6829ffb640 10 client.25519 handle_command_reply: tid=0 2024-04-25T16:34:17.382 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 client.25519 shutdown 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 -- 172.21.15.98:0/2038342461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 msgr2=0x55a05572be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572be30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6830009e40 tx=0x7f6830011810 comp rx=0 tx=0).stop 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 -- 172.21.15.98:0/2038342461 shutdown_connections 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55a055377390 0x55a05579f3e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a0553566b0 0x55a05579b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a05534e4d0 0x55a05572be30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f6829ffb640 0 client.25519 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 --2- 172.21.15.98:0/2038342461 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a055335f60 0x55a05572b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 -- 172.21.15.98:0/2038342461 >> 172.21.15.98:0/2038342461 conn(0x55a05536b4e0 msgr2=0x55a055717e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.378+0000 7f684f3cd740 1 -- 172.21.15.98:0/2038342461 shutdown_connections 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684f3cd740 1 -- 172.21.15.98:0/2038342461 wait complete. 2024-04-25T16:34:17.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684f3cd740 20 client.25519 trim_cache size 0 max 16384 2024-04-25T16:34:17.384 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 -- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f682c0afb70 msgr2=0x7f682c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f682c0afb70 0x7f682c0b2080 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f68380094e0 tx=0x7f6838031000 comp rx=0 tx=0).stop 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 -- 172.21.15.98:0/648475176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68401029e0 msgr2=0x7f684019a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68401029e0 0x7f684019a3e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f683c00e9f0 tx=0x7f683c00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 -- 172.21.15.98:0/648475176 shutdown_connections 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f682c0afb70 0x7f682c0b2080 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6840108920 0x7f68401a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6840103c20 0x7f684019a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 --2- 172.21.15.98:0/648475176 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68401029e0 0x7f684019a3e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 -- 172.21.15.98:0/648475176 >> 172.21.15.98:0/648475176 conn(0x7f68400fdbd0 msgr2=0x7f6840104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.379+0000 7f684851f640 1 -- 172.21.15.98:0/648475176 shutdown_connections 2024-04-25T16:34:17.385 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.380+0000 7f684851f640 1 -- 172.21.15.98:0/648475176 wait complete. 2024-04-25T16:34:17.391 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:17 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/725236323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:34:17.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:34:17.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:17 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/725236323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:34:17.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:17 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/725236323' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.524+0000 7f4805f0d640 1 -- 172.21.15.98:0/941258090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48001029e0 msgr2=0x7f48001036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.524+0000 7f4805f0d640 1 --2- 172.21.15.98:0/941258090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48001029e0 0x7f48001036e0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f47f000fb10 tx=0x7f47f002e570 comp rx=0 tx=0).stop 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.524+0000 7f4805f0d640 1 -- 172.21.15.98:0/941258090 shutdown_connections 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.524+0000 7f4805f0d640 1 --2- 172.21.15.98:0/941258090 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4800108920 0x7f480010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.524+0000 7f4805f0d640 1 --2- 172.21.15.98:0/941258090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4800103c20 0x7f4800104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.524+0000 7f4805f0d640 1 --2- 172.21.15.98:0/941258090 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f48001029e0 0x7f48001036e0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.524+0000 7f4805f0d640 1 -- 172.21.15.98:0/941258090 >> 172.21.15.98:0/941258090 conn(0x7f48000fdbd0 msgr2=0x7f4800100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.525+0000 7f4805f0d640 1 -- 172.21.15.98:0/941258090 shutdown_connections 2024-04-25T16:34:17.524 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.525+0000 7f4805f0d640 1 -- 172.21.15.98:0/941258090 wait complete. 2024-04-25T16:34:17.526 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.525+0000 7f4805f0d640 1 Processor -- start 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.525+0000 7f4805f0d640 1 -- start start 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f4805f0d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48001029e0 0x7f4800198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f4805f0d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4800103c20 0x7f48001987a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f4805f0d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800108920 0x7f480019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f4805f0d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f480010d2b0 con 0x7f4800108920 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f4805f0d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f480010d130 con 0x7f4800103c20 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f4805f0d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f480010d430 con 0x7f48001029e0 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800108920 0x7f480019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48001029e0 0x7f4800198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800108920 0x7f480019f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37506/0 (socket says 172.21.15.98:37506) 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 -- 172.21.15.98:0/3880671237 learned_addr learned my addr 172.21.15.98:0/3880671237 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47feffd640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4800103c20 0x7f48001987a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 -- 172.21.15.98:0/3880671237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48001029e0 msgr2=0x7f4800198260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.527 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48001029e0 0x7f4800198260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 -- 172.21.15.98:0/3880671237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4800103c20 msgr2=0x7f48001987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4800103c20 0x7f48001987a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 -- 172.21.15.98:0/3880671237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480019fda0 con 0x7f4800108920 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47feffd640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4800103c20 0x7f48001987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48001029e0 0x7f4800198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.526+0000 7f47fffff640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800108920 0x7f480019f730 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f47f800eb10 tx=0x7f47f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.527+0000 7f47fcff9640 1 -- 172.21.15.98:0/3880671237 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f8004b20 con 0x7f4800108920 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.527+0000 7f47fcff9640 1 -- 172.21.15.98:0/3880671237 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47f8004cc0 con 0x7f4800108920 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.527+0000 7f47fcff9640 1 -- 172.21.15.98:0/3880671237 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f8010690 con 0x7f4800108920 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.527+0000 7f4805f0d640 1 -- 172.21.15.98:0/3880671237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48001a0090 con 0x7f4800108920 2024-04-25T16:34:17.528 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.527+0000 7f4805f0d640 1 -- 172.21.15.98:0/3880671237 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48001a0630 con 0x7f4800108920 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.530+0000 7f47fcff9640 1 -- 172.21.15.98:0/3880671237 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f47f8010830 con 0x7f4800108920 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.530+0000 7f480cdc3740 1 Processor -- start 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.530+0000 7f480cdc3740 1 -- start start 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb42810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7774d0 0x562ebdb42d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb43a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f480cdc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ebdaa6d80 con 0x562ebd7774d0 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f480cdc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562ebda60a20 con 0x562ebd77f6b0 2024-04-25T16:34:17.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f480cdc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562ebdaa2d40 con 0x562ebd7a0390 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f47feffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb42810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7774d0 0x562ebdb42d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7774d0 0x562ebdb42d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37516/0 (socket says 172.21.15.98:37516) 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f47ff7fe640 1 -- 172.21.15.98:0/1125590828 learned_addr learned my addr 172.21.15.98:0/1125590828 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f47fcff9640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47e80afb70 0x7f47e80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f47feffd640 1 --2- 172.21.15.98:0/1125590828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb43a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.531+0000 7f47fffff640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47e80afb70 0x7f47e80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47feffd640 1 -- 172.21.15.98:0/1125590828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 msgr2=0x562ebdb43a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47feffd640 1 --2- 172.21.15.98:0/1125590828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb43a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47feffd640 1 -- 172.21.15.98:0/1125590828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7774d0 msgr2=0x562ebdb42d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47feffd640 1 --2- 172.21.15.98:0/1125590828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7774d0 0x562ebdb42d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47feffd640 1 -- 172.21.15.98:0/1125590828 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ebda6bc30 con 0x562ebd7a0390 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47fcff9640 1 -- 172.21.15.98:0/3880671237 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f47f80df540 con 0x7f4800108920 2024-04-25T16:34:17.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47feffd640 1 --2- 172.21.15.98:0/1125590828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb42810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f47f400bfc0 tx=0x7f47f4005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4394137c4effcc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47fffff640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47e80afb70 0x7f47e80b2080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f47f80a2f10 tx=0x7f47f80a2da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47e5ffb640 1 -- 172.21.15.98:0/1125590828 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f401f070 con 0x562ebd7a0390 2024-04-25T16:34:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47e5ffb640 1 -- 172.21.15.98:0/1125590828 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47f40054b0 con 0x562ebd7a0390 2024-04-25T16:34:17.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.532+0000 7f47e5ffb640 1 -- 172.21.15.98:0/1125590828 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4004ca0 con 0x562ebd7a0390 2024-04-25T16:34:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 -- 172.21.15.98:0/1125590828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 msgr2=0x562ebdb42810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 --2- 172.21.15.98:0/1125590828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb42810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f47f400bfc0 tx=0x7f47f4005f90 comp rx=0 tx=0).stop 2024-04-25T16:34:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 -- 172.21.15.98:0/1125590828 shutdown_connections 2024-04-25T16:34:17.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 --2- 172.21.15.98:0/1125590828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb43a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 --2- 172.21.15.98:0/1125590828 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7774d0 0x562ebdb42d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 --2- 172.21.15.98:0/1125590828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb42810 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 -- 172.21.15.98:0/1125590828 >> 172.21.15.98:0/1125590828 conn(0x562ebd7944e0 msgr2=0x562ebdb3e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 -- 172.21.15.98:0/1125590828 shutdown_connections 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 -- 172.21.15.98:0/1125590828 wait complete. 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.533+0000 7f480cdc3740 1 -- start start 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f480cdc3740 1 Processor -- start 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7774d0 0x562ebdb4c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb4cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb53a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f480cdc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ebdb44530 con 0x562ebd77f6b0 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f480cdc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562ebda2ddc0 con 0x562ebd7774d0 2024-04-25T16:34:17.538 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f480cdc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562ebdb446b0 con 0x562ebd7a0390 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f480cdc3740 10 client.0 _ll_register_callbacks cb 0x562ebda49600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb4cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f47feffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7774d0 0x562ebdb4c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f47feffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7774d0 0x562ebdb4c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52202/0 (socket says 172.21.15.98:52202) 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb4cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37522/0 (socket says 172.21.15.98:37522) 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f47feffd640 1 -- 172.21.15.98:0/3687406158 learned_addr learned my addr 172.21.15.98:0/3687406158 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.534+0000 7f47feffd640 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb53a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47feffd640 1 -- 172.21.15.98:0/3687406158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 msgr2=0x562ebdb53a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47feffd640 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb53a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47feffd640 1 -- 172.21.15.98:0/3687406158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 msgr2=0x562ebdb4cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47feffd640 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb4cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47feffd640 1 -- 172.21.15.98:0/3687406158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ebd7b83f0 con 0x562ebd7774d0 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47feffd640 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7774d0 0x562ebdb4c590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47f400bab0 tx=0x7f47f400d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=207f291b76cb576f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47deffd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:17.539 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47deffd640 1 -- 172.21.15.98:0/3687406158 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f401f030 con 0x562ebd7774d0 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f480cdc3740 1 -- 172.21.15.98:0/3687406158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f40054b0 con 0x562ebd7774d0 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47deffd640 1 -- 172.21.15.98:0/3687406158 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f47f4004230 con 0x562ebd7774d0 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f47deffd640 1 -- 172.21.15.98:0/3687406158 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47f4004430 con 0x562ebd7774d0 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f480cdc3740 10 client.25604 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.535+0000 7f480cdc3740 1 -- 172.21.15.98:0/3687406158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562ebd8827d0 con 0x562ebd7774d0 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f47deffd640 1 -- 172.21.15.98:0/3687406158 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47f40054b0 con 0x562ebd7774d0 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f47deffd640 1 -- 172.21.15.98:0/3687406158 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f47f4030030 con 0x562ebd7774d0 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f480cdc3740 10 client.25604 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f480cdc3740 10 client.25604 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f480cdc3740 20 client.25604 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562ebd75ef60 0x562ebdb59f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f480cdc3740 4 client.25604 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f480cdc3740 1 -- 172.21.15.98:0/3687406158 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ebdab4990 con 0x562ebd75ef60 2024-04-25T16:34:17.540 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.536+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562ebd75ef60 0x562ebdb59f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.537+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562ebd75ef60 0x562ebdb59f70 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=695ef6468d7af835 server_cookie=9ca51b9744daec84 in_seq=0 out_seq=0 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.537+0000 7f47deffd640 10 client.25604 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.537+0000 7f47deffd640 1 -- 172.21.15.98:0/3687406158 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562ebdab4990 con 0x562ebd75ef60 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.537+0000 7f47deffd640 10 client.25604 handle_command_reply: tid=0 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.537+0000 7f480cdc3740 1 client.25604 shutdown 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 -- 172.21.15.98:0/3687406158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7774d0 msgr2=0x562ebdb4c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7774d0 0x562ebdb4c590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47f400bab0 tx=0x7f47f400d040 comp rx=0 tx=0).stop 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 -- 172.21.15.98:0/3687406158 shutdown_connections 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562ebd75ef60 0x562ebdb59f70 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7a0390 0x562ebdb53a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f47deffd640 0 client.25604 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb4cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3687406158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7774d0 0x562ebdb4c590 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.541 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 -- 172.21.15.98:0/3687406158 >> 172.21.15.98:0/3687406158 conn(0x562ebd7944e0 msgr2=0x562ebdb3d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 -- 172.21.15.98:0/3687406158 shutdown_connections 2024-04-25T16:34:17.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 1 -- 172.21.15.98:0/3687406158 wait complete. 2024-04-25T16:34:17.542 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.538+0000 7f480cdc3740 20 client.25604 trim_cache size 0 max 16384 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f480cdc3740 1 Processor -- start 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f480cdc3740 1 -- start start 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb427d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb42d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7a0390 0x562ebdb47310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f480cdc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48001aa7d0 con 0x562ebd77f6b0 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f480cdc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f48001ac310 con 0x562ebd7a0390 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f480cdc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48001aaea0 con 0x562ebd7774d0 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.556+0000 7f47feffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb427d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:45964/0 (socket says 172.21.15.98:45964) 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 -- 172.21.15.98:0/3271662092 learned_addr learned my addr 172.21.15.98:0/3271662092 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb42d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.556 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7a0390 0x562ebdb47310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 -- 172.21.15.98:0/3271662092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7a0390 msgr2=0x562ebdb47310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7a0390 0x562ebdb47310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 -- 172.21.15.98:0/3271662092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 msgr2=0x562ebdb42d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb42d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.557 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 -- 172.21.15.98:0/3271662092 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ebdb4a680 con 0x562ebd7774d0 2024-04-25T16:34:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47feffd640 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb427d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005780 tx=0x7f47f4005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=189b6273a04c2c6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47e57fa640 1 -- 172.21.15.98:0/3271662092 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f401f070 con 0x562ebd7774d0 2024-04-25T16:34:17.562 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47e57fa640 1 -- 172.21.15.98:0/3271662092 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47f40037f0 con 0x562ebd7774d0 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.557+0000 7f47e57fa640 1 -- 172.21.15.98:0/3271662092 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4003e50 con 0x562ebd7774d0 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.558+0000 7f480cdc3740 1 -- 172.21.15.98:0/3271662092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 msgr2=0x562ebdb427d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.558+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb427d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005780 tx=0x7f47f4005970 comp rx=0 tx=0).stop 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.558+0000 7f480cdc3740 1 -- 172.21.15.98:0/3271662092 shutdown_connections 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.558+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd7a0390 0x562ebdb47310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.558+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd77f6b0 0x562ebdb42d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.558+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3271662092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb427d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.558+0000 7f480cdc3740 1 -- 172.21.15.98:0/3271662092 >> 172.21.15.98:0/3271662092 conn(0x562ebd7944e0 msgr2=0x562ebdb3da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 -- 172.21.15.98:0/3271662092 shutdown_connections 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 -- 172.21.15.98:0/3271662092 wait complete. 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 -- start start 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 Processor -- start 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb52cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.563 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb4f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7a0390 0x562ebdb4fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f48001abc40 con 0x562ebd7a0390 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f48001ab570 con 0x562ebd77f6b0 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.559+0000 7f480cdc3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f48001a87b0 con 0x562ebd7774d0 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f480cdc3740 10 client.0 _ll_register_callbacks cb 0x7f47f80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb4f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47feffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb52cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb4f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52220/0 (socket says 172.21.15.98:52220) 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 -- 172.21.15.98:0/3924049404 learned_addr learned my addr 172.21.15.98:0/3924049404 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47feffd640 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7a0390 0x562ebdb4fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 -- 172.21.15.98:0/3924049404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 msgr2=0x562ebdb52cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb52cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 -- 172.21.15.98:0/3924049404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7a0390 msgr2=0x562ebdb4fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7a0390 0x562ebdb4fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 -- 172.21.15.98:0/3924049404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ebdb502d0 con 0x562ebd77f6b0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47feffd640 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb52cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47ff7fe640 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb4f850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f47f0006010 tx=0x7f47f00041c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e4c878eab7e1f8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47deffd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.560+0000 7f47deffd640 1 -- 172.21.15.98:0/3924049404 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f000e9b0 con 0x562ebd77f6b0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 1 -- 172.21.15.98:0/3924049404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562ebdaae800 con 0x562ebd77f6b0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f47deffd640 1 -- 172.21.15.98:0/3924049404 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f47f000eb50 con 0x562ebd77f6b0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f47deffd640 1 -- 172.21.15.98:0/3924049404 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47f005f050 con 0x562ebd77f6b0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 10 client.25610 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 1 -- 172.21.15.98:0/3924049404 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562ebdbc11e0 con 0x562ebd77f6b0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f47deffd640 1 -- 172.21.15.98:0/3924049404 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x562ebdaae800 con 0x562ebd77f6b0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f47deffd640 1 -- 172.21.15.98:0/3924049404 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f47f004d390 con 0x562ebd77f6b0 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 10 client.25610 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 10 client.25610 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-25T16:34:17.565 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 20 client.25610 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562ebd7343a0 0x562ebdbc3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 4 client.25610 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f480cdc3740 1 -- 172.21.15.98:0/3924049404 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562ebda29610 con 0x562ebd7343a0 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.561+0000 7f47feffd640 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562ebd7343a0 0x562ebdbc3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.562+0000 7f47deffd640 10 client.25610 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.562+0000 7f47feffd640 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562ebd7343a0 0x562ebdbc3bb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aaaa74e20f182b94 server_cookie=9ab36024020ea0a2 in_seq=0 out_seq=0 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.562+0000 7f47deffd640 1 -- 172.21.15.98:0/3924049404 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562ebda29610 con 0x562ebd7343a0 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.562+0000 7f47deffd640 10 client.25610 handle_command_reply: tid=0 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.562+0000 7f480cdc3740 1 client.25610 shutdown 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 -- 172.21.15.98:0/3924049404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 msgr2=0x562ebdb4f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb4f850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f47f0006010 tx=0x7f47f00041c0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 -- 172.21.15.98:0/3924049404 shutdown_connections 2024-04-25T16:34:17.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x562ebd7343a0 0x562ebdbc3bb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebd7a0390 0x562ebdb4fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebd77f6b0 0x562ebdb4f850 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 --2- 172.21.15.98:0/3924049404 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebd7774d0 0x562ebdb52cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f47deffd640 0 client.25610 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 -- 172.21.15.98:0/3924049404 >> 172.21.15.98:0/3924049404 conn(0x562ebd7944e0 msgr2=0x562ebdb47310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 -- 172.21.15.98:0/3924049404 shutdown_connections 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 1 -- 172.21.15.98:0/3924049404 wait complete. 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.563+0000 7f480cdc3740 20 client.25610 trim_cache size 0 max 16384 2024-04-25T16:34:17.567 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 -- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47e80afb70 msgr2=0x7f47e80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47e80afb70 0x7f47e80b2080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f47f80a2f10 tx=0x7f47f80a2da0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 -- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800108920 msgr2=0x7f480019f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800108920 0x7f480019f730 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f47f800eb10 tx=0x7f47f800c2d0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.568 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 -- 172.21.15.98:0/3880671237 shutdown_connections 2024-04-25T16:34:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f47e80afb70 0x7f47e80b2080 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4800108920 0x7f480019f730 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4800103c20 0x7f48001987a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 --2- 172.21.15.98:0/3880671237 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f48001029e0 0x7f4800198260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 -- 172.21.15.98:0/3880671237 >> 172.21.15.98:0/3880671237 conn(0x7f48000fdbd0 msgr2=0x7f4800104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 -- 172.21.15.98:0/3880671237 shutdown_connections 2024-04-25T16:34:17.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.564+0000 7f4805f0d640 1 -- 172.21.15.98:0/3880671237 wait complete. 2024-04-25T16:34:17.576 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:34:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.706+0000 7fa54c59d640 1 -- 172.21.15.98:0/3478415310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544101e50 msgr2=0x7fa544106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.706+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3478415310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544101e50 0x7fa544106b10 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fa540009900 tx=0x7fa54002a410 comp rx=0 tx=0).stop 2024-04-25T16:34:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.706+0000 7fa54c59d640 1 -- 172.21.15.98:0/3478415310 shutdown_connections 2024-04-25T16:34:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.706+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3478415310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa544109770 0x7fa54410bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.706+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3478415310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa544102350 0x7fa5441090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.706+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3478415310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544101e50 0x7fa544106b10 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.706 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.706+0000 7fa54c59d640 1 -- 172.21.15.98:0/3478415310 >> 172.21.15.98:0/3478415310 conn(0x7fa5440fd340 msgr2=0x7fa5440ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.707+0000 7fa54c59d640 1 -- 172.21.15.98:0/3478415310 shutdown_connections 2024-04-25T16:34:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.707+0000 7fa54c59d640 1 -- 172.21.15.98:0/3478415310 wait complete. 2024-04-25T16:34:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.707+0000 7fa54c59d640 1 Processor -- start 2024-04-25T16:34:17.707 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.707+0000 7fa54c59d640 1 -- start start 2024-04-25T16:34:17.708 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54c59d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa544101e50 0x7fa54419a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54c59d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa544102350 0x7fa54419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54c59d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544109770 0x7fa5441a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54c59d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa54410f480 con 0x7fa544109770 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54c59d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa54410f300 con 0x7fa544101e50 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54c59d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa54410f600 con 0x7fa544102350 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa544101e50 0x7fa54419a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54ab3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544109770 0x7fa5441a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa544101e50 0x7fa54419a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52230/0 (socket says 172.21.15.98:52230) 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54ab3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544109770 0x7fa5441a1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37538/0 (socket says 172.21.15.98:37538) 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 -- 172.21.15.98:0/3664909666 learned_addr learned my addr 172.21.15.98:0/3664909666 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa549b38640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa544102350 0x7fa54419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 -- 172.21.15.98:0/3664909666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa544102350 msgr2=0x7fa54419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa544102350 0x7fa54419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.709 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 -- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544109770 msgr2=0x7fa5441a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544109770 0x7fa5441a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 -- 172.21.15.98:0/3664909666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5441a1c80 con 0x7fa544101e50 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.708+0000 7fa54a339640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa544101e50 0x7fa54419a140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa540009230 tx=0x7fa5400040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.709+0000 7fa53b7fe640 1 -- 172.21.15.98:0/3664909666 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa54003f030 con 0x7fa544101e50 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.709+0000 7fa54c59d640 1 -- 172.21.15.98:0/3664909666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5441a1f10 con 0x7fa544101e50 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.709+0000 7fa53b7fe640 1 -- 172.21.15.98:0/3664909666 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5400044c0 con 0x7fa544101e50 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.709+0000 7fa53b7fe640 1 -- 172.21.15.98:0/3664909666 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540035070 con 0x7fa544101e50 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.709+0000 7fa54c59d640 1 -- 172.21.15.98:0/3664909666 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5441a2370 con 0x7fa544101e50 2024-04-25T16:34:17.710 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.712+0000 7fa53b7fe640 1 -- 172.21.15.98:0/3664909666 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fa540042030 con 0x7fa544101e50 2024-04-25T16:34:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.712+0000 7fa55344d740 1 Processor -- start 2024-04-25T16:34:17.711 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.712+0000 7fa55344d740 1 -- start start 2024-04-25T16:34:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa55344d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160e9a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa55344d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160e9ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa55344d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160e9b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa55344d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562160dfed80 con 0x562160af8390 2024-04-25T16:34:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa55344d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562160db8a20 con 0x562160acf4d0 2024-04-25T16:34:17.712 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa55344d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562160dfad40 con 0x562160ad76b0 2024-04-25T16:34:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa53b7fe640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa5300afaf0 0x7fa5300b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa54ab3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160e9a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa54ab3a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160e9a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37548/0 (socket says 172.21.15.98:37548) 2024-04-25T16:34:17.713 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa54ab3a640 1 -- 172.21.15.98:0/2628863736 learned_addr learned my addr 172.21.15.98:0/2628863736 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa54a339640 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160e9ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa54ab3a640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa5300afaf0 0x7fa5300b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa549b38640 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160e9b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.713+0000 7fa53b7fe640 1 -- 172.21.15.98:0/3664909666 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa5400ff480 con 0x7fa544101e50 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa549b38640 1 -- 172.21.15.98:0/2628863736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 msgr2=0x562160e9ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa549b38640 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160e9ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa549b38640 1 -- 172.21.15.98:0/2628863736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 msgr2=0x562160e9a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa549b38640 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160e9a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa549b38640 1 -- 172.21.15.98:0/2628863736 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562160dc3c30 con 0x562160ad76b0 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa54ab3a640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa5300afaf0 0x7fa5300b2000 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa53c005c20 tx=0x7fa53c009bd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa549b38640 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160e9b960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa534009bf0 tx=0x7fa534011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9bdd68ed5af04a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa538bfe640 1 -- 172.21.15.98:0/2628863736 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534009da0 con 0x562160ad76b0 2024-04-25T16:34:17.714 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa538bfe640 1 -- 172.21.15.98:0/2628863736 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa5340044c0 con 0x562160ad76b0 2024-04-25T16:34:17.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.714+0000 7fa538bfe640 1 -- 172.21.15.98:0/2628863736 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534004a00 con 0x562160ad76b0 2024-04-25T16:34:17.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 -- 172.21.15.98:0/2628863736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 msgr2=0x562160e9b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160e9b960 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa534009bf0 tx=0x7fa534011810 comp rx=0 tx=0).stop 2024-04-25T16:34:17.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 -- 172.21.15.98:0/2628863736 shutdown_connections 2024-04-25T16:34:17.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160e9b960 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160e9ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 --2- 172.21.15.98:0/2628863736 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160e9a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 -- 172.21.15.98:0/2628863736 >> 172.21.15.98:0/2628863736 conn(0x562160aec4e0 msgr2=0x562160e96020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 -- 172.21.15.98:0/2628863736 shutdown_connections 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.715+0000 7fa55344d740 1 -- 172.21.15.98:0/2628863736 wait complete. 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 1 -- start start 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 1 Processor -- start 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160eacd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160ead280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160eb41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562160e9c520 con 0x562160af8390 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562160d85dc0 con 0x562160acf4d0 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562160e9c6a0 con 0x562160ad76b0 2024-04-25T16:34:17.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa55344d740 10 client.0 _ll_register_callbacks cb 0x562160da1600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa54a339640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160eacd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa549b38640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160ead280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa54a339640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160eacd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52254/0 (socket says 172.21.15.98:52254) 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa54a339640 1 -- 172.21.15.98:0/3076020117 learned_addr learned my addr 172.21.15.98:0/3076020117 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.716+0000 7fa54ab3a640 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160eb41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa549b38640 1 -- 172.21.15.98:0/3076020117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 msgr2=0x562160eacd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa549b38640 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160eacd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa549b38640 1 -- 172.21.15.98:0/3076020117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 msgr2=0x562160eb41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa549b38640 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160eb41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa549b38640 1 -- 172.21.15.98:0/3076020117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562160b103f0 con 0x562160ad76b0 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa54a339640 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160eacd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa549b38640 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160ead280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa53401b040 tx=0x7fa534019a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4133ffb674e35616 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa52dffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa52dffb640 1 -- 172.21.15.98:0/3076020117 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534029020 con 0x562160ad76b0 2024-04-25T16:34:17.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa52dffb640 1 -- 172.21.15.98:0/3076020117 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa534019c80 con 0x562160ad76b0 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa55344d740 1 -- 172.21.15.98:0/3076020117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562160e0c990 con 0x562160ad76b0 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa52dffb640 1 -- 172.21.15.98:0/3076020117 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa534034050 con 0x562160ad76b0 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa55344d740 10 client.25549 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.717+0000 7fa55344d740 1 -- 172.21.15.98:0/3076020117 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562160bda7d0 con 0x562160ad76b0 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa52dffb640 1 -- 172.21.15.98:0/3076020117 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x562160e0c990 con 0x562160ad76b0 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa52dffb640 1 -- 172.21.15.98:0/3076020117 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa534025c00 con 0x562160ad76b0 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa55344d740 10 client.25549 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa55344d740 10 client.25549 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa55344d740 20 client.25549 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa55344d740 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x562160ab6f60 0x562160eba8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa55344d740 4 client.25549 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa55344d740 1 -- 172.21.15.98:0/3076020117 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562160e06800 con 0x562160ab6f60 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.718+0000 7fa54a339640 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x562160ab6f60 0x562160eba8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.722 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.719+0000 7fa54a339640 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x562160ab6f60 0x562160eba8c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9bde0be9ae4a9b11 server_cookie=692baf5e415f7f4b in_seq=0 out_seq=0 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.719+0000 7fa52dffb640 10 client.25549 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.719+0000 7fa52dffb640 1 -- 172.21.15.98:0/3076020117 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562160e06800 con 0x562160ab6f60 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.719+0000 7fa52dffb640 10 client.25549 handle_command_reply: tid=0 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.719+0000 7fa55344d740 1 client.25549 shutdown 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 -- 172.21.15.98:0/3076020117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 msgr2=0x562160ead280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160ead280 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa53401b040 tx=0x7fa534019a90 comp rx=0 tx=0).stop 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 -- 172.21.15.98:0/3076020117 shutdown_connections 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x562160ab6f60 0x562160eba8c0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160af8390 0x562160eb41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160ead280 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 --2- 172.21.15.98:0/3076020117 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160acf4d0 0x562160eacd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 -- 172.21.15.98:0/3076020117 >> 172.21.15.98:0/3076020117 conn(0x562160aec4e0 msgr2=0x562160e98f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa52dffb640 0 client.25549 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 -- 172.21.15.98:0/3076020117 shutdown_connections 2024-04-25T16:34:17.723 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 1 -- 172.21.15.98:0/3076020117 wait complete. 2024-04-25T16:34:17.724 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.720+0000 7fa55344d740 20 client.25549 trim_cache size 0 max 16384 2024-04-25T16:34:17.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.736+0000 7fa55344d740 1 Processor -- start 2024-04-25T16:34:17.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.736+0000 7fa55344d740 1 -- start start 2024-04-25T16:34:17.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa55344d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160aec4e0 0x562160eaba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.736 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa55344d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa55344d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160ad76b0 0x562160e9cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa55344d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5441aa7e0 con 0x562160acf4d0 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa55344d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5441ac320 con 0x562160ad76b0 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa55344d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa5441aaeb0 con 0x562160aec4e0 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa549b38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa549b38640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37558/0 (socket says 172.21.15.98:37558) 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa54a339640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160aec4e0 0x562160eaba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa549b38640 1 -- 172.21.15.98:0/2304203062 learned_addr learned my addr 172.21.15.98:0/2304203062 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa54ab3a640 1 --2- 172.21.15.98:0/2304203062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160ad76b0 0x562160e9cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa54ab3a640 1 -- 172.21.15.98:0/2304203062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160aec4e0 msgr2=0x562160eaba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa54ab3a640 1 --2- 172.21.15.98:0/2304203062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160aec4e0 0x562160eaba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa54ab3a640 1 -- 172.21.15.98:0/2304203062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 msgr2=0x562160eb4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa54ab3a640 1 --2- 172.21.15.98:0/2304203062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.737 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.737+0000 7fa54ab3a640 1 -- 172.21.15.98:0/2304203062 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562160eb4640 con 0x562160ad76b0 2024-04-25T16:34:17.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.738+0000 7fa54ab3a640 1 --2- 172.21.15.98:0/2304203062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160ad76b0 0x562160e9cc90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa53c005bf0 tx=0x7fa53c01f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a7a06fcb91295b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.738+0000 7fa52ffff640 1 -- 172.21.15.98:0/2304203062 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c012e60 con 0x562160ad76b0 2024-04-25T16:34:17.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.738+0000 7fa52ffff640 1 -- 172.21.15.98:0/2304203062 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa53c005020 con 0x562160ad76b0 2024-04-25T16:34:17.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.738+0000 7fa52ffff640 1 -- 172.21.15.98:0/2304203062 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa53c029430 con 0x562160ad76b0 2024-04-25T16:34:17.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 -- 172.21.15.98:0/2304203062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160ad76b0 msgr2=0x562160e9cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 --2- 172.21.15.98:0/2304203062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160ad76b0 0x562160e9cc90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa53c005bf0 tx=0x7fa53c01f000 comp rx=0 tx=0).stop 2024-04-25T16:34:17.740 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 -- 172.21.15.98:0/2304203062 shutdown_connections 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 --2- 172.21.15.98:0/2304203062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160ad76b0 0x562160e9cc90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 --2- 172.21.15.98:0/2304203062 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 --2- 172.21.15.98:0/2304203062 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160aec4e0 0x562160eaba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 -- 172.21.15.98:0/2304203062 >> 172.21.15.98:0/2304203062 conn(0x562160ea75a0 msgr2=0x562160eab360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 -- 172.21.15.98:0/2304203062 shutdown_connections 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.739+0000 7fa55344d740 1 -- 172.21.15.98:0/2304203062 wait complete. 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 1 -- start start 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 1 Processor -- start 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160eb1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160aec4e0 0x562160eb2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa5441abc50 con 0x562160acf4d0 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa5441ab580 con 0x562160aec4e0 2024-04-25T16:34:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa5441a87c0 con 0x562160ad76b0 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa55344d740 10 client.0 _ll_register_callbacks cb 0x7fa53c0123f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa54a339640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa54a339640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37562/0 (socket says 172.21.15.98:37562) 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa54a339640 1 -- 172.21.15.98:0/1093582214 learned_addr learned my addr 172.21.15.98:0/1093582214 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa549b38640 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160eb1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa54ab3a640 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160aec4e0 0x562160eb2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.740+0000 7fa54a339640 1 -- 172.21.15.98:0/1093582214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 msgr2=0x562160eb1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa54a339640 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160eb1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa54a339640 1 -- 172.21.15.98:0/1093582214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160aec4e0 msgr2=0x562160eb2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa54a339640 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160aec4e0 0x562160eb2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa54a339640 1 -- 172.21.15.98:0/1093582214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562160eb2d20 con 0x562160acf4d0 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa54a339640 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb12f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa5400fcf40 tx=0x7fa540047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549f8ac21a20bf7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa52dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:17.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa52dffb640 1 -- 172.21.15.98:0/1093582214 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa540049040 con 0x562160acf4d0 2024-04-25T16:34:17.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa52dffb640 1 -- 172.21.15.98:0/1093582214 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa5400c3c50 con 0x562160acf4d0 2024-04-25T16:34:17.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa55344d740 1 -- 172.21.15.98:0/1093582214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa53c005020 con 0x562160acf4d0 2024-04-25T16:34:17.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa52dffb640 1 -- 172.21.15.98:0/1093582214 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa5400a7050 con 0x562160acf4d0 2024-04-25T16:34:17.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa55344d740 10 client.16107 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.741+0000 7fa55344d740 1 -- 172.21.15.98:0/1093582214 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562160eb4070 con 0x562160acf4d0 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa52dffb640 1 -- 172.21.15.98:0/1093582214 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa53c005020 con 0x562160acf4d0 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa52dffb640 1 -- 172.21.15.98:0/1093582214 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7fa5400cece0 con 0x562160acf4d0 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa55344d740 10 client.16107 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa55344d740 10 client.16107 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa55344d740 20 client.16107 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa55344d740 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x562160af8390 0x562160ea0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa55344d740 4 client.16107 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa55344d740 1 -- 172.21.15.98:0/1093582214 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562160d81610 con 0x562160af8390 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.742+0000 7fa549b38640 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x562160af8390 0x562160ea0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa52dffb640 10 client.16107 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa549b38640 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x562160af8390 0x562160ea0af0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1c71419e2d517225 server_cookie=5d4faccc3e75feed in_seq=0 out_seq=0 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa52dffb640 1 -- 172.21.15.98:0/1093582214 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562160d81610 con 0x562160af8390 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa52dffb640 10 client.16107 handle_command_reply: tid=0 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 client.16107 shutdown 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 -- 172.21.15.98:0/1093582214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 msgr2=0x562160eb12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb12f0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa5400fcf40 tx=0x7fa540047000 comp rx=0 tx=0).stop 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 -- 172.21.15.98:0/1093582214 shutdown_connections 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x562160af8390 0x562160ea0af0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa52dffb640 0 client.16107 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562160aec4e0 0x562160eb2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562160ad76b0 0x562160eb1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 --2- 172.21.15.98:0/1093582214 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562160acf4d0 0x562160eb12f0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 -- 172.21.15.98:0/1093582214 >> 172.21.15.98:0/1093582214 conn(0x562160ea75a0 msgr2=0x562160e98e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.743+0000 7fa55344d740 1 -- 172.21.15.98:0/1093582214 shutdown_connections 2024-04-25T16:34:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa55344d740 1 -- 172.21.15.98:0/1093582214 wait complete. 2024-04-25T16:34:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa55344d740 20 client.16107 trim_cache size 0 max 16384 2024-04-25T16:34:17.747 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 -- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa5300afaf0 msgr2=0x7fa5300b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa5300afaf0 0x7fa5300b2000 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa53c005c20 tx=0x7fa53c009bd0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 -- 172.21.15.98:0/3664909666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa544101e50 msgr2=0x7fa54419a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa544101e50 0x7fa54419a140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa540009230 tx=0x7fa5400040c0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 -- 172.21.15.98:0/3664909666 shutdown_connections 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa5300afaf0 0x7fa5300b2000 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa544109770 0x7fa5441a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa544102350 0x7fa54419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 --2- 172.21.15.98:0/3664909666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa544101e50 0x7fa54419a140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 -- 172.21.15.98:0/3664909666 >> 172.21.15.98:0/3664909666 conn(0x7fa5440fd340 msgr2=0x7fa544106780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 -- 172.21.15.98:0/3664909666 shutdown_connections 2024-04-25T16:34:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.744+0000 7fa54c59d640 1 -- 172.21.15.98:0/3664909666 wait complete. 2024-04-25T16:34:17.757 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:34:17.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.888+0000 7f499c46e640 1 -- 172.21.15.98:0/1155284020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994103c30 msgr2=0x7f4994104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.888+0000 7f499c46e640 1 --2- 172.21.15.98:0/1155284020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994103c30 0x7f4994104040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4984009860 tx=0x7f498402a340 comp rx=0 tx=0).stop 2024-04-25T16:34:17.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.889+0000 7f499c46e640 1 -- 172.21.15.98:0/1155284020 shutdown_connections 2024-04-25T16:34:17.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.889+0000 7f499c46e640 1 --2- 172.21.15.98:0/1155284020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4994108930 0x7f499410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.889+0000 7f499c46e640 1 --2- 172.21.15.98:0/1155284020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994103c30 0x7f4994104040 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.889+0000 7f499c46e640 1 --2- 172.21.15.98:0/1155284020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f49941029f0 0x7f49941036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.888 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.889+0000 7f499c46e640 1 -- 172.21.15.98:0/1155284020 >> 172.21.15.98:0/1155284020 conn(0x7f49940fdbe0 msgr2=0x7f4994100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.889+0000 7f499c46e640 1 -- 172.21.15.98:0/1155284020 shutdown_connections 2024-04-25T16:34:17.889 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.889+0000 7f499c46e640 1 -- 172.21.15.98:0/1155284020 wait complete. 2024-04-25T16:34:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499c46e640 1 Processor -- start 2024-04-25T16:34:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499c46e640 1 -- start start 2024-04-25T16:34:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499c46e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49941029f0 0x7f49941990d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499c46e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4994103c30 0x7f4994199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.891 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499c46e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994108930 0x7f49941a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499c46e640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f499410e3a0 con 0x7f4994103c30 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499c46e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f499410e220 con 0x7f49941029f0 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499c46e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f499410e520 con 0x7f4994108930 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499a20a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49941029f0 0x7f49941990d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499aa0b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994108930 0x7f49941a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499a20a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49941029f0 0x7f49941990d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52302/0 (socket says 172.21.15.98:52302) 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f499a20a640 1 -- 172.21.15.98:0/491898355 learned_addr learned my addr 172.21.15.98:0/491898355 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.890+0000 7f4999a09640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4994103c30 0x7f4994199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499aa0b640 1 -- 172.21.15.98:0/491898355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49941029f0 msgr2=0x7f49941990d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499aa0b640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49941029f0 0x7f49941990d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499aa0b640 1 -- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4994103c30 msgr2=0x7f4994199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499aa0b640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4994103c30 0x7f4994199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499aa0b640 1 -- 172.21.15.98:0/491898355 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499406b3e0 con 0x7f4994108930 2024-04-25T16:34:17.892 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499a20a640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49941029f0 0x7f49941990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499aa0b640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994108930 0x7f49941a05a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f498c00ec50 tx=0x7f498c00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f498b7fe640 1 -- 172.21.15.98:0/491898355 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c003b70 con 0x7f4994108930 2024-04-25T16:34:17.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f498b7fe640 1 -- 172.21.15.98:0/491898355 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f498c003d10 con 0x7f4994108930 2024-04-25T16:34:17.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499c46e640 1 -- 172.21.15.98:0/491898355 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499406b6d0 con 0x7f4994108930 2024-04-25T16:34:17.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f498b7fe640 1 -- 172.21.15.98:0/491898355 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498c0107a0 con 0x7f4994108930 2024-04-25T16:34:17.893 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.891+0000 7f499c46e640 1 -- 172.21.15.98:0/491898355 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f499406bbd0 con 0x7f4994108930 2024-04-25T16:34:17.895 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.894+0000 7f49a3404740 1 Processor -- start 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.894+0000 7f498b7fe640 1 -- 172.21.15.98:0/491898355 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f498c010940 con 0x7f4994108930 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f49a3404740 1 -- start start 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f49a3404740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a47620c390 0x55a4765ae810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f49a3404740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765aed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f49a3404740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765afa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f49a3404740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a476512d80 con 0x55a47620c390 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f49a3404740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a4764cca20 con 0x55a4761e34d0 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f49a3404740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a47650ed40 con 0x55a4761eb6b0 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f4999a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a47620c390 0x55a4765ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f4999a09640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a47620c390 0x55a4765ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37588/0 (socket says 172.21.15.98:37588) 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f4999a09640 1 -- 172.21.15.98:0/1452235811 learned_addr learned my addr 172.21.15.98:0/1452235811 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f499a20a640 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f498b7fe640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49800afaf0 0x7f49800b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.896 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.895+0000 7f4999a09640 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f499aa0b640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49800afaf0 0x7f49800b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f499a20a640 1 -- 172.21.15.98:0/1452235811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 msgr2=0x55a4765afa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f499a20a640 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765afa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f499a20a640 1 -- 172.21.15.98:0/1452235811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a47620c390 msgr2=0x55a4765ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f499a20a640 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a47620c390 0x55a4765ae810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f499a20a640 1 -- 172.21.15.98:0/1452235811 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4764d7c30 con 0x55a4761e34d0 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f4999a09640 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765afa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f498b7fe640 1 -- 172.21.15.98:0/491898355 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f498c0df530 con 0x7f4994108930 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f499a20a640 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765aed90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49900114c0 tx=0x7f4990011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d49a7b2901b0ff6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f499aa0b640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49800afaf0 0x7f49800b2000 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f498c0a36e0 tx=0x7f498c0d5db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f4988bfe640 1 -- 172.21.15.98:0/1452235811 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990004250 con 0x55a4761e34d0 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f4988bfe640 1 -- 172.21.15.98:0/1452235811 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f49900043f0 con 0x55a4761e34d0 2024-04-25T16:34:17.897 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.896+0000 7f4988bfe640 1 -- 172.21.15.98:0/1452235811 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49900035f0 con 0x55a4761e34d0 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 -- 172.21.15.98:0/1452235811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 msgr2=0x55a4765aed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765aed90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f49900114c0 tx=0x7f4990011980 comp rx=0 tx=0).stop 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 -- 172.21.15.98:0/1452235811 shutdown_connections 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765afa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765aed90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 --2- 172.21.15.98:0/1452235811 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a47620c390 0x55a4765ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 -- 172.21.15.98:0/1452235811 >> 172.21.15.98:0/1452235811 conn(0x55a4762004e0 msgr2=0x55a4765aa020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 -- 172.21.15.98:0/1452235811 shutdown_connections 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.897+0000 7f49a3404740 1 -- 172.21.15.98:0/1452235811 wait complete. 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 1 -- start start 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 1 Processor -- start 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765b8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761eb6b0 0x55a4765b8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a47620c390 0x55a4765bfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55a4765b0530 con 0x55a4761eb6b0 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55a476499dc0 con 0x55a4761e34d0 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55a4765b06b0 con 0x55a47620c390 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f49a3404740 10 client.0 _ll_register_callbacks cb 0x55a4764b5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f4999a09640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765b8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f499a20a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761eb6b0 0x55a4765b8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f4999a09640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765b8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52324/0 (socket says 172.21.15.98:52324) 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f4999a09640 1 -- 172.21.15.98:0/778698489 learned_addr learned my addr 172.21.15.98:0/778698489 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.898+0000 7f4999a09640 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a47620c390 0x55a4765bfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f499a20a640 1 -- 172.21.15.98:0/778698489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a47620c390 msgr2=0x55a4765bfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f499a20a640 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a47620c390 0x55a4765bfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f499a20a640 1 -- 172.21.15.98:0/778698489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 msgr2=0x55a4765b8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f499a20a640 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765b8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f499a20a640 1 -- 172.21.15.98:0/778698489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4762243f0 con 0x55a4761eb6b0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f4999a09640 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f499a20a640 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761eb6b0 0x55a4765b8ad0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4990005c80 tx=0x7f4990009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf36957db44981c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f497dffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f497dffb640 1 -- 172.21.15.98:0/778698489 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990028070 con 0x55a4761eb6b0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f497dffb640 1 -- 172.21.15.98:0/778698489 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f49900037f0 con 0x55a4761eb6b0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f49a3404740 1 -- 172.21.15.98:0/778698489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f49900043f0 con 0x55a4761eb6b0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f497dffb640 1 -- 172.21.15.98:0/778698489 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4990034050 con 0x55a4761eb6b0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f49a3404740 10 client.16119 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.899+0000 7f49a3404740 1 -- 172.21.15.98:0/778698489 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a4762ee7d0 con 0x55a4761eb6b0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f497dffb640 1 -- 172.21.15.98:0/778698489 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f49900043f0 con 0x55a4761eb6b0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f497dffb640 1 -- 172.21.15.98:0/778698489 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4990004c10 con 0x55a4761eb6b0 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f49a3404740 10 client.16119 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f49a3404740 10 client.16119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:17.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f49a3404740 20 client.16119 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f49a3404740 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55a4761caf60 0x55a4765c6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f49a3404740 4 client.16119 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f49a3404740 1 -- 172.21.15.98:0/778698489 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a476520990 con 0x55a4761caf60 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f4999a09640 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55a4761caf60 0x55a4765c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f497dffb640 10 client.16119 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.900+0000 7f4999a09640 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55a4761caf60 0x55a4765c6010 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=132e36aab155621c server_cookie=6c26fe6f377d65c in_seq=0 out_seq=0 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f497dffb640 1 -- 172.21.15.98:0/778698489 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a476520990 con 0x55a4761caf60 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f497dffb640 10 client.16119 handle_command_reply: tid=0 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 client.16119 shutdown 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 -- 172.21.15.98:0/778698489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761eb6b0 msgr2=0x55a4765b8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761eb6b0 0x55a4765b8ad0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f4990005c80 tx=0x7f4990009db0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 -- 172.21.15.98:0/778698489 shutdown_connections 2024-04-25T16:34:17.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55a4761caf60 0x55a4765c6010 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a47620c390 0x55a4765bfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761eb6b0 0x55a4765b8ad0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f497dffb640 0 client.16119 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 --2- 172.21.15.98:0/778698489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761e34d0 0x55a4765b8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 -- 172.21.15.98:0/778698489 >> 172.21.15.98:0/778698489 conn(0x55a4762004e0 msgr2=0x55a4765acf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 -- 172.21.15.98:0/778698489 shutdown_connections 2024-04-25T16:34:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 1 -- 172.21.15.98:0/778698489 wait complete. 2024-04-25T16:34:17.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.902+0000 7f49a3404740 20 client.16119 trim_cache size 0 max 16384 2024-04-25T16:34:17.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.921+0000 7f49a3404740 1 Processor -- start 2024-04-25T16:34:17.920 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f49a3404740 1 -- start start 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f49a3404740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761caf60 0x55a4765cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f49a3404740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f49a3404740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761eb6b0 0x55a4765cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f49a3404740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49941acab0 con 0x55a4761e34d0 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f49a3404740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49941ae5f0 con 0x55a4761eb6b0 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f49a3404740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f49941ad180 con 0x55a4761caf60 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f499a20a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f4999a09640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761caf60 0x55a4765cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.922 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f499a20a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37612/0 (socket says 172.21.15.98:37612) 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f4999a09640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761caf60 0x55a4765cb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:46046/0 (socket says 172.21.15.98:46046) 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f499a20a640 1 -- 172.21.15.98:0/2351829673 learned_addr learned my addr 172.21.15.98:0/2351829673 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f4999a09640 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761eb6b0 0x55a4765cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f499a20a640 1 -- 172.21.15.98:0/2351829673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761caf60 msgr2=0x55a4765cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f499a20a640 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761caf60 0x55a4765cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f499a20a640 1 -- 172.21.15.98:0/2351829673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761eb6b0 msgr2=0x55a4765cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f499a20a640 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761eb6b0 0x55a4765cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f499a20a640 1 -- 172.21.15.98:0/2351829673 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4765b0410 con 0x55a4761e34d0 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.922+0000 7f4999a09640 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761eb6b0 0x55a4765cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.923+0000 7f499a20a640 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bf950 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4990005150 tx=0x7f4990011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0dfafa1378b7dde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.923+0000 7f497ffff640 1 -- 172.21.15.98:0/2351829673 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4990028070 con 0x55a4761e34d0 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.923+0000 7f497ffff640 1 -- 172.21.15.98:0/2351829673 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4990009260 con 0x55a4761e34d0 2024-04-25T16:34:17.923 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.923+0000 7f497ffff640 1 -- 172.21.15.98:0/2351829673 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499001d040 con 0x55a4761e34d0 2024-04-25T16:34:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 -- 172.21.15.98:0/2351829673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 msgr2=0x55a4765bf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bf950 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f4990005150 tx=0x7f4990011980 comp rx=0 tx=0).stop 2024-04-25T16:34:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 -- 172.21.15.98:0/2351829673 shutdown_connections 2024-04-25T16:34:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761eb6b0 0x55a4765cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bf950 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 --2- 172.21.15.98:0/2351829673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761caf60 0x55a4765cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 -- 172.21.15.98:0/2351829673 >> 172.21.15.98:0/2351829673 conn(0x55a4762004e0 msgr2=0x55a4765b3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.928 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 -- 172.21.15.98:0/2351829673 shutdown_connections 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 -- 172.21.15.98:0/2351829673 wait complete. 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 -- start start 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 Processor -- start 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.924+0000 7f49a3404740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761caf60 0x55a4765bf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f49a3404740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f49a3404740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765be130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f49a3404740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f49941adf20 con 0x55a4761e34d0 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f49a3404740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f49941ad850 con 0x55a4761caf60 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f49a3404740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f49941aaa90 con 0x55a4761eb6b0 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f49a3404740 10 client.0 _ll_register_callbacks cb 0x7f4990005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f4999a09640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761caf60 0x55a4765bf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f499a20a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f499a20a640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37616/0 (socket says 172.21.15.98:37616) 2024-04-25T16:34:17.929 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f499a20a640 1 -- 172.21.15.98:0/3180261805 learned_addr learned my addr 172.21.15.98:0/3180261805 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f4999a09640 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765be130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f4999a09640 1 -- 172.21.15.98:0/3180261805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 msgr2=0x55a4765be130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f4999a09640 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765be130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f4999a09640 1 -- 172.21.15.98:0/3180261805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 msgr2=0x55a4765bdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f4999a09640 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f4999a09640 1 -- 172.21.15.98:0/3180261805 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4765be670 con 0x55a4761caf60 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f4999a09640 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761caf60 0x55a4765bf580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4984035040 tx=0x7f4984045040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48f31141f8191481 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f497dffb640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.925+0000 7f497dffb640 1 -- 172.21.15.98:0/3180261805 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984037040 con 0x55a4761caf60 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f497dffb640 1 -- 172.21.15.98:0/3180261805 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f498404e4c0 con 0x55a4761caf60 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 1 -- 172.21.15.98:0/3180261805 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a47651a800 con 0x55a4761caf60 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f497dffb640 1 -- 172.21.15.98:0/3180261805 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f498404e6c0 con 0x55a4761caf60 2024-04-25T16:34:17.930 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 10 client.25643 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 1 -- 172.21.15.98:0/3180261805 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a47662d050 con 0x55a4761caf60 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f497dffb640 1 -- 172.21.15.98:0/3180261805 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55a47651a800 con 0x55a4761caf60 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f497dffb640 1 -- 172.21.15.98:0/3180261805 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f4984003c80 con 0x55a4761caf60 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 10 client.25643 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 10 client.25643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 20 client.25643 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55a47620c390 0x55a47662fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 4 client.25643 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.926+0000 7f49a3404740 1 -- 172.21.15.98:0/3180261805 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a476495610 con 0x55a47620c390 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.927+0000 7f499a20a640 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55a47620c390 0x55a47662fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.927+0000 7f497dffb640 10 client.25643 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.927+0000 7f499a20a640 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55a47620c390 0x55a47662fa40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e18a85f0149840dc server_cookie=f1986c3e55770f85 in_seq=0 out_seq=0 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.927+0000 7f497dffb640 1 -- 172.21.15.98:0/3180261805 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a476495610 con 0x55a47620c390 2024-04-25T16:34:17.931 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.927+0000 7f497dffb640 10 client.25643 handle_command_reply: tid=0 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.927+0000 7f49a3404740 1 client.25643 shutdown 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 -- 172.21.15.98:0/3180261805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761caf60 msgr2=0x55a4765bf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761caf60 0x55a4765bf580 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4984035040 tx=0x7f4984045040 comp rx=0 tx=0).stop 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 -- 172.21.15.98:0/3180261805 shutdown_connections 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55a47620c390 0x55a47662fa40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a4761eb6b0 0x55a4765be130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f497dffb640 0 client.25643 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55a4761e34d0 0x55a4765bdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 --2- 172.21.15.98:0/3180261805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55a4761caf60 0x55a4765bf580 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 -- 172.21.15.98:0/3180261805 >> 172.21.15.98:0/3180261805 conn(0x55a4762004e0 msgr2=0x55a4765ace20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 -- 172.21.15.98:0/3180261805 shutdown_connections 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 1 -- 172.21.15.98:0/3180261805 wait complete. 2024-04-25T16:34:17.932 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f49a3404740 20 client.25643 trim_cache size 0 max 16384 2024-04-25T16:34:17.933 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:17.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 -- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49800afaf0 msgr2=0x7f49800b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49800afaf0 0x7f49800b2000 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f498c0a36e0 tx=0x7f498c0d5db0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 -- 172.21.15.98:0/491898355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994108930 msgr2=0x7f49941a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994108930 0x7f49941a05a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f498c00ec50 tx=0x7f498c00c6a0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 -- 172.21.15.98:0/491898355 shutdown_connections 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f49800afaf0 0x7f49800b2000 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4994108930 0x7f49941a05a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4994103c30 0x7f4994199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.928+0000 7f499c46e640 1 --2- 172.21.15.98:0/491898355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49941029f0 0x7f49941990d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.929+0000 7f499c46e640 1 -- 172.21.15.98:0/491898355 >> 172.21.15.98:0/491898355 conn(0x7f49940fdbe0 msgr2=0x7f4994104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.929+0000 7f499c46e640 1 -- 172.21.15.98:0/491898355 shutdown_connections 2024-04-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:17.929+0000 7f499c46e640 1 -- 172.21.15.98:0/491898355 wait complete. 2024-04-25T16:34:18.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:18 smithi203 ceph-mon[33115]: pgmap v410: 129 pgs: 129 active+clean; 138 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 11 MiB/s wr, 75 op/s 2024-04-25T16:34:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:18 smithi098 ceph-mon[29162]: pgmap v410: 129 pgs: 129 active+clean; 138 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 11 MiB/s wr, 75 op/s 2024-04-25T16:34:18.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:18 smithi173 ceph-mon[32556]: pgmap v410: 129 pgs: 129 active+clean; 138 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 11 MiB/s wr, 75 op/s 2024-04-25T16:34:18.941 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:34:18.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:34:18.941 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.082+0000 7f116bfd7640 1 -- 172.21.15.98:0/4181531606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164103c30 msgr2=0x7f1164104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.082+0000 7f116bfd7640 1 --2- 172.21.15.98:0/4181531606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164103c30 0x7f1164104040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1160009860 tx=0x7f116002a340 comp rx=0 tx=0).stop 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.082+0000 7f116bfd7640 1 -- 172.21.15.98:0/4181531606 shutdown_connections 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.082+0000 7f116bfd7640 1 --2- 172.21.15.98:0/4181531606 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1164108930 0x7f116410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.082+0000 7f116bfd7640 1 --2- 172.21.15.98:0/4181531606 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164103c30 0x7f1164104040 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.082+0000 7f116bfd7640 1 --2- 172.21.15.98:0/4181531606 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f11641029f0 0x7f11641036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.082+0000 7f116bfd7640 1 -- 172.21.15.98:0/4181531606 >> 172.21.15.98:0/4181531606 conn(0x7f11640fdc00 msgr2=0x7f1164100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.083+0000 7f116bfd7640 1 -- 172.21.15.98:0/4181531606 shutdown_connections 2024-04-25T16:34:19.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.083+0000 7f116bfd7640 1 -- 172.21.15.98:0/4181531606 wait complete. 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.083+0000 7f116bfd7640 1 Processor -- start 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.083+0000 7f116bfd7640 1 -- start start 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.083+0000 7f116bfd7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11641029f0 0x7f116419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.083+0000 7f116bfd7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1164103c30 0x7f116419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116bfd7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164108930 0x7f11641a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116bfd7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f116410f4c0 con 0x7f1164103c30 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116bfd7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f116410f340 con 0x7f11641029f0 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116bfd7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f116410f640 con 0x7f1164108930 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f1169572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1164103c30 0x7f116419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f1169572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1164103c30 0x7f116419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37632/0 (socket says 172.21.15.98:37632) 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f1169572640 1 -- 172.21.15.98:0/329700535 learned_addr learned my addr 172.21.15.98:0/329700535 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f1169d73640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11641029f0 0x7f116419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.085 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116a574640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164108930 0x7f11641a15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116a574640 1 -- 172.21.15.98:0/329700535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11641029f0 msgr2=0x7f116419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116a574640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11641029f0 0x7f116419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116a574640 1 -- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1164103c30 msgr2=0x7f116419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116a574640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1164103c30 0x7f116419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116a574640 1 -- 172.21.15.98:0/329700535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11641a1c40 con 0x7f1164108930 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f1169572640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1164103c30 0x7f116419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f116a574640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164108930 0x7f11641a15d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f115400e970 tx=0x7f115400ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.084+0000 7f115affd640 1 -- 172.21.15.98:0/329700535 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115400ccc0 con 0x7f1164108930 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.085+0000 7f115affd640 1 -- 172.21.15.98:0/329700535 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1154005020 con 0x7f1164108930 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.085+0000 7f115affd640 1 -- 172.21.15.98:0/329700535 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115400ce60 con 0x7f1164108930 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.085+0000 7f116bfd7640 1 -- 172.21.15.98:0/329700535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11641a1f30 con 0x7f1164108930 2024-04-25T16:34:19.086 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.085+0000 7f116bfd7640 1 -- 172.21.15.98:0/329700535 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11641a23f0 con 0x7f1164108930 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.087+0000 7f115affd640 1 -- 172.21.15.98:0/329700535 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f1154004a00 con 0x7f1164108930 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1172f63740 1 Processor -- start 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1172f63740 1 -- start start 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1172f63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcf6410 0x56385e0974a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1172f63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e097a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1172f63740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e09e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1172f63740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56385dffcb00 con 0x56385dcd5730 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1172f63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56385df98920 con 0x56385dcf6410 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1172f63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56385dff8ac0 con 0x56385dccd550 2024-04-25T16:34:19.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1169572640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcf6410 0x56385e0974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1169572640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcf6410 0x56385e0974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52384/0 (socket says 172.21.15.98:52384) 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1169572640 1 -- 172.21.15.98:0/3897047525 learned_addr learned my addr 172.21.15.98:0/3897047525 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1169572640 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e09e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f115affd640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f11500afb70 0x7f11500b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.088+0000 7f1169d73640 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e097a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f116a574640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f11500afb70 0x7f11500b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f1169572640 1 -- 172.21.15.98:0/3897047525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 msgr2=0x56385e097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f1169572640 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e097a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f1169572640 1 -- 172.21.15.98:0/3897047525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 msgr2=0x56385e09e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f1169572640 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e09e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f115affd640 1 -- 172.21.15.98:0/329700535 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f11540deea0 con 0x7f1164108930 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f1169572640 1 -- 172.21.15.98:0/3897047525 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385ddd67e0 con 0x56385dcf6410 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f1169572640 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcf6410 0x56385e0974a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1160005760 tx=0x7f11600056d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e45f0ebc280832bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f116a574640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f11500afb70 0x7f11500b2080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f115400e940 tx=0x7f115406d990 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f114ffff640 1 -- 172.21.15.98:0/3897047525 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160037040 con 0x56385dcf6410 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f114ffff640 1 -- 172.21.15.98:0/3897047525 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11600044d0 con 0x56385dcf6410 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.089+0000 7f114ffff640 1 -- 172.21.15.98:0/3897047525 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160032b30 con 0x56385dcf6410 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.090+0000 7f1172f63740 1 -- 172.21.15.98:0/3897047525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcf6410 msgr2=0x56385e0974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.090+0000 7f1172f63740 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcf6410 0x56385e0974a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1160005760 tx=0x7f11600056d0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.090+0000 7f1172f63740 1 -- 172.21.15.98:0/3897047525 shutdown_connections 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.090+0000 7f1172f63740 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e09e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.090+0000 7f1172f63740 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e097a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.090+0000 7f1172f63740 1 --2- 172.21.15.98:0/3897047525 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcf6410 0x56385e0974a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.090+0000 7f1172f63740 1 -- 172.21.15.98:0/3897047525 >> 172.21.15.98:0/3897047525 conn(0x56385dcea560 msgr2=0x56385e092e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.091+0000 7f1172f63740 1 -- 172.21.15.98:0/3897047525 shutdown_connections 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.091+0000 7f1172f63740 1 -- 172.21.15.98:0/3897047525 wait complete. 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.091+0000 7f1172f63740 1 -- start start 2024-04-25T16:34:19.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.091+0000 7f1172f63740 1 Processor -- start 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1172f63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1172f63740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e0a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1172f63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcf6410 0x56385e0a8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1172f63740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x56385e09f310 con 0x56385dcd5730 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1172f63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x56385df965d0 con 0x56385dccd550 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1172f63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x56385e09f490 con 0x56385dcf6410 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1172f63740 10 client.0 _ll_register_callbacks cb 0x56385df84080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169d73640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e0a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169572640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169d73640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e0a17e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37652/0 (socket says 172.21.15.98:37652) 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169d73640 1 -- 172.21.15.98:0/3109007753 learned_addr learned my addr 172.21.15.98:0/3109007753 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169572640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52396/0 (socket says 172.21.15.98:52396) 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169572640 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcf6410 0x56385e0a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169572640 1 -- 172.21.15.98:0/3109007753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcf6410 msgr2=0x56385e0a8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169572640 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcf6410 0x56385e0a8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169572640 1 -- 172.21.15.98:0/3109007753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 msgr2=0x56385e0a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169572640 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e0a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169572640 1 -- 172.21.15.98:0/3109007753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385de36f70 con 0x56385dccd550 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.092+0000 7f1169d73640 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e0a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f1169572640 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a12a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1160005760 tx=0x7f116002adb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eee95b19f9166ab7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f114d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:19.097 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f114d7fa640 1 -- 172.21.15.98:0/3109007753 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1160053060 con 0x56385dccd550 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f1172f63740 1 -- 172.21.15.98:0/3109007753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11600044d0 con 0x56385dccd550 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f114d7fa640 1 -- 172.21.15.98:0/3109007753 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f11600328d0 con 0x56385dccd550 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f114d7fa640 1 -- 172.21.15.98:0/3109007753 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f116005a050 con 0x56385dccd550 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f1172f63740 10 client.25655 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f1172f63740 1 -- 172.21.15.98:0/3109007753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56385dcce170 con 0x56385dccd550 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f114d7fa640 1 -- 172.21.15.98:0/3109007753 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f11600044d0 con 0x56385dccd550 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f114d7fa640 1 -- 172.21.15.98:0/3109007753 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f116003d040 con 0x56385dccd550 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f1172f63740 10 client.25655 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f1172f63740 10 client.25655 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f1172f63740 20 client.25655 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.093+0000 7f1172f63740 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56385dcb51f0 0x56385e0ae800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.094+0000 7f1172f63740 4 client.25655 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:19.098 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.094+0000 7f1172f63740 1 -- 172.21.15.98:0/3109007753 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56385e00a710 con 0x56385dcb51f0 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.094+0000 7f1169d73640 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56385dcb51f0 0x56385e0ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.094+0000 7f114d7fa640 10 client.25655 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.094+0000 7f1169d73640 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56385dcb51f0 0x56385e0ae800 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd186a2ca76a5ff server_cookie=8e1a74fd1d85a570 in_seq=0 out_seq=0 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f114d7fa640 1 -- 172.21.15.98:0/3109007753 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56385e00a710 con 0x56385dcb51f0 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f114d7fa640 10 client.25655 handle_command_reply: tid=0 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 client.25655 shutdown 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 -- 172.21.15.98:0/3109007753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 msgr2=0x56385e0a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a12a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1160005760 tx=0x7f116002adb0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 -- 172.21.15.98:0/3109007753 shutdown_connections 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56385dcb51f0 0x56385e0ae800 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcf6410 0x56385e0a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcd5730 0x56385e0a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f114d7fa640 0 client.25655 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:19.099 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 --2- 172.21.15.98:0/3109007753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a12a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 -- 172.21.15.98:0/3109007753 >> 172.21.15.98:0/3109007753 conn(0x56385dcea560 msgr2=0x56385e091ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 -- 172.21.15.98:0/3109007753 shutdown_connections 2024-04-25T16:34:19.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 1 -- 172.21.15.98:0/3109007753 wait complete. 2024-04-25T16:34:19.100 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.096+0000 7f1172f63740 20 client.25655 trim_cache size 0 max 16384 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.120+0000 7f1172f63740 1 Processor -- start 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.120+0000 7f1172f63740 1 -- start start 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1172f63740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e09ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1172f63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1172f63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcd5730 0x56385e0a1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1172f63740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11641ac880 con 0x56385dcb51f0 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1172f63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11641ae3c0 con 0x56385dccd550 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1172f63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f11641acf50 con 0x56385dcd5730 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e09ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.121 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e09ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37666/0 (socket says 172.21.15.98:37666) 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169572640 1 -- 172.21.15.98:0/3485761160 learned_addr learned my addr 172.21.15.98:0/3485761160 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169d73640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169d73640 1 -- 172.21.15.98:0/3485761160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcd5730 msgr2=0x56385e0a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169d73640 1 --2- 172.21.15.98:0/3485761160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcd5730 0x56385e0a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169d73640 1 -- 172.21.15.98:0/3485761160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 msgr2=0x56385e09ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169d73640 1 --2- 172.21.15.98:0/3485761160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e09ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169d73640 1 -- 172.21.15.98:0/3485761160 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385e09c1b0 con 0x56385dccd550 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.121+0000 7f1169572640 1 --2- 172.21.15.98:0/3485761160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e09ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.122+0000 7f1169d73640 1 --2- 172.21.15.98:0/3485761160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a86d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x56385e0aedf0 tx=0x7f115c010040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15a4190aba403608 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.122+0000 7f114f7fe640 1 -- 172.21.15.98:0/3485761160 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c028070 con 0x56385dccd550 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.122+0000 7f114f7fe640 1 -- 172.21.15.98:0/3485761160 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f115c01ad60 con 0x56385dccd550 2024-04-25T16:34:19.122 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.122+0000 7f114f7fe640 1 -- 172.21.15.98:0/3485761160 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c023e60 con 0x56385dccd550 2024-04-25T16:34:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 -- 172.21.15.98:0/3485761160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 msgr2=0x56385e0a86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 --2- 172.21.15.98:0/3485761160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a86d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x56385e0aedf0 tx=0x7f115c010040 comp rx=0 tx=0).stop 2024-04-25T16:34:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 -- 172.21.15.98:0/3485761160 shutdown_connections 2024-04-25T16:34:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 --2- 172.21.15.98:0/3485761160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dcd5730 0x56385e0a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.126 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 --2- 172.21.15.98:0/3485761160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dccd550 0x56385e0a86d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 --2- 172.21.15.98:0/3485761160 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e09ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 -- 172.21.15.98:0/3485761160 >> 172.21.15.98:0/3485761160 conn(0x56385dcea560 msgr2=0x56385e0a0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 -- 172.21.15.98:0/3485761160 shutdown_connections 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 -- 172.21.15.98:0/3485761160 wait complete. 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.123+0000 7f1172f63740 1 -- start start 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1172f63740 1 Processor -- start 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1172f63740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e0a7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1172f63740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e0a7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1172f63740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcd5730 0x56385e114a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1172f63740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f11641adcf0 con 0x56385dcb51f0 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1172f63740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f11641ad620 con 0x56385dcd5730 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1172f63740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f11641aa860 con 0x56385dccd550 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1172f63740 10 client.0 _ll_register_callbacks cb 0x7f1160004340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:19.127 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.124+0000 7f1169572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e0a7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169d73640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e0a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169572640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e0a7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37668/0 (socket says 172.21.15.98:37668) 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169572640 1 -- 172.21.15.98:0/493278029 learned_addr learned my addr 172.21.15.98:0/493278029 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169572640 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcd5730 0x56385e114a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169d73640 1 -- 172.21.15.98:0/493278029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcd5730 msgr2=0x56385e114a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169d73640 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcd5730 0x56385e114a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169d73640 1 -- 172.21.15.98:0/493278029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 msgr2=0x56385e0a7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169d73640 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e0a7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169d73640 1 -- 172.21.15.98:0/493278029 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385e0a7fb0 con 0x56385dccd550 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169572640 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e0a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f1169d73640 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e0a7760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f115c0098d0 tx=0x7f115c01ab60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10acbb01b5a025b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.125+0000 7f114d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f114d7fa640 1 -- 172.21.15.98:0/493278029 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f115c030070 con 0x56385dccd550 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f1172f63740 1 -- 172.21.15.98:0/493278029 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f115c01ad60 con 0x56385dccd550 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f114d7fa640 1 -- 172.21.15.98:0/493278029 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f115c023400 con 0x56385dccd550 2024-04-25T16:34:19.128 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f114d7fa640 1 -- 172.21.15.98:0/493278029 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f115c03c050 con 0x56385dccd550 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f1172f63740 10 client.25588 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f1172f63740 1 -- 172.21.15.98:0/493278029 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56385e115fc0 con 0x56385dccd550 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f114d7fa640 1 -- 172.21.15.98:0/493278029 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f115c01ad60 con 0x56385dccd550 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f114d7fa640 1 -- 172.21.15.98:0/493278029 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7f115c00b040 con 0x56385dccd550 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f1172f63740 10 client.25588 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f1172f63740 10 client.25588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.126+0000 7f1172f63740 20 client.25588 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.127+0000 7f1172f63740 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56385dcf6410 0x56385e118910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.127+0000 7f1172f63740 4 client.25588 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.127+0000 7f1172f63740 1 -- 172.21.15.98:0/493278029 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56385e004580 con 0x56385dcf6410 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.127+0000 7f1169572640 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56385dcf6410 0x56385e118910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.127+0000 7f114d7fa640 10 client.25588 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:19.129 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.127+0000 7f1169572640 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56385dcf6410 0x56385e118910 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a92b8d3c3005361 server_cookie=f9c4c4dde3a25482 in_seq=0 out_seq=0 2024-04-25T16:34:19.130 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f114d7fa640 1 -- 172.21.15.98:0/493278029 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56385e004580 con 0x56385dcf6410 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f114d7fa640 10 client.25588 handle_command_reply: tid=0 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 client.25588 shutdown 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 -- 172.21.15.98:0/493278029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 msgr2=0x56385e0a7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e0a7760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f115c0098d0 tx=0x7f115c01ab60 comp rx=0 tx=0).stop 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 -- 172.21.15.98:0/493278029 shutdown_connections 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x56385dcf6410 0x56385e118910 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56385dcd5730 0x56385e114a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56385dccd550 0x56385e0a7760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 --2- 172.21.15.98:0/493278029 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56385dcb51f0 0x56385e0a7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f114d7fa640 0 client.25588 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 -- 172.21.15.98:0/493278029 >> 172.21.15.98:0/493278029 conn(0x56385dcea560 msgr2=0x56385e095c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 -- 172.21.15.98:0/493278029 shutdown_connections 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.128+0000 7f1172f63740 1 -- 172.21.15.98:0/493278029 wait complete. 2024-04-25T16:34:19.131 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f1172f63740 20 client.25588 trim_cache size 0 max 16384 2024-04-25T16:34:19.132 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:34:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-04-25T16:34:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "21671286-4083-4d78-8e8a-68844e389427", 2024-04-25T16:34:19.132 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-04-25T16:34:19.132 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 -- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f11500afb70 msgr2=0x7f11500b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f11500afb70 0x7f11500b2080 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f115400e940 tx=0x7f115406d990 comp rx=0 tx=0).stop 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 -- 172.21.15.98:0/329700535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164108930 msgr2=0x7f11641a15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164108930 0x7f11641a15d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f115400e970 tx=0x7f115400ee30 comp rx=0 tx=0).stop 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 -- 172.21.15.98:0/329700535 shutdown_connections 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f11500afb70 0x7f11500b2080 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1164108930 0x7f11641a15d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1164103c30 0x7f116419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 --2- 172.21.15.98:0/329700535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11641029f0 0x7f116419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 -- 172.21.15.98:0/329700535 >> 172.21.15.98:0/329700535 conn(0x7f11640fdc00 msgr2=0x7f1164104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 -- 172.21.15.98:0/329700535 shutdown_connections 2024-04-25T16:34:19.134 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.129+0000 7f116bfd7640 1 -- 172.21.15.98:0/329700535 wait complete. 2024-04-25T16:34:19.140 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:34:19.141 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:34:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 -- 172.21.15.98:0/1033563993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff268108930 msgr2=0x7ff26810ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/1033563993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff268108930 0x7ff26810ad50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff260009be0 tx=0x7ff26002a6c0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 -- 172.21.15.98:0/1033563993 shutdown_connections 2024-04-25T16:34:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/1033563993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff268108930 0x7ff26810ad50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/1033563993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff268103c30 0x7ff268104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/1033563993 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff2681029f0 0x7ff2681036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 -- 172.21.15.98:0/1033563993 >> 172.21.15.98:0/1033563993 conn(0x7ff2680fdbe0 msgr2=0x7ff268100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 -- 172.21.15.98:0/1033563993 shutdown_connections 2024-04-25T16:34:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.278+0000 7ff26d3a8640 1 -- 172.21.15.98:0/1033563993 wait complete. 2024-04-25T16:34:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.279+0000 7ff26d3a8640 1 Processor -- start 2024-04-25T16:34:19.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.279+0000 7ff26d3a8640 1 -- start start 2024-04-25T16:34:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.279+0000 7ff26d3a8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2681029f0 0x7ff26806bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.281 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.279+0000 7ff26d3a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff268103c30 0x7ff26819ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.279+0000 7ff26d3a8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268108930 0x7ff2681a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff25ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff268103c30 0x7ff26819ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff26d3a8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff26810e3d0 con 0x7ff268108930 2024-04-25T16:34:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff26d3a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff26810e250 con 0x7ff268103c30 2024-04-25T16:34:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff26d3a8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff26810e550 con 0x7ff2681029f0 2024-04-25T16:34:19.282 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff25ffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff268103c30 0x7ff26819ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:52436/0 (socket says 172.21.15.98:52436) 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff25ffff640 1 -- 172.21.15.98:0/2923746507 learned_addr learned my addr 172.21.15.98:0/2923746507 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff266ffd640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2681029f0 0x7ff26806bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff2677fe640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268108930 0x7ff2681a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff266ffd640 1 -- 172.21.15.98:0/2923746507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff268103c30 msgr2=0x7ff26819ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff266ffd640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff268103c30 0x7ff26819ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff266ffd640 1 -- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268108930 msgr2=0x7ff2681a1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff266ffd640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268108930 0x7ff2681a1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff266ffd640 1 -- 172.21.15.98:0/2923746507 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2681a1c80 con 0x7ff2681029f0 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.280+0000 7ff266ffd640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2681029f0 0x7ff26806bd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff2681035f0 tx=0x7ff25800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.283 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.281+0000 7ff264ff9640 1 -- 172.21.15.98:0/2923746507 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258004440 con 0x7ff2681029f0 2024-04-25T16:34:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.281+0000 7ff264ff9640 1 -- 172.21.15.98:0/2923746507 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff25800cd50 con 0x7ff2681029f0 2024-04-25T16:34:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.281+0000 7ff264ff9640 1 -- 172.21.15.98:0/2923746507 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff258019480 con 0x7ff2681029f0 2024-04-25T16:34:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.281+0000 7ff26d3a8640 1 -- 172.21.15.98:0/2923746507 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff268198d40 con 0x7ff2681029f0 2024-04-25T16:34:19.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.281+0000 7ff26d3a8640 1 -- 172.21.15.98:0/2923746507 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff268199240 con 0x7ff2681029f0 2024-04-25T16:34:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.284+0000 7ff274257740 1 Processor -- start 2024-04-25T16:34:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.284+0000 7ff274257740 1 -- start start 2024-04-25T16:34:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff274257740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff274257740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680d2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff274257740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680d3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.286 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff274257740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b468037d80 con 0x55b467d106b0 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff274257740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b467ff18b0 con 0x55b467d084d0 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff274257740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b468033d40 con 0x55b467d31390 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff2677fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff2677fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680d2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:46108/0 (socket says 172.21.15.98:46108) 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff264ff9640 1 -- 172.21.15.98:0/2923746507 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7ff258019620 con 0x7ff2681029f0 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff2677fe640 1 -- 172.21.15.98:0/3393216366 learned_addr learned my addr 172.21.15.98:0/3393216366 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.285+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680d3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff266ffd640 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680d2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff2677fe640 1 -- 172.21.15.98:0/3393216366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 msgr2=0x55b4680d2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680d2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff2677fe640 1 -- 172.21.15.98:0/3393216366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 msgr2=0x55b4680d3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680d3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff2677fe640 1 -- 172.21.15.98:0/3393216366 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b467fc2a40 con 0x55b467d31390 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680d3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:19.287 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff266ffd640 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680d2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:34:19.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.286+0000 7ff264ff9640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff2540b4000 0x7ff2540b6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.287+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680d2a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff260009be0 tx=0x7ff260030fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30268b27a45a249b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.287+0000 7ff266ffd640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff2540b4000 0x7ff2540b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.287+0000 7ff25d7fa640 1 -- 172.21.15.98:0/3393216366 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260041030 con 0x55b467d31390 2024-04-25T16:34:19.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.287+0000 7ff264ff9640 1 -- 172.21.15.98:0/2923746507 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7ff2580df340 con 0x7ff2681029f0 2024-04-25T16:34:19.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.287+0000 7ff25d7fa640 1 -- 172.21.15.98:0/3393216366 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff260037040 con 0x55b467d31390 2024-04-25T16:34:19.288 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.287+0000 7ff25d7fa640 1 -- 172.21.15.98:0/3393216366 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260036040 con 0x55b467d31390 2024-04-25T16:34:19.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 -- 172.21.15.98:0/3393216366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 msgr2=0x55b4680d2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff266ffd640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff2540b4000 0x7ff2540b6510 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff2580a3490 tx=0x7ff25806d930 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680d2a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff260009be0 tx=0x7ff260030fe0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 -- 172.21.15.98:0/3393216366 shutdown_connections 2024-04-25T16:34:19.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680d3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680d2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 --2- 172.21.15.98:0/3393216366 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680d2a60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 -- 172.21.15.98:0/3393216366 >> 172.21.15.98:0/3393216366 conn(0x55b467d254e0 msgr2=0x55b4680ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 -- 172.21.15.98:0/3393216366 shutdown_connections 2024-04-25T16:34:19.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 -- 172.21.15.98:0/3393216366 wait complete. 2024-04-25T16:34:19.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 -- start start 2024-04-25T16:34:19.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.288+0000 7ff274257740 1 Processor -- start 2024-04-25T16:34:19.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff274257740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 0x55b4680dc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff274257740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680dce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff274257740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 0x55b4680e3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff274257740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55b4680d4720 con 0x55b467d106b0 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff274257740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55b467fbed10 con 0x55b467d31390 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff274257740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55b4680d48a0 con 0x55b467d084d0 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff274257740 10 client.0 _ll_register_callbacks cb 0x55b467feb210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff25ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680dce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff25ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680dce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37706/0 (socket says 172.21.15.98:37706) 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff25ffff640 1 -- 172.21.15.98:0/3095868112 learned_addr learned my addr 172.21.15.98:0/3095868112 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 0x55b4680dc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.289+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 0x55b4680e3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff25ffff640 1 -- 172.21.15.98:0/3095868112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 msgr2=0x55b4680dc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 0x55b4680dc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff25ffff640 1 -- 172.21.15.98:0/3095868112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 msgr2=0x55b4680e3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 0x55b4680e3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.295 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff25ffff640 1 -- 172.21.15.98:0/3095868112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b467e137d0 con 0x55b467d106b0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680dce00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff250012700 tx=0x7ff250012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd4f841429961eed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff24e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3095868112 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff25000f9a0 con 0x55b467d106b0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff274257740 1 -- 172.21.15.98:0/3095868112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff260037040 con 0x55b467d106b0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3095868112 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7ff25000fb40 con 0x55b467d106b0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3095868112 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff25000fd40 con 0x55b467d106b0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff274257740 10 client.16155 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.290+0000 7ff274257740 1 -- 172.21.15.98:0/3095868112 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b467d493f0 con 0x55b467d106b0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3095868112 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff260037040 con 0x55b467d106b0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3095868112 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff25001c420 con 0x55b467d106b0 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff274257740 10 client.16155 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff274257740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:19.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff274257740 20 client.16155 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff274257740 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b467ceff60 0x55b4680ea240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff274257740 4 client.16155 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff274257740 1 -- 172.21.15.98:0/3095868112 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b468045990 con 0x55b467ceff60 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.291+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b467ceff60 0x55b4680ea240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.292+0000 7ff24e7fc640 10 client.16155 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.292+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b467ceff60 0x55b4680ea240 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c480abcd251eea4f server_cookie=b76e3b176e1ce4a3 in_seq=0 out_seq=0 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3095868112 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b468045990 con 0x55b467ceff60 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff24e7fc640 10 client.16155 handle_command_reply: tid=0 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 client.16155 shutdown 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 -- 172.21.15.98:0/3095868112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 msgr2=0x55b4680dce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680dce00 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff250012700 tx=0x7ff250012bc0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 -- 172.21.15.98:0/3095868112 shutdown_connections 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b467ceff60 0x55b4680ea240 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 0x55b4680e3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467d106b0 0x55b4680dce00 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 --2- 172.21.15.98:0/3095868112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 0x55b4680dc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 -- 172.21.15.98:0/3095868112 >> 172.21.15.98:0/3095868112 conn(0x55b467d254e0 msgr2=0x55b4680d1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff24e7fc640 0 client.16155 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 -- 172.21.15.98:0/3095868112 shutdown_connections 2024-04-25T16:34:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 1 -- 172.21.15.98:0/3095868112 wait complete. 2024-04-25T16:34:19.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.293+0000 7ff274257740 20 client.16155 trim_cache size 0 max 16384 2024-04-25T16:34:19.318 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.317+0000 7ff274257740 1 Processor -- start 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.317+0000 7ff274257740 1 -- start start 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff274257740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 0x55b4680cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff274257740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 0x55b4680ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff274257740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680d1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff274257740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2681aa790 con 0x55b467cc53a0 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff274257740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff2681ac2d0 con 0x55b467d31390 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff274257740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2681aae60 con 0x55b467d084d0 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680d1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37720/0 (socket says 172.21.15.98:37720) 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 -- 172.21.15.98:0/4095271465 learned_addr learned my addr 172.21.15.98:0/4095271465 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff25ffff640 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 0x55b4680ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 0x55b4680cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 -- 172.21.15.98:0/4095271465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 msgr2=0x55b4680cdc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.319 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 0x55b4680cdc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 -- 172.21.15.98:0/4095271465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 msgr2=0x55b4680ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 0x55b4680ce150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 -- 172.21.15.98:0/4095271465 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b467d090f0 con 0x55b467cc53a0 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.318+0000 7ff2677fe640 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680d1070 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff260005c40 tx=0x7ff26002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78216a88d32eaa12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.319+0000 7ff25cff9640 1 -- 172.21.15.98:0/4095271465 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260036040 con 0x55b467cc53a0 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.319+0000 7ff25cff9640 1 -- 172.21.15.98:0/4095271465 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff260003ab0 con 0x55b467cc53a0 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.319+0000 7ff25cff9640 1 -- 172.21.15.98:0/4095271465 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260041040 con 0x55b467cc53a0 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 -- 172.21.15.98:0/4095271465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 msgr2=0x55b4680d1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680d1070 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff260005c40 tx=0x7ff26002a6c0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 -- 172.21.15.98:0/4095271465 shutdown_connections 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680d1070 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d31390 0x55b4680ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 --2- 172.21.15.98:0/4095271465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d084d0 0x55b4680cdc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 -- 172.21.15.98:0/4095271465 >> 172.21.15.98:0/4095271465 conn(0x55b467d254e0 msgr2=0x55b4680daf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 -- 172.21.15.98:0/4095271465 shutdown_connections 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.320+0000 7ff274257740 1 -- 172.21.15.98:0/4095271465 wait complete. 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.321+0000 7ff274257740 1 -- start start 2024-04-25T16:34:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.321+0000 7ff274257740 1 Processor -- start 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.321+0000 7ff274257740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680e08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.321+0000 7ff274257740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680e0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.321+0000 7ff274257740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680e1d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff274257740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ff2681abc00 con 0x55b467cc53a0 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff274257740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ff2681ab530 con 0x55b467d084d0 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680e08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff274257740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ff2681a8770 con 0x55b467d31390 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680e08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:37722/0 (socket says 172.21.15.98:37722) 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 -- 172.21.15.98:0/3217388961 learned_addr learned my addr 172.21.15.98:0/3217388961 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff274257740 10 client.0 _ll_register_callbacks cb 0x7ff250009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680e0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 -- 172.21.15.98:0/3217388961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 msgr2=0x55b4680e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680e1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 -- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 msgr2=0x55b4680e0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680e0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 -- 172.21.15.98:0/3217388961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b467ff84b0 con 0x55b467cc53a0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680e0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff2677fe640 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680e08f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ff260009560 tx=0x7ff260004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a68c62cdfb40567 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff24e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.322+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3217388961 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260041070 con 0x55b467cc53a0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.323+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3217388961 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7ff260004b40 con 0x55b467cc53a0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.323+0000 7ff274257740 1 -- 172.21.15.98:0/3217388961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff260003ab0 con 0x55b467cc53a0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.323+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3217388961 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff260036040 con 0x55b467cc53a0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.323+0000 7ff274257740 10 client.16161 fetch_fsmap learned FSMap version 29 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.323+0000 7ff274257740 1 -- 172.21.15.98:0/3217388961 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b4680e33e0 con 0x55b467cc53a0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.323+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3217388961 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff260003ab0 con 0x55b467cc53a0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.324+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3217388961 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 29) ==== 8362+0+0 (secure 0 0 0) 0x7ff26003c040 con 0x55b467cc53a0 2024-04-25T16:34:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.324+0000 7ff274257740 10 client.16161 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:34:19.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.324+0000 7ff274257740 10 client.16161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:19.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.324+0000 7ff274257740 20 client.16161 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:19.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.324+0000 7ff274257740 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b4681525b0 0x55b4681549d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:19.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.324+0000 7ff274257740 4 client.16161 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:34:19.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.324+0000 7ff274257740 1 -- 172.21.15.98:0/3217388961 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b46803f800 con 0x55b4681525b0 2024-04-25T16:34:19.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.324+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b4681525b0 0x55b4681549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:19.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.325+0000 7ff24e7fc640 10 client.16161 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:19.327 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.325+0000 7ff25ffff640 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b4681525b0 0x55b4681549d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e361533087cf309 server_cookie=a768f7eb9b37e57c in_seq=0 out_seq=0 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.326+0000 7ff24e7fc640 1 -- 172.21.15.98:0/3217388961 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55b46803f800 con 0x55b4681525b0 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.326+0000 7ff24e7fc640 10 client.16161 handle_command_reply: tid=0 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.326+0000 7ff274257740 1 client.16161 shutdown 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.326+0000 7ff274257740 1 -- 172.21.15.98:0/3217388961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 msgr2=0x55b4680e08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.326+0000 7ff274257740 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680e08f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7ff260009560 tx=0x7ff260004630 comp rx=0 tx=0).stop 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.326+0000 7ff274257740 1 -- 172.21.15.98:0/3217388961 shutdown_connections 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.326+0000 7ff274257740 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55b4681525b0 0x55b4681549d0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.326+0000 7ff274257740 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55b467d31390 0x55b4680e1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.327+0000 7ff24e7fc640 0 client.16161 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.327+0000 7ff274257740 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55b467d084d0 0x55b4680e0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.327+0000 7ff274257740 1 --2- 172.21.15.98:0/3217388961 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55b467cc53a0 0x55b4680e08f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.327+0000 7ff274257740 1 -- 172.21.15.98:0/3217388961 >> 172.21.15.98:0/3217388961 conn(0x55b467d254e0 msgr2=0x55b4680cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.329 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.327+0000 7ff274257740 1 -- 172.21.15.98:0/3217388961 shutdown_connections 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.327+0000 7ff274257740 1 -- 172.21.15.98:0/3217388961 wait complete. 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.327+0000 7ff274257740 20 client.16161 trim_cache size 0 max 16384 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (439 inodes in the stack)", 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stdout: "21671286-4083-4d78-8e8a-68844e389427": { 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "21671286-4083-4d78-8e8a-68844e389427", 2024-04-25T16:34:19.330 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-04-25T16:34:19.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:34:19.331 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:34:19.331 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:34:19.331 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 -- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff2540b4000 msgr2=0x7ff2540b6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff2540b4000 0x7ff2540b6510 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff2580a3490 tx=0x7ff25806d930 comp rx=0 tx=0).stop 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 -- 172.21.15.98:0/2923746507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2681029f0 msgr2=0x7ff26806bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2681029f0 0x7ff26806bd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff2681035f0 tx=0x7ff25800ee50 comp rx=0 tx=0).stop 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 -- 172.21.15.98:0/2923746507 shutdown_connections 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ff2540b4000 0x7ff2540b6510 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ff268108930 0x7ff2681a1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff268103c30 0x7ff26819ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 --2- 172.21.15.98:0/2923746507 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff2681029f0 0x7ff26806bd20 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff2681035f0 tx=0x7ff25800ee50 comp rx=0 tx=0).stop 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 -- 172.21.15.98:0/2923746507 >> 172.21.15.98:0/2923746507 conn(0x7ff2680fdbe0 msgr2=0x7ff268104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 -- 172.21.15.98:0/2923746507 shutdown_connections 2024-04-25T16:34:19.332 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:19.328+0000 7ff26d3a8640 1 -- 172.21.15.98:0/2923746507 wait complete. 2024-04-25T16:34:19.342 INFO:tasks.cephfs.filesystem:scrub status for tag:21671286-4083-4d78-8e8a-68844e389427 - {'path': '/', 'tag': '21671286-4083-4d78-8e8a-68844e389427', 'options': 'recursive,force'} 2024-04-25T16:34:20.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:19 smithi098 ceph-mon[29162]: scrub queued for path: / 2024-04-25T16:34:20.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:19 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:20.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:19 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:20.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:19 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:20.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:19 smithi098 ceph-mon[29162]: scrub summary: idle 2024-04-25T16:34:20.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:19 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:34:20.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:19 smithi173 ceph-mon[32556]: scrub queued for path: / 2024-04-25T16:34:20.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:19 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:20.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:19 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:20.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:19 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:20.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:19 smithi173 ceph-mon[32556]: scrub summary: idle 2024-04-25T16:34:20.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:19 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:34:20.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:19 smithi203 ceph-mon[33115]: scrub queued for path: / 2024-04-25T16:34:20.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:19 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:20.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:19 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:20.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:19 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:20.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:19 smithi203 ceph-mon[33115]: scrub summary: idle 2024-04-25T16:34:20.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:19 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:34:21.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:20 smithi098 ceph-mon[29162]: pgmap v411: 129 pgs: 129 active+clean; 142 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.1 MiB/s wr, 122 op/s 2024-04-25T16:34:21.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:20 smithi173 ceph-mon[32556]: pgmap v411: 129 pgs: 129 active+clean; 142 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.1 MiB/s wr, 122 op/s 2024-04-25T16:34:21.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:20 smithi203 ceph-mon[33115]: pgmap v411: 129 pgs: 129 active+clean; 142 MiB data, 706 MiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 8.1 MiB/s wr, 122 op/s 2024-04-25T16:34:22.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:22 smithi203 ceph-mon[33115]: pgmap v412: 129 pgs: 129 active+clean; 145 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 4.5 MiB/s wr, 127 op/s 2024-04-25T16:34:22.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:22 smithi098 ceph-mon[29162]: pgmap v412: 129 pgs: 129 active+clean; 145 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 4.5 MiB/s wr, 127 op/s 2024-04-25T16:34:22.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:22 smithi173 ceph-mon[32556]: pgmap v412: 129 pgs: 129 active+clean; 145 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 4.5 MiB/s wr, 127 op/s 2024-04-25T16:34:24.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:24 smithi203 ceph-mon[33115]: pgmap v413: 129 pgs: 129 active+clean; 150 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-04-25T16:34:24.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:24 smithi098 ceph-mon[29162]: pgmap v413: 129 pgs: 129 active+clean; 150 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-04-25T16:34:24.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:24 smithi173 ceph-mon[32556]: pgmap v413: 129 pgs: 129 active+clean; 150 MiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 4.5 MiB/s wr, 193 op/s 2024-04-25T16:34:26.332 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:26 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:34:26.332 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:26 smithi173 ceph-mon[32556]: pgmap v414: 129 pgs: 129 active+clean; 150 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-25T16:34:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:26 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:34:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:26 smithi203 ceph-mon[33115]: pgmap v414: 129 pgs: 129 active+clean; 150 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-25T16:34:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:26 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:34:26.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:26 smithi098 ceph-mon[29162]: pgmap v414: 129 pgs: 129 active+clean; 150 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 1.5 MiB/s wr, 192 op/s 2024-04-25T16:34:28.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:28 smithi203 ceph-mon[33115]: pgmap v415: 129 pgs: 129 active+clean; 155 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-04-25T16:34:28.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:28 smithi098 ceph-mon[29162]: pgmap v415: 129 pgs: 129 active+clean; 155 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-04-25T16:34:28.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:28 smithi173 ceph-mon[32556]: pgmap v415: 129 pgs: 129 active+clean; 155 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.9 MiB/s wr, 258 op/s 2024-04-25T16:34:30.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:30 smithi203 ceph-mon[33115]: pgmap v416: 129 pgs: 129 active+clean; 159 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-04-25T16:34:30.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:30 smithi098 ceph-mon[29162]: pgmap v416: 129 pgs: 129 active+clean; 159 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-04-25T16:34:30.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:30 smithi173 ceph-mon[32556]: pgmap v416: 129 pgs: 129 active+clean; 159 MiB data, 779 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.9 MiB/s wr, 276 op/s 2024-04-25T16:34:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:32 smithi203 ceph-mon[33115]: pgmap v417: 129 pgs: 129 active+clean; 159 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.6 MiB/s wr, 233 op/s 2024-04-25T16:34:32.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:32 smithi098 ceph-mon[29162]: pgmap v417: 129 pgs: 129 active+clean; 159 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.6 MiB/s wr, 233 op/s 2024-04-25T16:34:32.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:32 smithi173 ceph-mon[32556]: pgmap v417: 129 pgs: 129 active+clean; 159 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 1.6 MiB/s wr, 233 op/s 2024-04-25T16:34:34.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:34 smithi203 ceph-mon[33115]: pgmap v418: 129 pgs: 129 active+clean; 164 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 1.8 MiB/s wr, 324 op/s 2024-04-25T16:34:34.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:34 smithi098 ceph-mon[29162]: pgmap v418: 129 pgs: 129 active+clean; 164 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 1.8 MiB/s wr, 324 op/s 2024-04-25T16:34:34.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:34 smithi173 ceph-mon[32556]: pgmap v418: 129 pgs: 129 active+clean; 164 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 1.8 MiB/s wr, 324 op/s 2024-04-25T16:34:36.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:36 smithi203 ceph-mon[33115]: pgmap v419: 129 pgs: 129 active+clean; 168 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.7 MiB/s wr, 283 op/s 2024-04-25T16:34:36.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:36 smithi098 ceph-mon[29162]: pgmap v419: 129 pgs: 129 active+clean; 168 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.7 MiB/s wr, 283 op/s 2024-04-25T16:34:36.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:36 smithi173 ceph-mon[32556]: pgmap v419: 129 pgs: 129 active+clean; 168 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 1.7 MiB/s wr, 283 op/s 2024-04-25T16:34:38.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:38 smithi203 ceph-mon[33115]: pgmap v420: 129 pgs: 129 active+clean; 174 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 348 op/s 2024-04-25T16:34:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:38 smithi098 ceph-mon[29162]: pgmap v420: 129 pgs: 129 active+clean; 174 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 348 op/s 2024-04-25T16:34:38.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:38 smithi173 ceph-mon[32556]: pgmap v420: 129 pgs: 129 active+clean; 174 MiB data, 840 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 348 op/s 2024-04-25T16:34:40.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:40 smithi203 ceph-mon[33115]: pgmap v421: 129 pgs: 129 active+clean; 176 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 2.0 MiB/s wr, 331 op/s 2024-04-25T16:34:40.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:40 smithi098 ceph-mon[29162]: pgmap v421: 129 pgs: 129 active+clean; 176 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 2.0 MiB/s wr, 331 op/s 2024-04-25T16:34:40.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:40 smithi173 ceph-mon[32556]: pgmap v421: 129 pgs: 129 active+clean; 176 MiB data, 862 MiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 2.0 MiB/s wr, 331 op/s 2024-04-25T16:34:42.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:42 smithi203 ceph-mon[33115]: pgmap v422: 129 pgs: 129 active+clean; 180 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 284 op/s 2024-04-25T16:34:42.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:42 smithi098 ceph-mon[29162]: pgmap v422: 129 pgs: 129 active+clean; 180 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 284 op/s 2024-04-25T16:34:42.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:42 smithi173 ceph-mon[32556]: pgmap v422: 129 pgs: 129 active+clean; 180 MiB data, 879 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.0 MiB/s wr, 284 op/s 2024-04-25T16:34:44.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:44 smithi203 ceph-mon[33115]: pgmap v423: 129 pgs: 129 active+clean; 186 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 356 op/s 2024-04-25T16:34:44.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:44 smithi098 ceph-mon[29162]: pgmap v423: 129 pgs: 129 active+clean; 186 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 356 op/s 2024-04-25T16:34:44.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:44 smithi173 ceph-mon[32556]: pgmap v423: 129 pgs: 129 active+clean; 186 MiB data, 900 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 356 op/s 2024-04-25T16:34:46.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:46 smithi203 ceph-mon[33115]: pgmap v424: 129 pgs: 129 active+clean; 188 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-04-25T16:34:46.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:46 smithi098 ceph-mon[29162]: pgmap v424: 129 pgs: 129 active+clean; 188 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-04-25T16:34:46.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:46 smithi173 ceph-mon[32556]: pgmap v424: 129 pgs: 129 active+clean; 188 MiB data, 915 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.3 MiB/s wr, 278 op/s 2024-04-25T16:34:48.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:48 smithi203 ceph-mon[33115]: pgmap v425: 129 pgs: 129 active+clean; 200 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 323 op/s 2024-04-25T16:34:48.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:48 smithi098 ceph-mon[29162]: pgmap v425: 129 pgs: 129 active+clean; 200 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 323 op/s 2024-04-25T16:34:48.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:48 smithi173 ceph-mon[32556]: pgmap v425: 129 pgs: 129 active+clean; 200 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 323 op/s 2024-04-25T16:34:49.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:34:49.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:34:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.457+0000 7f860cb06640 1 -- 172.21.15.98:0/1634808935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 msgr2=0x7f8608103370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.457+0000 7f860cb06640 1 --2- 172.21.15.98:0/1634808935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 0x7f8608103370 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009900 tx=0x7f85f802a410 comp rx=0 tx=0).stop 2024-04-25T16:34:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 -- 172.21.15.98:0/1634808935 shutdown_connections 2024-04-25T16:34:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 --2- 172.21.15.98:0/1634808935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8608104460 0x7f860810abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 --2- 172.21.15.98:0/1634808935 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8608103940 0x7f8608103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.459 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 --2- 172.21.15.98:0/1634808935 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 0x7f8608103370 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 -- 172.21.15.98:0/1634808935 >> 172.21.15.98:0/1634808935 conn(0x7f86080fd340 msgr2=0x7f86080ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 -- 172.21.15.98:0/1634808935 shutdown_connections 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 -- 172.21.15.98:0/1634808935 wait complete. 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 Processor -- start 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.458+0000 7f860cb06640 1 -- start start 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860cb06640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 0x7f8608102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860cb06640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8608103940 0x7f860819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860cb06640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8608104460 0x7f86081a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860cb06640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f860810e390 con 0x7f8608102f60 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860cb06640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f860810e210 con 0x7f8608104460 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860cb06640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f860810e510 con 0x7f8608103940 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 0x7f8608102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 0x7f8608102200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50482/0 (socket says 172.21.15.98:50482) 2024-04-25T16:34:49.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 -- 172.21.15.98:0/1174208608 learned_addr learned my addr 172.21.15.98:0/1174208608 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f8606d9d640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8608104460 0x7f86081a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 -- 172.21.15.98:0/1174208608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8608103940 msgr2=0x7f860819ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8608103940 0x7f860819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 -- 172.21.15.98:0/1174208608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8608104460 msgr2=0x7f86081a1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8608104460 0x7f86081a1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 -- 172.21.15.98:0/1174208608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86081a1c70 con 0x7f8608102f60 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f8606d9d640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8608104460 0x7f86081a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860659c640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 0x7f8608102200 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f85f8031460 tx=0x7f85f8005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f85f77fe640 1 -- 172.21.15.98:0/1174208608 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f803f030 con 0x7f8608102f60 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f85f77fe640 1 -- 172.21.15.98:0/1174208608 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f85f80043d0 con 0x7f8608102f60 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860cb06640 1 -- 172.21.15.98:0/1174208608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86081a1f00 con 0x7f8608102f60 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f85f77fe640 1 -- 172.21.15.98:0/1174208608 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8035070 con 0x7f8608102f60 2024-04-25T16:34:49.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.459+0000 7f860cb06640 1 -- 172.21.15.98:0/1174208608 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86081a2360 con 0x7f8608102f60 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f85f77fe640 1 -- 172.21.15.98:0/1174208608 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f85f8042030 con 0x7f8608102f60 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f86139ba740 1 Processor -- start 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f86139ba740 1 -- start start 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f86139ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fe4390 0x561822385a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f86139ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x561822385fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f86139ba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fc36b0 0x561822386c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f86139ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5618222ead80 con 0x561821fbb4d0 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f86139ba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x5618222a48b0 con 0x561821fc36b0 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f86139ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x5618222e6d40 con 0x561821fe4390 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f860659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x561822385fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f860659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x561822385fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50492/0 (socket says 172.21.15.98:50492) 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f860659c640 1 -- 172.21.15.98:0/1952963752 learned_addr learned my addr 172.21.15.98:0/1952963752 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:49.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f8606d9d640 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fe4390 0x561822385a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.462+0000 7f8605d9b640 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fc36b0 0x561822386c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f85f77fe640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85f00afaf0 0x7f85f00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f860659c640 1 -- 172.21.15.98:0/1952963752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fe4390 msgr2=0x561822385a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f860659c640 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fe4390 0x561822385a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f860659c640 1 -- 172.21.15.98:0/1952963752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fc36b0 msgr2=0x561822386c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f860659c640 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fc36b0 0x561822386c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f860659c640 1 -- 172.21.15.98:0/1952963752 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561822275a40 con 0x561821fbb4d0 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f8605d9b640 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fc36b0 0x561822386c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f8606d9d640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85f00afaf0 0x7f85f00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f85f77fe640 1 -- 172.21.15.98:0/1174208608 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f85f80ff450 con 0x7f8608102f60 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f860659c640 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x561822385fe0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f85f8093de0 tx=0x7f85f8091da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30cb4e4923d4b173 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f85f4ff9640 1 -- 172.21.15.98:0/1952963752 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f80c6ed0 con 0x561821fbb4d0 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f85f4ff9640 1 -- 172.21.15.98:0/1952963752 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f85f8094310 con 0x561821fbb4d0 2024-04-25T16:34:49.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f8606d9d640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85f00afaf0 0x7f85f00b2000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x561822386ab0 tx=0x7f8600004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.463+0000 7f85f4ff9640 1 -- 172.21.15.98:0/1952963752 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f80c6cf0 con 0x561821fbb4d0 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 -- 172.21.15.98:0/1952963752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 msgr2=0x561822385fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x561822385fe0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f85f8093de0 tx=0x7f85f8091da0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 -- 172.21.15.98:0/1952963752 shutdown_connections 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fc36b0 0x561822386c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x561822385fe0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 --2- 172.21.15.98:0/1952963752 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fe4390 0x561822385a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 -- 172.21.15.98:0/1952963752 >> 172.21.15.98:0/1952963752 conn(0x561821fd84e0 msgr2=0x561822381270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 -- 172.21.15.98:0/1952963752 shutdown_connections 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 -- 172.21.15.98:0/1952963752 wait complete. 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.464+0000 7f86139ba740 1 -- start start 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f86139ba740 1 Processor -- start 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f86139ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f86139ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x56182238fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.471 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f86139ba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fe4390 0x561822396990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f86139ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x5618222ead80 con 0x561821fbb4d0 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f86139ba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x561822271d10 con 0x561821fe4390 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f86139ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x561822387540 con 0x561821fc36b0 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f86139ba740 10 client.0 _ll_register_callbacks cb 0x56182229e210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50502/0 (socket says 172.21.15.98:50502) 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 -- 172.21.15.98:0/3813810083 learned_addr learned my addr 172.21.15.98:0/3813810083 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f8606d9d640 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fe4390 0x561822396990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f8605d9b640 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x56182238fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 -- 172.21.15.98:0/3813810083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 msgr2=0x56182238fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x56182238fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 -- 172.21.15.98:0/3813810083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fe4390 msgr2=0x561822396990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fe4390 0x561822396990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 -- 172.21.15.98:0/3813810083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618220c67d0 con 0x561821fbb4d0 2024-04-25T16:34:49.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f860659c640 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238f4f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f85f80c7510 tx=0x7f85f8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7eb43d90dbc7192 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.465+0000 7f85edffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f85edffb640 1 -- 172.21.15.98:0/3813810083 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8049070 con 0x561821fbb4d0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f85edffb640 1 -- 172.21.15.98:0/3813810083 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f85f8102bc0 con 0x561821fbb4d0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 1 -- 172.21.15.98:0/3813810083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5618222f8990 con 0x561821fbb4d0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f85edffb640 1 -- 172.21.15.98:0/3813810083 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f85f80c6cf0 con 0x561821fbb4d0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 10 client.16173 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 1 -- 172.21.15.98:0/3813810083 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561821ffc3f0 con 0x561821fbb4d0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f85edffb640 1 -- 172.21.15.98:0/3813810083 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x5618222f8990 con 0x561821fbb4d0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f85edffb640 1 -- 172.21.15.98:0/3813810083 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f85f80c3810 con 0x561821fbb4d0 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 10 client.16173 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 20 client.16173 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x561821fa2f60 0x56182239cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 4 client.16173 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:49.473 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f86139ba740 1 -- 172.21.15.98:0/3813810083 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618222f2800 con 0x561821fa2f60 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.466+0000 7f8605d9b640 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x561821fa2f60 0x56182239cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.467+0000 7f85edffb640 10 client.16173 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.467+0000 7f8605d9b640 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x561821fa2f60 0x56182239cc40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72b5d19932d19f22 server_cookie=f942881418e16398 in_seq=0 out_seq=0 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.467+0000 7f85edffb640 1 -- 172.21.15.98:0/3813810083 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5618222f2800 con 0x561821fa2f60 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.467+0000 7f85edffb640 10 client.16173 handle_command_reply: tid=0 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.467+0000 7f86139ba740 1 client.16173 shutdown 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 -- 172.21.15.98:0/3813810083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 msgr2=0x56182238f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238f4f0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f85f80c7510 tx=0x7f85f8009260 comp rx=0 tx=0).stop 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 -- 172.21.15.98:0/3813810083 shutdown_connections 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x561821fa2f60 0x56182239cc40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fe4390 0x561822396990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f85edffb640 0 client.16173 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x56182238fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 --2- 172.21.15.98:0/3813810083 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238f4f0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.474 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 -- 172.21.15.98:0/3813810083 >> 172.21.15.98:0/3813810083 conn(0x561821fd84e0 msgr2=0x561822384180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 -- 172.21.15.98:0/3813810083 shutdown_connections 2024-04-25T16:34:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 1 -- 172.21.15.98:0/3813810083 wait complete. 2024-04-25T16:34:49.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.468+0000 7f86139ba740 20 client.16173 trim_cache size 0 max 16384 2024-04-25T16:34:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.485+0000 7f86139ba740 1 Processor -- start 2024-04-25T16:34:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.485+0000 7f86139ba740 1 -- start start 2024-04-25T16:34:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.485+0000 7f86139ba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fa2f60 0x5618223803f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f86139ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f86139ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x561822396920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f86139ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86081ac9e0 con 0x561821fbb4d0 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f86139ba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86081ae5e0 con 0x561821fa2f60 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f86139ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86081ad0b0 con 0x561821fc36b0 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50504/0 (socket says 172.21.15.98:50504) 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 -- 172.21.15.98:0/1145596903 learned_addr learned my addr 172.21.15.98:0/1145596903 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f860659c640 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fa2f60 0x5618223803f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8606d9d640 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x561822396920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 -- 172.21.15.98:0/1145596903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 msgr2=0x561822396920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x561822396920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 -- 172.21.15.98:0/1145596903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fa2f60 msgr2=0x5618223803f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:49.491 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fa2f60 0x5618223803f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 -- 172.21.15.98:0/1145596903 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561821fbc0f0 con 0x561821fbb4d0 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8606d9d640 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x561822396920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f8605d9b640 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238db10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0095f0 tx=0x7f85fc010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b74d16e76f03dea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f85effff640 1 -- 172.21.15.98:0/1145596903 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc004250 con 0x561821fbb4d0 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f85effff640 1 -- 172.21.15.98:0/1145596903 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f85fc022070 con 0x561821fbb4d0 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.486+0000 7f85effff640 1 -- 172.21.15.98:0/1145596903 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85fc0038c0 con 0x561821fbb4d0 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 -- 172.21.15.98:0/1145596903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 msgr2=0x56182238db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238db10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f85fc0095f0 tx=0x7f85fc010800 comp rx=0 tx=0).stop 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 -- 172.21.15.98:0/1145596903 shutdown_connections 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fc36b0 0x561822396920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fbb4d0 0x56182238db10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 --2- 172.21.15.98:0/1145596903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fa2f60 0x5618223803f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 -- 172.21.15.98:0/1145596903 >> 172.21.15.98:0/1145596903 conn(0x561821fd84e0 msgr2=0x561822381280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:49.492 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 -- 172.21.15.98:0/1145596903 shutdown_connections 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 -- 172.21.15.98:0/1145596903 wait complete. 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 -- start start 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.487+0000 7f86139ba740 1 Processor -- start 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f86139ba740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fa2f60 0x561822394390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f86139ba740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fbb4d0 0x5618223948d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f86139ba740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fc36b0 0x561822395800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f86139ba740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86081ade50 con 0x561821fc36b0 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f86139ba740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86081ad780 con 0x561821fbb4d0 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f86139ba740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86081aa9c0 con 0x561821fa2f60 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f86139ba740 10 client.0 _ll_register_callbacks cb 0x7f85fc0093f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fc36b0 0x561822395800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fc36b0 0x561822395800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50520/0 (socket says 172.21.15.98:50520) 2024-04-25T16:34:49.493 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 -- 172.21.15.98:0/3667279240 learned_addr learned my addr 172.21.15.98:0/3667279240 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8605d9b640 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fbb4d0 0x5618223948d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f860659c640 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fa2f60 0x561822394390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 -- 172.21.15.98:0/3667279240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fa2f60 msgr2=0x561822394390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fa2f60 0x561822394390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 -- 172.21.15.98:0/3667279240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fbb4d0 msgr2=0x5618223948d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fbb4d0 0x5618223948d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 -- 172.21.15.98:0/3667279240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618222ab4b0 con 0x561821fc36b0 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f860659c640 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fa2f60 0x561822394390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f8606d9d640 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fc36b0 0x561822395800 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8600010b90 tx=0x7f8600004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4dc32c175e0861 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f85edffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f85edffb640 1 -- 172.21.15.98:0/3667279240 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860002e060 con 0x561821fc36b0 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f85edffb640 1 -- 172.21.15.98:0/3667279240 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f860001f420 con 0x561821fc36b0 2024-04-25T16:34:49.494 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.488+0000 7f86139ba740 1 -- 172.21.15.98:0/3667279240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f85fc022070 con 0x561821fc36b0 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f85edffb640 1 -- 172.21.15.98:0/3667279240 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8600035050 con 0x561821fc36b0 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f86139ba740 10 client.16179 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f86139ba740 1 -- 172.21.15.98:0/3667279240 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x561822396f80 con 0x561821fc36b0 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f85edffb640 1 -- 172.21.15.98:0/3667279240 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f85fc022070 con 0x561821fc36b0 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f85edffb640 1 -- 172.21.15.98:0/3667279240 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f8600028030 con 0x561821fc36b0 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f86139ba740 10 client.16179 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f86139ba740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f86139ba740 20 client.16179 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f86139ba740 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x561821fe4390 0x5618224072e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f86139ba740 4 client.16179 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f86139ba740 1 -- 172.21.15.98:0/3667279240 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56182226d4a0 con 0x561821fe4390 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.489+0000 7f860659c640 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x561821fe4390 0x5618224072e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f85edffb640 10 client.16179 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860659c640 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x561821fe4390 0x5618224072e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b30a86dea4119ca8 server_cookie=44da6ba6ac3a9eda in_seq=0 out_seq=0 2024-04-25T16:34:49.495 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f85edffb640 1 -- 172.21.15.98:0/3667279240 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56182226d4a0 con 0x561821fe4390 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f85edffb640 10 client.16179 handle_command_reply: tid=0 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 client.16179 shutdown 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 -- 172.21.15.98:0/3667279240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fc36b0 msgr2=0x561822395800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fc36b0 0x561822395800 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f8600010b90 tx=0x7f8600004ab0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 -- 172.21.15.98:0/3667279240 shutdown_connections 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x561821fe4390 0x5618224072e0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x561821fc36b0 0x561822395800 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f85edffb640 0 client.16179 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x561821fbb4d0 0x5618223948d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 --2- 172.21.15.98:0/3667279240 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x561821fa2f60 0x561822394390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 -- 172.21.15.98:0/3667279240 >> 172.21.15.98:0/3667279240 conn(0x561821fd84e0 msgr2=0x561822381280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 -- 172.21.15.98:0/3667279240 shutdown_connections 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 1 -- 172.21.15.98:0/3667279240 wait complete. 2024-04-25T16:34:49.496 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f86139ba740 20 client.16179 trim_cache size 0 max 16384 2024-04-25T16:34:49.497 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:34:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "no active scrubs running", 2024-04-25T16:34:49.497 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": {} 2024-04-25T16:34:49.497 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:34:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860cb06640 1 -- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85f00afaf0 msgr2=0x7f85f00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860cb06640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85f00afaf0 0x7f85f00b2000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x561822386ab0 tx=0x7f8600004a40 comp rx=0 tx=0).stop 2024-04-25T16:34:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860cb06640 1 -- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 msgr2=0x7f8608102200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860cb06640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 0x7f8608102200 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f85f8031460 tx=0x7f85f8005e70 comp rx=0 tx=0).stop 2024-04-25T16:34:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860cb06640 1 -- 172.21.15.98:0/1174208608 shutdown_connections 2024-04-25T16:34:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860cb06640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f85f00afaf0 0x7f85f00b2000 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860cb06640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8608104460 0x7f86081a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.498 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.490+0000 7f860cb06640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8608103940 0x7f860819ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.491+0000 7f860cb06640 1 --2- 172.21.15.98:0/1174208608 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8608102f60 0x7f8608102200 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.491+0000 7f860cb06640 1 -- 172.21.15.98:0/1174208608 >> 172.21.15.98:0/1174208608 conn(0x7f86080fd340 msgr2=0x7f8608104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:49.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.491+0000 7f860cb06640 1 -- 172.21.15.98:0/1174208608 shutdown_connections 2024-04-25T16:34:49.499 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.491+0000 7f860cb06640 1 -- 172.21.15.98:0/1174208608 wait complete. 2024-04-25T16:34:49.500 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:34:49.500 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 -- 172.21.15.98:0/1422326915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 msgr2=0x7f0710104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 --2- 172.21.15.98:0/1422326915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 0x7f0710104010 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0700009900 tx=0x7f070002a410 comp rx=0 tx=0).stop 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 -- 172.21.15.98:0/1422326915 shutdown_connections 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 --2- 172.21.15.98:0/1422326915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0710108900 0x7f071010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 --2- 172.21.15.98:0/1422326915 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 0x7f0710104010 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 --2- 172.21.15.98:0/1422326915 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07101029c0 0x7f07101036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 -- 172.21.15.98:0/1422326915 >> 172.21.15.98:0/1422326915 conn(0x7f07100fdbd0 msgr2=0x7f0710100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 -- 172.21.15.98:0/1422326915 shutdown_connections 2024-04-25T16:34:49.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 -- 172.21.15.98:0/1422326915 wait complete. 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 Processor -- start 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.609+0000 7f0716892640 1 -- start start 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f0716892640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07101029c0 0x7f0710199050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f0716892640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 0x7f0710199590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f0716892640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0710108900 0x7f07101a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f0716892640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f071010e390 con 0x7f0710103c00 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f0716892640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f071010e210 con 0x7f07101029c0 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f0716892640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f071010e510 con 0x7f0710108900 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 0x7f0710199590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 0x7f0710199590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50538/0 (socket says 172.21.15.98:50538) 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 -- 172.21.15.98:0/1210287640 learned_addr learned my addr 172.21.15.98:0/1210287640 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:49.612 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070ffff640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07101029c0 0x7f0710199050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f0714e2f640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0710108900 0x7f07101a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 -- 172.21.15.98:0/1210287640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0710108900 msgr2=0x7f07101a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0710108900 0x7f07101a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 -- 172.21.15.98:0/1210287640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07101029c0 msgr2=0x7f0710199050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07101029c0 0x7f0710199050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 -- 172.21.15.98:0/1210287640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07101a0b90 con 0x7f0710103c00 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070ffff640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07101029c0 0x7f0710199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f0714e2f640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0710108900 0x7f07101a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.610+0000 7f070f7fe640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 0x7f0710199590 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0700009230 tx=0x7f0700005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.611+0000 7f070d7fa640 1 -- 172.21.15.98:0/1210287640 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070003f030 con 0x7f0710103c00 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.611+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07101a0e20 con 0x7f0710103c00 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.611+0000 7f070d7fa640 1 -- 172.21.15.98:0/1210287640 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f07000043d0 con 0x7f0710103c00 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.611+0000 7f070d7fa640 1 -- 172.21.15.98:0/1210287640 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700035070 con 0x7f0710103c00 2024-04-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.611+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07101a1230 con 0x7f0710103c00 2024-04-25T16:34:49.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.612+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0710107710 con 0x7f0710103c00 2024-04-25T16:34:49.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.613+0000 7f070d7fa640 1 -- 172.21.15.98:0/1210287640 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f0700042030 con 0x7f0710103c00 2024-04-25T16:34:49.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.613+0000 7f070d7fa640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06fc0afaf0 0x7f06fc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:49.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.614+0000 7f070ffff640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06fc0afaf0 0x7f06fc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:49.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.614+0000 7f070d7fa640 1 -- 172.21.15.98:0/1210287640 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f07000ff890 con 0x7f0710103c00 2024-04-25T16:34:49.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.614+0000 7f070ffff640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06fc0afaf0 0x7f06fc0b2000 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f07101035c0 tx=0x7f06f8005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:49.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.617+0000 7f070d7fa640 1 -- 172.21.15.98:0/1210287640 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07000c80c0 con 0x7f0710103c00 2024-04-25T16:34:49.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.863+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f071002db50 con 0x7f0710103c00 2024-04-25T16:34:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.864+0000 7f070d7fa640 1 -- 172.21.15.98:0/1210287640 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+11942 (secure 0 0 0) 0x7f07000d0140 con 0x7f0710103c00 2024-04-25T16:34:49.863 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:34:49.863 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6835/441838145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6834","nonce":441838145},{"type":"v1","addr":"172.21.15.98:6835","nonce":441838145}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14571,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.98:6837/1001942637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6836","nonce":1001942637},{"type":"v1","addr":"172.21.15.98:6837","nonce":1001942637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T16:30:09.342794+0000","modified":"2024-04-25T16:34:24.633300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24539,"mds_1":24494,"mds_2":24484,"mds_3":24473,"mds_4":24457},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6839/3308601909","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6838","nonce":3308601909},{"type":"v1","addr":"172.21.15.98:6839","nonce":3308601909}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.98:6841/373690634","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6840","nonce":373690634},{"type":"v1","addr":"172.21.15.98:6841","nonce":373690634}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6833/1945729206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":1945729206},{"type":"v1","addr":"172.21.15.203:6833","nonce":1945729206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24443":{"gid":24443,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.173:6833/3037734321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3037734321},{"type":"v1","addr":"172.21.15.173:6833","nonce":3037734321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24445":{"gid":24445,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.203:6835/2230353021","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2230353021},{"type":"v1","addr":"172.21.15.203:6835","nonce":2230353021}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24457":{"gid":24457,"name":"c","rank":4,"incarnation":26,"state":"up:active","state_seq":13,"addr":"172.21.15.203:6837/998442662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6836","nonce":998442662},{"type":"v1","addr":"172.21.15.203:6837","nonce":998442662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24473":{"gid":24473,"name":"e","rank":3,"incarnation":24,"state":"up:active","state_seq":14,"addr":"172.21.15.173:6835/960684117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":960684117},{"type":"v1","addr":"172.21.15.173:6835","nonce":960684117}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.203:6839/2982027693","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6838","nonce":2982027693},{"type":"v1","addr":"172.21.15.203:6839","nonce":2982027693}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":20,"state":"up:active","state_seq":12,"addr":"172.21.15.173:6837/311453164","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6836","nonce":311453164},{"type":"v1","addr":"172.21.15.173:6837","nonce":311453164}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6839/2050448663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6838","nonce":2050448663},{"type":"v1","addr":"172.21.15.173:6839","nonce":2050448663}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24539,"qdb_cluster":[24473,24494,24457,24539,24484]},"id":1}]} 2024-04-25T16:34:49.863 INFO:teuthology.orchestra.run.smithi098.stderr:dumped fsmap epoch 31 2024-04-25T16:34:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06fc0afaf0 msgr2=0x7f06fc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06fc0afaf0 0x7f06fc0b2000 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f07101035c0 tx=0x7f06f8005f90 comp rx=0 tx=0).stop 2024-04-25T16:34:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 msgr2=0x7f0710199590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 0x7f0710199590 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0700009230 tx=0x7f0700005e80 comp rx=0 tx=0).stop 2024-04-25T16:34:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 shutdown_connections 2024-04-25T16:34:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f06fc0afaf0 0x7f06fc0b2000 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.865 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0710108900 0x7f07101a0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0710103c00 0x7f0710199590 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 --2- 172.21.15.98:0/1210287640 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07101029c0 0x7f0710199050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 >> 172.21.15.98:0/1210287640 conn(0x7f07100fdbd0 msgr2=0x7f07101048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 shutdown_connections 2024-04-25T16:34:49.866 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:49.866+0000 7f0716892640 1 -- 172.21.15.98:0/1210287640 wait complete. 2024-04-25T16:34:49.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:34:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 -- 172.21.15.98:0/2628257387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c1095e0 msgr2=0x7fa46c10bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 --2- 172.21.15.98:0/2628257387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c1095e0 0x7fa46c10bad0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fa464009ca0 tx=0x7fa46402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 -- 172.21.15.98:0/2628257387 shutdown_connections 2024-04-25T16:34:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 --2- 172.21.15.98:0/2628257387 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c1095e0 0x7fa46c10bad0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 --2- 172.21.15.98:0/2628257387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa46c106c30 0x7fa46c109050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 --2- 172.21.15.98:0/2628257387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa46c101670 0x7fa46c101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 -- 172.21.15.98:0/2628257387 >> 172.21.15.98:0/2628257387 conn(0x7fa46c0fcd40 msgr2=0x7fa46c0ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 -- 172.21.15.98:0/2628257387 shutdown_connections 2024-04-25T16:34:50.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.013+0000 7fa47199d640 1 -- 172.21.15.98:0/2628257387 wait complete. 2024-04-25T16:34:50.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.014+0000 7fa47199d640 1 Processor -- start 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.014+0000 7fa47199d640 1 -- start start 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.014+0000 7fa47199d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c101670 0x7fa46c199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.014+0000 7fa47199d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa46c106c30 0x7fa46c19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.014+0000 7fa47199d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa46c1095e0 0x7fa46c1a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.014+0000 7fa47199d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c10f360 con 0x7fa46c101670 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.014+0000 7fa47199d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c10f1e0 con 0x7fa46c1095e0 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.014+0000 7fa47199d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c10f4e0 con 0x7fa46c106c30 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa47099b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c101670 0x7fa46c199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa47099b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c101670 0x7fa46c199f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50556/0 (socket says 172.21.15.98:50556) 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa47099b640 1 -- 172.21.15.98:0/2490131664 learned_addr learned my addr 172.21.15.98:0/2490131664 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa46bfff640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa46c106c30 0x7fa46c19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa47119c640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa46c1095e0 0x7fa46c1a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa46bfff640 1 -- 172.21.15.98:0/2490131664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa46c1095e0 msgr2=0x7fa46c1a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa46bfff640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa46c1095e0 0x7fa46c1a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa46bfff640 1 -- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c101670 msgr2=0x7fa46c199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa46bfff640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c101670 0x7fa46c199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa46bfff640 1 -- 172.21.15.98:0/2490131664 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa46c1a1a80 con 0x7fa46c106c30 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa47099b640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c101670 0x7fa46c199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa46bfff640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa46c106c30 0x7fa46c19a480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa45800e9f0 tx=0x7fa45800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa469ffb640 1 -- 172.21.15.98:0/2490131664 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45800ccc0 con 0x7fa46c106c30 2024-04-25T16:34:50.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa47199d640 1 -- 172.21.15.98:0/2490131664 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa46c1a1d70 con 0x7fa46c106c30 2024-04-25T16:34:50.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa469ffb640 1 -- 172.21.15.98:0/2490131664 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa458005020 con 0x7fa46c106c30 2024-04-25T16:34:50.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.015+0000 7fa469ffb640 1 -- 172.21.15.98:0/2490131664 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45800ce60 con 0x7fa46c106c30 2024-04-25T16:34:50.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.016+0000 7fa47199d640 1 -- 172.21.15.98:0/2490131664 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa46c1a2200 con 0x7fa46c106c30 2024-04-25T16:34:50.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.018+0000 7fa469ffb640 1 -- 172.21.15.98:0/2490131664 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fa458004a40 con 0x7fa46c106c30 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.018+0000 7fa469ffb640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa45c0afae0 0x7fa45c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa47099b640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa45c0afae0 0x7fa45c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa469ffb640 1 -- 172.21.15.98:0/2490131664 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa4580def60 con 0x7fa46c106c30 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa479b87740 1 Processor -- start 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa47099b640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa45c0afae0 0x7fa45c0b1ff0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa460009fd0 tx=0x7fa460005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa479b87740 1 -- start start 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa479b87740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f4f390 0x564bb22f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa479b87740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 0x564bb22f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa479b87740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f2e6b0 0x564bb22f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa479b87740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564bb2255d80 con 0x564bb1f4f390 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa479b87740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564bb220fa20 con 0x564bb1f2e6b0 2024-04-25T16:34:50.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa479b87740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564bb2251d40 con 0x564bb1f264d0 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa47099b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f4f390 0x564bb22f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa47099b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f4f390 0x564bb22f19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50566/0 (socket says 172.21.15.98:50566) 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.019+0000 7fa47099b640 1 -- 172.21.15.98:0/3544606039 learned_addr learned my addr 172.21.15.98:0/3544606039 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa46bfff640 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 0x564bb22f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa47119c640 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f2e6b0 0x564bb22f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa47099b640 1 -- 172.21.15.98:0/3544606039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 msgr2=0x564bb22f1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa47099b640 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 0x564bb22f1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa47099b640 1 -- 172.21.15.98:0/3544606039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f2e6b0 msgr2=0x564bb22f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.026 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa47099b640 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f2e6b0 0x564bb22f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa47099b640 1 -- 172.21.15.98:0/3544606039 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bb221ac30 con 0x564bb1f4f390 2024-04-25T16:34:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa47099b640 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f4f390 0x564bb22f19e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa4600058c0 tx=0x7fa460014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1e59f868d1f2e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa456ffd640 1 -- 172.21.15.98:0/3544606039 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46002a070 con 0x564bb1f4f390 2024-04-25T16:34:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa456ffd640 1 -- 172.21.15.98:0/3544606039 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa46001b8c0 con 0x564bb1f4f390 2024-04-25T16:34:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.020+0000 7fa456ffd640 1 -- 172.21.15.98:0/3544606039 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46001c940 con 0x564bb1f4f390 2024-04-25T16:34:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 -- 172.21.15.98:0/3544606039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f4f390 msgr2=0x564bb22f19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f4f390 0x564bb22f19e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa4600058c0 tx=0x7fa460014000 comp rx=0 tx=0).stop 2024-04-25T16:34:50.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 -- 172.21.15.98:0/3544606039 shutdown_connections 2024-04-25T16:34:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f2e6b0 0x564bb22f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 0x564bb22f1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 --2- 172.21.15.98:0/3544606039 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f4f390 0x564bb22f19e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 -- 172.21.15.98:0/3544606039 >> 172.21.15.98:0/3544606039 conn(0x564bb1f434e0 msgr2=0x564bb22ed1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 -- 172.21.15.98:0/3544606039 shutdown_connections 2024-04-25T16:34:50.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.021+0000 7fa479b87740 1 -- 172.21.15.98:0/3544606039 wait complete. 2024-04-25T16:34:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 1 -- start start 2024-04-25T16:34:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 1 Processor -- start 2024-04-25T16:34:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22fb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb22fbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2302c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x564bb22f3700 con 0x564bb1f2e6b0 2024-04-25T16:34:50.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x564bb21dcdc0 con 0x564bb1f264d0 2024-04-25T16:34:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x564bb22f3880 con 0x564bb1f4f390 2024-04-25T16:34:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa479b87740 10 client.0 _ll_register_callbacks cb 0x564bb21f8600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa46bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb22fbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa46bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb22fbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50570/0 (socket says 172.21.15.98:50570) 2024-04-25T16:34:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa46bfff640 1 -- 172.21.15.98:0/3954369885 learned_addr learned my addr 172.21.15.98:0/3954369885 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa47119c640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2302c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.022+0000 7fa47099b640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22fb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa46bfff640 1 -- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 msgr2=0x564bb2302c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa46bfff640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2302c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa46bfff640 1 -- 172.21.15.98:0/3954369885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 msgr2=0x564bb22fb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa46bfff640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22fb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa46bfff640 1 -- 172.21.15.98:0/3954369885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bb1f673f0 con 0x564bb1f2e6b0 2024-04-25T16:34:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa47119c640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2302c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa47099b640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22fb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa46bfff640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb22fbcb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa4580dc6b0 tx=0x7fa4580a7790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8857a12241091e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa44bfff640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa44bfff640 1 -- 172.21.15.98:0/3954369885 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4580c0070 con 0x564bb1f2e6b0 2024-04-25T16:34:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa44bfff640 1 -- 172.21.15.98:0/3954369885 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa458004d30 con 0x564bb1f2e6b0 2024-04-25T16:34:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa479b87740 1 -- 172.21.15.98:0/3954369885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa46001b8c0 con 0x564bb1f2e6b0 2024-04-25T16:34:50.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa44bfff640 1 -- 172.21.15.98:0/3954369885 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa45801c040 con 0x564bb1f2e6b0 2024-04-25T16:34:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa479b87740 10 client.16197 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa479b87740 1 -- 172.21.15.98:0/3954369885 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564bb20317d0 con 0x564bb1f2e6b0 2024-04-25T16:34:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.023+0000 7fa44bfff640 1 -- 172.21.15.98:0/3954369885 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa46001b8c0 con 0x564bb1f2e6b0 2024-04-25T16:34:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.024+0000 7fa44bfff640 1 -- 172.21.15.98:0/3954369885 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fa4580afad0 con 0x564bb1f2e6b0 2024-04-25T16:34:50.034 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.024+0000 7fa479b87740 10 client.16197 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.024+0000 7fa479b87740 10 client.16197 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-25T16:34:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.024+0000 7fa479b87740 20 client.16197 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.024+0000 7fa479b87740 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x564bb1f0df60 0x564bb23094a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.035 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.024+0000 7fa479b87740 4 client.16197 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.024+0000 7fa479b87740 1 -- 172.21.15.98:0/3954369885 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564bb2263990 con 0x564bb1f0df60 2024-04-25T16:34:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.024+0000 7fa47099b640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x564bb1f0df60 0x564bb23094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.025+0000 7fa44bfff640 10 client.16197 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:34:50.036 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.025+0000 7fa47099b640 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x564bb1f0df60 0x564bb23094a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bb65535384f09e0c server_cookie=ddd1e9fb896d0ade in_seq=0 out_seq=0 2024-04-25T16:34:50.039 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa44bfff640 1 -- 172.21.15.98:0/3954369885 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564bb2263990 con 0x564bb1f0df60 2024-04-25T16:34:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa44bfff640 10 client.16197 handle_command_reply: tid=0 2024-04-25T16:34:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 client.16197 shutdown 2024-04-25T16:34:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 -- 172.21.15.98:0/3954369885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 msgr2=0x564bb22fbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb22fbcb0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa4580dc6b0 tx=0x7fa4580a7790 comp rx=0 tx=0).stop 2024-04-25T16:34:50.040 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 -- 172.21.15.98:0/3954369885 shutdown_connections 2024-04-25T16:34:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x564bb1f0df60 0x564bb23094a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2302c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb22fbcb0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.041 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 --2- 172.21.15.98:0/3954369885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22fb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa479b87740 1 -- 172.21.15.98:0/3954369885 >> 172.21.15.98:0/3954369885 conn(0x564bb1f434e0 msgr2=0x564bb22f0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.028+0000 7fa44bfff640 0 client.16197 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:34:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.029+0000 7fa479b87740 1 -- 172.21.15.98:0/3954369885 shutdown_connections 2024-04-25T16:34:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.029+0000 7fa479b87740 1 -- 172.21.15.98:0/3954369885 wait complete. 2024-04-25T16:34:50.042 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.029+0000 7fa479b87740 20 client.16197 trim_cache size 0 max 16384 2024-04-25T16:34:50.062 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.042+0000 7fa479b87740 1 Processor -- start 2024-04-25T16:34:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.042+0000 7fa479b87740 1 -- start start 2024-04-25T16:34:50.063 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.042+0000 7fa479b87740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 0x564bb22f9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.042+0000 7fa479b87740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2302b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.042+0000 7fa479b87740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f4f390 0x564bb22fa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.042+0000 7fa479b87740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1ac7a0 con 0x564bb1f2e6b0 2024-04-25T16:34:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.042+0000 7fa479b87740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1ae3a0 con 0x564bb1f4f390 2024-04-25T16:34:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.042+0000 7fa479b87740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1ace70 con 0x564bb1f264d0 2024-04-25T16:34:50.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2302b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2302b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50576/0 (socket says 172.21.15.98:50576) 2024-04-25T16:34:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 -- 172.21.15.98:0/3746337269 learned_addr learned my addr 172.21.15.98:0/3746337269 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.066 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa47099b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 0x564bb22f9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa47119c640 1 --2- 172.21.15.98:0/3746337269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f4f390 0x564bb22fa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 -- 172.21.15.98:0/3746337269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 msgr2=0x564bb22f9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 --2- 172.21.15.98:0/3746337269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 0x564bb22f9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 -- 172.21.15.98:0/3746337269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f4f390 msgr2=0x564bb22fa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 --2- 172.21.15.98:0/3746337269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f4f390 0x564bb22fa3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 -- 172.21.15.98:0/3746337269 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bb22deba0 con 0x564bb1f2e6b0 2024-04-25T16:34:50.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa46bfff640 1 --2- 172.21.15.98:0/3746337269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2302b30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa4580a3630 tx=0x7fa4580a7900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853a547adc196f1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa4567fc640 1 -- 172.21.15.98:0/3746337269 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458089070 con 0x564bb1f2e6b0 2024-04-25T16:34:50.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa4567fc640 1 -- 172.21.15.98:0/3746337269 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa458004d30 con 0x564bb1f2e6b0 2024-04-25T16:34:50.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.043+0000 7fa4567fc640 1 -- 172.21.15.98:0/3746337269 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4580c0070 con 0x564bb1f2e6b0 2024-04-25T16:34:50.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 -- 172.21.15.98:0/3746337269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 msgr2=0x564bb2302b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 --2- 172.21.15.98:0/3746337269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2302b30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa4580a3630 tx=0x7fa4580a7900 comp rx=0 tx=0).stop 2024-04-25T16:34:50.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 -- 172.21.15.98:0/3746337269 shutdown_connections 2024-04-25T16:34:50.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 --2- 172.21.15.98:0/3746337269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f4f390 0x564bb22fa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 --2- 172.21.15.98:0/3746337269 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2302b30 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 --2- 172.21.15.98:0/3746337269 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f264d0 0x564bb22f9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 -- 172.21.15.98:0/3746337269 >> 172.21.15.98:0/3746337269 conn(0x564bb1f434e0 msgr2=0x564bb22f6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 -- 172.21.15.98:0/3746337269 shutdown_connections 2024-04-25T16:34:50.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 -- 172.21.15.98:0/3746337269 wait complete. 2024-04-25T16:34:50.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.044+0000 7fa479b87740 1 -- start start 2024-04-25T16:34:50.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa479b87740 1 Processor -- start 2024-04-25T16:34:50.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa479b87740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22ffcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa479b87740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2300210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa479b87740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2301140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa479b87740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1adc10 con 0x564bb1f2e6b0 2024-04-25T16:34:50.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa479b87740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1ad540 con 0x564bb1f264d0 2024-04-25T16:34:50.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa479b87740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa46c1aa780 con 0x564bb1f4f390 2024-04-25T16:34:50.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa479b87740 10 client.0 _ll_register_callbacks cb 0x7fa45800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.075 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa47099b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22ffcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa47099b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22ffcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49562/0 (socket says 172.21.15.98:49562) 2024-04-25T16:34:50.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa46bfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2300210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa47099b640 1 -- 172.21.15.98:0/4138608799 learned_addr learned my addr 172.21.15.98:0/4138608799 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.076 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.045+0000 7fa47119c640 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2301140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa47099b640 1 -- 172.21.15.98:0/4138608799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 msgr2=0x564bb2301140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa47099b640 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2301140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa47099b640 1 -- 172.21.15.98:0/4138608799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 msgr2=0x564bb2300210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.077 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa47099b640 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2300210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa47099b640 1 -- 172.21.15.98:0/4138608799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bb2301700 con 0x564bb1f264d0 2024-04-25T16:34:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa46bfff640 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2300210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa47099b640 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22ffcd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x564bb22f2a30 tx=0x7fa460009d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e62c7beccc499619 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.078 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa44bfff640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa44bfff640 1 -- 172.21.15.98:0/4138608799 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa460037070 con 0x564bb1f264d0 2024-04-25T16:34:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa44bfff640 1 -- 172.21.15.98:0/4138608799 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa46000fdf0 con 0x564bb1f264d0 2024-04-25T16:34:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.046+0000 7fa479b87740 1 -- 172.21.15.98:0/4138608799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa458004d30 con 0x564bb1f264d0 2024-04-25T16:34:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa44bfff640 1 -- 172.21.15.98:0/4138608799 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa460045050 con 0x564bb1f264d0 2024-04-25T16:34:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa479b87740 10 client.25718 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.079 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa479b87740 1 -- 172.21.15.98:0/4138608799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x564bb2302980 con 0x564bb1f264d0 2024-04-25T16:34:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa44bfff640 1 -- 172.21.15.98:0/4138608799 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa458004d30 con 0x564bb1f264d0 2024-04-25T16:34:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa44bfff640 1 -- 172.21.15.98:0/4138608799 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fa460024030 con 0x564bb1f264d0 2024-04-25T16:34:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa479b87740 10 client.25718 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa479b87740 10 client.25718 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-25T16:34:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa479b87740 20 client.25718 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.080 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa479b87740 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x564bb1ee33a0 0x564bb2373820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa479b87740 4 client.25718 mds_command: new command op to 24457 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa479b87740 1 -- 172.21.15.98:0/4138608799 --> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564bb225d800 con 0x564bb1ee33a0 2024-04-25T16:34:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.047+0000 7fa46bfff640 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x564bb1ee33a0 0x564bb2373820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.048+0000 7fa44bfff640 10 client.25718 ms_handle_connect on v2:172.21.15.203:6836/998442662 2024-04-25T16:34:50.081 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.048+0000 7fa46bfff640 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x564bb1ee33a0 0x564bb2373820 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bfffd8856bc3b939 server_cookie=de7ac3faa1c98c41 in_seq=0 out_seq=0 2024-04-25T16:34:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.048+0000 7fa44bfff640 1 -- 172.21.15.98:0/4138608799 <== mds.4 v2:172.21.15.203:6836/998442662 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564bb225d800 con 0x564bb1ee33a0 2024-04-25T16:34:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.048+0000 7fa44bfff640 10 client.25718 handle_command_reply: tid=0 2024-04-25T16:34:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.048+0000 7fa479b87740 1 client.25718 shutdown 2024-04-25T16:34:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 -- 172.21.15.98:0/4138608799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 msgr2=0x564bb22ffcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22ffcd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x564bb22f2a30 tx=0x7fa460009d50 comp rx=0 tx=0).stop 2024-04-25T16:34:50.082 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 -- 172.21.15.98:0/4138608799 shutdown_connections 2024-04-25T16:34:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.203:6836/998442662,v1:172.21.15.203:6837/998442662] conn(0x564bb1ee33a0 0x564bb2373820 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x564bb1f4f390 0x564bb2301140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x564bb1f2e6b0 0x564bb2300210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa44bfff640 0 client.25718 ms_handle_reset on v2:172.21.15.203:6836/998442662 2024-04-25T16:34:50.083 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 --2- 172.21.15.98:0/4138608799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x564bb1f264d0 0x564bb22ffcd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 -- 172.21.15.98:0/4138608799 >> 172.21.15.98:0/4138608799 conn(0x564bb1f434e0 msgr2=0x564bb22f6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 -- 172.21.15.98:0/4138608799 shutdown_connections 2024-04-25T16:34:50.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 1 -- 172.21.15.98:0/4138608799 wait complete. 2024-04-25T16:34:50.084 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa479b87740 20 client.25718 trim_cache size 0 max 16384 2024-04-25T16:34:50.085 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 -- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa45c0afae0 msgr2=0x7fa45c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.087 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa45c0afae0 0x7fa45c0b1ff0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa460009fd0 tx=0x7fa460005f90 comp rx=0 tx=0).stop 2024-04-25T16:34:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 -- 172.21.15.98:0/2490131664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa46c106c30 msgr2=0x7fa46c19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa46c106c30 0x7fa46c19a480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa45800e9f0 tx=0x7fa45800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 -- 172.21.15.98:0/2490131664 shutdown_connections 2024-04-25T16:34:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa45c0afae0 0x7fa45c0b1ff0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa46c1095e0 0x7fa46c1a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa46c106c30 0x7fa46c19a480 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 --2- 172.21.15.98:0/2490131664 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa46c101670 0x7fa46c199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.049+0000 7fa47199d640 1 -- 172.21.15.98:0/2490131664 >> 172.21.15.98:0/2490131664 conn(0x7fa46c0fcd40 msgr2=0x7fa46c107b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.050+0000 7fa47199d640 1 -- 172.21.15.98:0/2490131664 shutdown_connections 2024-04-25T16:34:50.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.050+0000 7fa47199d640 1 -- 172.21.15.98:0/2490131664 wait complete. 2024-04-25T16:34:50.090 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:34:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 -- 172.21.15.98:0/1037946480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 msgr2=0x7f7f88102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/1037946480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 0x7f7f88102e70 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800aa10 tx=0x7f7f7802b720 comp rx=0 tx=0).stop 2024-04-25T16:34:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 -- 172.21.15.98:0/1037946480 shutdown_connections 2024-04-25T16:34:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/1037946480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f881080b0 0x7f7f8810a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/1037946480 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f881033b0 0x7f7f881037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/1037946480 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 0x7f7f88102e70 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 -- 172.21.15.98:0/1037946480 >> 172.21.15.98:0/1037946480 conn(0x7f7f880fd340 msgr2=0x7f7f880ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 -- 172.21.15.98:0/1037946480 shutdown_connections 2024-04-25T16:34:50.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 -- 172.21.15.98:0/1037946480 wait complete. 2024-04-25T16:34:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.223+0000 7f7f8db1f640 1 Processor -- start 2024-04-25T16:34:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f8db1f640 1 -- start start 2024-04-25T16:34:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f8db1f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 0x7f7f8819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f8db1f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f881033b0 0x7f7f8819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.242 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f8db1f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f881080b0 0x7f7f881a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f8db1f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8810d2b0 con 0x7f7f88102170 2024-04-25T16:34:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f8db1f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8810d130 con 0x7f7f881033b0 2024-04-25T16:34:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f8db1f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8810d430 con 0x7f7f881080b0 2024-04-25T16:34:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 0x7f7f8819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 0x7f7f8819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50598/0 (socket says 172.21.15.98:50598) 2024-04-25T16:34:50.243 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f877fe640 1 -- 172.21.15.98:0/4097331552 learned_addr learned my addr 172.21.15.98:0/4097331552 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f87fff640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f881080b0 0x7f7f881a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f881033b0 0x7f7f8819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f877fe640 1 -- 172.21.15.98:0/4097331552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f881080b0 msgr2=0x7f7f881a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f877fe640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f881080b0 0x7f7f881a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f877fe640 1 -- 172.21.15.98:0/4097331552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f881033b0 msgr2=0x7f7f8819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f877fe640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f881033b0 0x7f7f8819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.224+0000 7f7f877fe640 1 -- 172.21.15.98:0/4097331552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f881a1c90 con 0x7f7f88102170 2024-04-25T16:34:50.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.225+0000 7f7f877fe640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 0x7f7f8819a1a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7f78005760 tx=0x7f7f78004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.225+0000 7f7f84ff9640 1 -- 172.21.15.98:0/4097331552 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7803f070 con 0x7f7f88102170 2024-04-25T16:34:50.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.225+0000 7f7f8db1f640 1 -- 172.21.15.98:0/4097331552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f881a1f20 con 0x7f7f88102170 2024-04-25T16:34:50.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.225+0000 7f7f84ff9640 1 -- 172.21.15.98:0/4097331552 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f78030980 con 0x7f7f88102170 2024-04-25T16:34:50.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.225+0000 7f7f84ff9640 1 -- 172.21.15.98:0/4097331552 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78036040 con 0x7f7f88102170 2024-04-25T16:34:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.225+0000 7f7f8db1f640 1 -- 172.21.15.98:0/4097331552 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f881a2380 con 0x7f7f88102170 2024-04-25T16:34:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f949d3740 1 Processor -- start 2024-04-25T16:34:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f949d3740 1 -- start start 2024-04-25T16:34:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbcbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbcc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbcce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f949d3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d1ecb2bd80 con 0x55d1ec825390 2024-04-25T16:34:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f949d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d1ecae5a20 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f949d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d1ecb27d40 con 0x55d1ec8046b0 2024-04-25T16:34:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f84ff9640 1 -- 172.21.15.98:0/4097331552 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f7f78042070 con 0x7f7f88102170 2024-04-25T16:34:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbcbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.247 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f86ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbcce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbcbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50604/0 (socket says 172.21.15.98:50604) 2024-04-25T16:34:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.228+0000 7f7f87fff640 1 -- 172.21.15.98:0/4110219627 learned_addr learned my addr 172.21.15.98:0/4110219627 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f877fe640 1 --2- 172.21.15.98:0/4110219627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbcc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f86ffd640 1 -- 172.21.15.98:0/4110219627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 msgr2=0x55d1ecbcc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/4110219627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbcc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.248 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f86ffd640 1 -- 172.21.15.98:0/4110219627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 msgr2=0x55d1ecbcbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/4110219627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbcbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f86ffd640 1 -- 172.21.15.98:0/4110219627 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1ecaf0c30 con 0x55d1ec8046b0 2024-04-25T16:34:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/4110219627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbcce30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009bf0 tx=0x7f7f7c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb47c618dd21e21d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f6dffb640 1 -- 172.21.15.98:0/4110219627 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c004050 con 0x55d1ec8046b0 2024-04-25T16:34:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f84ff9640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7f700afaf0 0x7f7f700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f6dffb640 1 -- 172.21.15.98:0/4110219627 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f7c005020 con 0x55d1ec8046b0 2024-04-25T16:34:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.229+0000 7f7f6dffb640 1 -- 172.21.15.98:0/4110219627 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c004c30 con 0x55d1ec8046b0 2024-04-25T16:34:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.230+0000 7f7f87fff640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7f700afaf0 0x7f7f700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.230+0000 7f7f84ff9640 1 -- 172.21.15.98:0/4097331552 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7f780ff560 con 0x7f7f88102170 2024-04-25T16:34:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.230+0000 7f7f949d3740 1 -- 172.21.15.98:0/4110219627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 msgr2=0x55d1ecbcce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.230+0000 7f7f87fff640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7f700afaf0 0x7f7f700b2000 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55d1ecbccc70 tx=0x7f7f80009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.230+0000 7f7f949d3740 1 --2- 172.21.15.98:0/4110219627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbcce30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009bf0 tx=0x7f7f7c011810 comp rx=0 tx=0).stop 2024-04-25T16:34:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 -- 172.21.15.98:0/4110219627 shutdown_connections 2024-04-25T16:34:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 --2- 172.21.15.98:0/4110219627 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbcce30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 --2- 172.21.15.98:0/4110219627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbcc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 --2- 172.21.15.98:0/4110219627 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbcbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 -- 172.21.15.98:0/4110219627 >> 172.21.15.98:0/4110219627 conn(0x55d1ec8194e0 msgr2=0x55d1ecbc3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 -- 172.21.15.98:0/4110219627 shutdown_connections 2024-04-25T16:34:50.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 -- 172.21.15.98:0/4110219627 wait complete. 2024-04-25T16:34:50.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 -- start start 2024-04-25T16:34:50.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.231+0000 7f7f949d3740 1 Processor -- start 2024-04-25T16:34:50.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbd5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbdcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f949d3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55d1ecbcd940 con 0x55d1ec825390 2024-04-25T16:34:50.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f949d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55d1ecab2dc0 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f949d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55d1ecbcdac0 con 0x55d1ec8046b0 2024-04-25T16:34:50.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f949d3740 10 client.0 _ll_register_callbacks cb 0x55d1ecace600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbd5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbdcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f86ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbd5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49598/0 (socket says 172.21.15.98:49598) 2024-04-25T16:34:50.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 -- 172.21.15.98:0/467302020 learned_addr learned my addr 172.21.15.98:0/467302020 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 -- 172.21.15.98:0/467302020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 msgr2=0x55d1ecbd5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 -- 172.21.15.98:0/467302020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 msgr2=0x55d1ecbdcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbdcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 -- 172.21.15.98:0/467302020 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1ec83d3f0 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f87fff640 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbdcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f877fe640 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbd5920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7f780c3760 tx=0x7f7f7800a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd2367d80a62ffc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:50.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f66ffd640 1 -- 172.21.15.98:0/467302020 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78049070 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f66ffd640 1 -- 172.21.15.98:0/467302020 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7f780c3970 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.232+0000 7f7f949d3740 1 -- 172.21.15.98:0/467302020 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f7c005020 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f66ffd640 1 -- 172.21.15.98:0/467302020 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7f780c74b0 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f949d3740 10 client.25730 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f949d3740 1 -- 172.21.15.98:0/467302020 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d1ec9077d0 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.256 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f66ffd640 1 -- 172.21.15.98:0/467302020 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f7c005020 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f66ffd640 1 -- 172.21.15.98:0/467302020 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f7f7800b040 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f949d3740 10 client.25730 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f949d3740 10 client.25730 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f949d3740 20 client.25730 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f949d3740 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55d1ec7e3f60 0x55d1ecbe3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f949d3740 4 client.25730 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f949d3740 1 -- 172.21.15.98:0/467302020 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d1ecb39990 con 0x55d1ec7e3f60 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.233+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55d1ec7e3f60 0x55d1ecbe3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.234+0000 7f7f66ffd640 10 client.25730 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:34:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.234+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55d1ec7e3f60 0x55d1ecbe3670 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f797c7be3d1a08ac server_cookie=a51d12ef9b1abe49 in_seq=0 out_seq=0 2024-04-25T16:34:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f66ffd640 1 -- 172.21.15.98:0/467302020 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d1ecb39990 con 0x55d1ec7e3f60 2024-04-25T16:34:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f66ffd640 10 client.25730 handle_command_reply: tid=0 2024-04-25T16:34:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 client.25730 shutdown 2024-04-25T16:34:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 -- 172.21.15.98:0/467302020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 msgr2=0x55d1ecbd5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbd5920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7f780c3760 tx=0x7f7f7800a730 comp rx=0 tx=0).stop 2024-04-25T16:34:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 -- 172.21.15.98:0/467302020 shutdown_connections 2024-04-25T16:34:50.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55d1ec7e3f60 0x55d1ecbe3670 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec825390 0x55d1ecbdcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 --2- 172.21.15.98:0/467302020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbd5920 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 -- 172.21.15.98:0/467302020 >> 172.21.15.98:0/467302020 conn(0x55d1ec8194e0 msgr2=0x55d1ecbc2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f66ffd640 0 client.25730 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:34:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 -- 172.21.15.98:0/467302020 shutdown_connections 2024-04-25T16:34:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 1 -- 172.21.15.98:0/467302020 wait complete. 2024-04-25T16:34:50.259 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.235+0000 7f7f949d3740 20 client.25730 trim_cache size 0 max 16384 2024-04-25T16:34:50.260 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:49 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1210287640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:34:50.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:49 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1210287640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:34:50.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.253+0000 7f7f949d3740 1 Processor -- start 2024-04-25T16:34:50.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.253+0000 7f7f949d3740 1 -- start start 2024-04-25T16:34:50.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbe35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.267 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f949d3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f881aa720 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f949d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f881ac320 con 0x55d1ec825390 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f949d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7f881aadf0 con 0x55d1ec8046b0 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbe35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f86ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbe35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50622/0 (socket says 172.21.15.98:50622) 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 -- 172.21.15.98:0/4022836492 learned_addr learned my addr 172.21.15.98:0/4022836492 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f87fff640 1 --2- 172.21.15.98:0/4022836492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 -- 172.21.15.98:0/4022836492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 msgr2=0x55d1ecbd0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 --2- 172.21.15.98:0/4022836492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 -- 172.21.15.98:0/4022836492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 msgr2=0x55d1ecbd0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 --2- 172.21.15.98:0/4022836492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 -- 172.21.15.98:0/4022836492 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1ecbbd2b0 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.254+0000 7f7f877fe640 1 --2- 172.21.15.98:0/4022836492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbe35a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7f780ff3e0 tx=0x7f7f780fccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248bf8d430cb1211 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.268 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.255+0000 7f7f6d7fa640 1 -- 172.21.15.98:0/4022836492 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78049070 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.255+0000 7f7f6d7fa640 1 -- 172.21.15.98:0/4022836492 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f780c71f0 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.255+0000 7f7f6d7fa640 1 -- 172.21.15.98:0/4022836492 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f780c6e50 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 -- 172.21.15.98:0/4022836492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 msgr2=0x55d1ecbe35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 --2- 172.21.15.98:0/4022836492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbe35a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f7f780ff3e0 tx=0x7f7f780fccf0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 -- 172.21.15.98:0/4022836492 shutdown_connections 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 --2- 172.21.15.98:0/4022836492 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 --2- 172.21.15.98:0/4022836492 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 --2- 172.21.15.98:0/4022836492 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbe35a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 -- 172.21.15.98:0/4022836492 >> 172.21.15.98:0/4022836492 conn(0x55d1ec8194e0 msgr2=0x55d1ecbc2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 -- 172.21.15.98:0/4022836492 shutdown_connections 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 -- 172.21.15.98:0/4022836492 wait complete. 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.256+0000 7f7f949d3740 1 -- start start 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f949d3740 1 Processor -- start 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbdc410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f949d3740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f949d3740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7f881abb90 con 0x55d1ec7fc4d0 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f949d3740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7f881ab4c0 con 0x55d1ec825390 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f949d3740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7f881a8700 con 0x55d1ec8046b0 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f949d3740 10 client.0 _ll_register_callbacks cb 0x7f7f78005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f877fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbdc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f86ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f86ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40236/0 (socket says 172.21.15.98:40236) 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f87fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49620/0 (socket says 172.21.15.98:49620) 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.257+0000 7f7f86ffd640 1 -- 172.21.15.98:0/3502018910 learned_addr learned my addr 172.21.15.98:0/3502018910 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f86ffd640 1 -- 172.21.15.98:0/3502018910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 msgr2=0x55d1ecbd95c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd95c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f86ffd640 1 -- 172.21.15.98:0/3502018910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 msgr2=0x55d1ecbdc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbdc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f86ffd640 1 -- 172.21.15.98:0/3502018910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1ecbcdbb0 con 0x55d1ec8046b0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f87fff640 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f86ffd640 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd8f90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c014040 tx=0x7f7f7c0097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9a2a40f02fa78a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f66ffd640 1 -- 172.21.15.98:0/3502018910 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c0037f0 con 0x55d1ec8046b0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f66ffd640 1 -- 172.21.15.98:0/3502018910 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7f7c003990 con 0x55d1ec8046b0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f949d3740 1 -- 172.21.15.98:0/3502018910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f780c71f0 con 0x55d1ec8046b0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f66ffd640 1 -- 172.21.15.98:0/3502018910 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7f7c005c50 con 0x55d1ec8046b0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f949d3740 10 client.25657 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.258+0000 7f7f949d3740 1 -- 172.21.15.98:0/3502018910 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55d1ecbc74c0 con 0x55d1ec8046b0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.259+0000 7f7f66ffd640 1 -- 172.21.15.98:0/3502018910 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7f780c71f0 con 0x55d1ec8046b0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.259+0000 7f7f66ffd640 1 -- 172.21.15.98:0/3502018910 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f7f7c02db80 con 0x55d1ec8046b0 2024-04-25T16:34:50.271 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.259+0000 7f7f949d3740 10 client.25657 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.259+0000 7f7f949d3740 10 client.25657 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.e 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.259+0000 7f7f949d3740 20 client.25657 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.259+0000 7f7f949d3740 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55d1ec7b93a0 0x55d1ecbc9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.259+0000 7f7f949d3740 4 client.25657 mds_command: new command op to 24473 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.259+0000 7f7f949d3740 1 -- 172.21.15.98:0/3502018910 --> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d1ecb33800 con 0x55d1ec7b93a0 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.260+0000 7f7f877fe640 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55d1ec7b93a0 0x55d1ecbc9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.260+0000 7f7f877fe640 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55d1ec7b93a0 0x55d1ecbc9d90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6bf8d0c788c4272a server_cookie=d733e7204449de21 in_seq=0 out_seq=0 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.260+0000 7f7f66ffd640 10 client.25657 ms_handle_connect on v2:172.21.15.173:6834/960684117 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.260+0000 7f7f66ffd640 1 -- 172.21.15.98:0/3502018910 <== mds.3 v2:172.21.15.173:6834/960684117 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d1ecb33800 con 0x55d1ec7b93a0 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.260+0000 7f7f66ffd640 10 client.25657 handle_command_reply: tid=0 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.260+0000 7f7f949d3740 1 client.25657 shutdown 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.260+0000 7f7f949d3740 1 -- 172.21.15.98:0/3502018910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 msgr2=0x55d1ecbd8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd8f90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c014040 tx=0x7f7f7c0097a0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 -- 172.21.15.98:0/3502018910 shutdown_connections 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.173:6834/960684117,v1:172.21.15.173:6835/960684117] conn(0x55d1ec7b93a0 0x55d1ecbc9d90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.272 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55d1ec825390 0x55d1ecbd95c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55d1ec8046b0 0x55d1ecbd8f90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 --2- 172.21.15.98:0/3502018910 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55d1ec7fc4d0 0x55d1ecbdc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 -- 172.21.15.98:0/3502018910 >> 172.21.15.98:0/3502018910 conn(0x55d1ec8194e0 msgr2=0x55d1ecbd54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f66ffd640 0 client.25657 ms_handle_reset on v2:172.21.15.173:6834/960684117 2024-04-25T16:34:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 -- 172.21.15.98:0/3502018910 shutdown_connections 2024-04-25T16:34:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 1 -- 172.21.15.98:0/3502018910 wait complete. 2024-04-25T16:34:50.273 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.261+0000 7f7f949d3740 20 client.25657 trim_cache size 0 max 16384 2024-04-25T16:34:50.273 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 -- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7f700afaf0 msgr2=0x7f7f700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7f700afaf0 0x7f7f700b2000 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x55d1ecbccc70 tx=0x7f7f80009500 comp rx=0 tx=0).stop 2024-04-25T16:34:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 -- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 msgr2=0x7f7f8819a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 0x7f7f8819a1a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f7f78005760 tx=0x7f7f78004270 comp rx=0 tx=0).stop 2024-04-25T16:34:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 -- 172.21.15.98:0/4097331552 shutdown_connections 2024-04-25T16:34:50.274 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7f700afaf0 0x7f7f700b2000 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7f881080b0 0x7f7f881a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f881033b0 0x7f7f8819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 --2- 172.21.15.98:0/4097331552 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7f88102170 0x7f7f8819a1a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 -- 172.21.15.98:0/4097331552 >> 172.21.15.98:0/4097331552 conn(0x7f7f880fd340 msgr2=0x7f7f88109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 -- 172.21.15.98:0/4097331552 shutdown_connections 2024-04-25T16:34:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.262+0000 7f7f8db1f640 1 -- 172.21.15.98:0/4097331552 wait complete. 2024-04-25T16:34:50.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:34:50.399 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:49 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1210287640' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 -- 172.21.15.98:0/1159371259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 msgr2=0x7f6ea81047e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 --2- 172.21.15.98:0/1159371259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 0x7f6ea81047e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009900 tx=0x7f6e9002a410 comp rx=0 tx=0).stop 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 -- 172.21.15.98:0/1159371259 shutdown_connections 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 --2- 172.21.15.98:0/1159371259 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea8109a20 0x7f6ea810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 --2- 172.21.15.98:0/1159371259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea8104d20 0x7f6ea8105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 --2- 172.21.15.98:0/1159371259 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 0x7f6ea81047e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 -- 172.21.15.98:0/1159371259 >> 172.21.15.98:0/1159371259 conn(0x7f6ea80fecd0 msgr2=0x7f6ea8101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 -- 172.21.15.98:0/1159371259 shutdown_connections 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.405+0000 7f6ead842640 1 -- 172.21.15.98:0/1159371259 wait complete. 2024-04-25T16:34:50.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.406+0000 7f6ead842640 1 Processor -- start 2024-04-25T16:34:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.406+0000 7f6ead842640 1 -- start start 2024-04-25T16:34:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ead842640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 0x7f6ea819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ead842640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea8104d20 0x7f6ea819a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ead842640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea8109a20 0x7f6ea81a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ead842640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea810f490 con 0x7f6ea8103ae0 2024-04-25T16:34:50.408 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ead842640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea810f310 con 0x7f6ea8109a20 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ead842640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea810f610 con 0x7f6ea8104d20 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 0x7f6ea819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 0x7f6ea819a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50670/0 (socket says 172.21.15.98:50670) 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/406187247 learned_addr learned my addr 172.21.15.98:0/406187247 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea77fe640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea8109a20 0x7f6ea81a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea8104d20 0x7f6ea819a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/406187247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea8104d20 msgr2=0x7f6ea819a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea8104d20 0x7f6ea819a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/406187247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea8109a20 msgr2=0x7f6ea81a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea8109a20 0x7f6ea81a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/406187247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea81a1cc0 con 0x7f6ea8103ae0 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.407+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 0x7f6ea819a180 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6e90005760 tx=0x7f6e90005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.409 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.408+0000 7f6eac840640 1 -- 172.21.15.98:0/406187247 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9003f030 con 0x7f6ea8103ae0 2024-04-25T16:34:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.408+0000 7f6eac840640 1 -- 172.21.15.98:0/406187247 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e900043d0 con 0x7f6ea8103ae0 2024-04-25T16:34:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.408+0000 7f6eac840640 1 -- 172.21.15.98:0/406187247 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90035070 con 0x7f6ea8103ae0 2024-04-25T16:34:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.408+0000 7f6ead842640 1 -- 172.21.15.98:0/406187247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea81a1f50 con 0x7f6ea8103ae0 2024-04-25T16:34:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.408+0000 7f6ead842640 1 -- 172.21.15.98:0/406187247 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea81a2350 con 0x7f6ea8103ae0 2024-04-25T16:34:50.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.410+0000 7f6eac840640 1 -- 172.21.15.98:0/406187247 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f6e90042030 con 0x7f6ea8103ae0 2024-04-25T16:34:50.411 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eac840640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e940afaf0 0x7f6e940b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e940afaf0 0x7f6e940b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.412 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eac840640 1 -- 172.21.15.98:0/406187247 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f6e900ff290 con 0x7f6ea8103ae0 2024-04-25T16:34:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eb47cf740 1 Processor -- start 2024-04-25T16:34:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eb47cf740 1 -- start start 2024-04-25T16:34:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e940afaf0 0x7f6e940b2000 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6ea819b220 tx=0x7f6e98005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.414 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b6bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56383779c4d0 0x563837b6c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 0x563837b6ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563837acbd80 con 0x56383779c4d0 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563837a85a20 con 0x5638377a46b0 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.411+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563837ac7d40 con 0x5638377c5390 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56383779c4d0 0x563837b6c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56383779c4d0 0x563837b6c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50680/0 (socket says 172.21.15.98:50680) 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b6bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea67fc640 1 -- 172.21.15.98:0/90683905 learned_addr learned my addr 172.21.15.98:0/90683905 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea77fe640 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 0x563837b6ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/90683905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 msgr2=0x563837b6ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 0x563837b6ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/90683905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56383779c4d0 msgr2=0x563837b6c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56383779c4d0 0x563837b6c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/90683905 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563837a90c30 con 0x5638377c5390 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56383779c4d0 0x563837b6c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b6bc20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6e900fc9e0 tx=0x7f6e90009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70cab29d53fff5f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6e9dffb640 1 -- 172.21.15.98:0/90683905 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90049040 con 0x5638377c5390 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6e9dffb640 1 -- 172.21.15.98:0/90683905 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e900d0220 con 0x5638377c5390 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.412+0000 7f6e9dffb640 1 -- 172.21.15.98:0/90683905 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e900c76d0 con 0x5638377c5390 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.413+0000 7f6eb47cf740 1 -- 172.21.15.98:0/90683905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 msgr2=0x563837b6bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.413+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b6bc20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6e900fc9e0 tx=0x7f6e90009260 comp rx=0 tx=0).stop 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.413+0000 7f6eb47cf740 1 -- 172.21.15.98:0/90683905 shutdown_connections 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.413+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 0x563837b6ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.413+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x56383779c4d0 0x563837b6c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.413+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/90683905 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b6bc20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.413+0000 7f6eb47cf740 1 -- 172.21.15.98:0/90683905 >> 172.21.15.98:0/90683905 conn(0x5638377b94e0 msgr2=0x563837b67430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.414+0000 7f6eb47cf740 1 -- 172.21.15.98:0/90683905 shutdown_connections 2024-04-25T16:34:50.416 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.414+0000 7f6eb47cf740 1 -- 172.21.15.98:0/90683905 wait complete. 2024-04-25T16:34:50.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.414+0000 7f6eb47cf740 1 -- start start 2024-04-25T16:34:50.421 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.414+0000 7f6eb47cf740 1 Processor -- start 2024-04-25T16:34:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 0x563837b75c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377c5390 0x563837b7cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x563837b6d700 con 0x5638377c5390 2024-04-25T16:34:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x563837a52dc0 con 0x5638377a46b0 2024-04-25T16:34:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x563837b6d880 con 0x56383779c4d0 2024-04-25T16:34:50.422 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6eb47cf740 10 client.0 _ll_register_callbacks cb 0x563837a6e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377c5390 0x563837b7cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377c5390 0x563837b7cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50688/0 (socket says 172.21.15.98:50688) 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea67fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 0x563837b75c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 -- 172.21.15.98:0/3745679940 learned_addr learned my addr 172.21.15.98:0/3745679940 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 -- 172.21.15.98:0/3745679940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 msgr2=0x563837b75740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 -- 172.21.15.98:0/3745679940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 msgr2=0x563837b75c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 0x563837b75c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 -- 172.21.15.98:0/3745679940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638377dd3f0 con 0x5638377c5390 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6ea77fe640 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377c5390 0x563837b7cb70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0011d10 tx=0x7f6ea000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e11f1aa17da2103d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6e86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6e86ffd640 1 -- 172.21.15.98:0/3745679940 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea000fe60 con 0x5638377c5390 2024-04-25T16:34:50.423 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6e86ffd640 1 -- 172.21.15.98:0/3745679940 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f6ea0004b20 con 0x5638377c5390 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.415+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3745679940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e900d0220 con 0x5638377c5390 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6e86ffd640 1 -- 172.21.15.98:0/3745679940 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6ea0025050 con 0x5638377c5390 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6eb47cf740 10 client.16233 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3745679940 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5638378a77d0 con 0x5638377c5390 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6e86ffd640 1 -- 172.21.15.98:0/3745679940 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e900d0220 con 0x5638377c5390 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6e86ffd640 1 -- 172.21.15.98:0/3745679940 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f6ea0019030 con 0x5638377c5390 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6eb47cf740 10 client.16233 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6eb47cf740 10 client.16233 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6eb47cf740 20 client.16233 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x563837783f60 0x563837b834f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6eb47cf740 4 client.16233 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3745679940 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563837ad9990 con 0x563837783f60 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.416+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x563837783f60 0x563837b834f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.417+0000 7f6e86ffd640 10 client.16233 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:34:50.424 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.417+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x563837783f60 0x563837b834f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f16de30b051bbe5 server_cookie=d19a2202f5f8c55b in_seq=0 out_seq=0 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6e86ffd640 1 -- 172.21.15.98:0/3745679940 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563837ad9990 con 0x563837783f60 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6e86ffd640 10 client.16233 handle_command_reply: tid=0 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 client.16233 shutdown 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3745679940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377c5390 msgr2=0x563837b7cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377c5390 0x563837b7cb70 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0011d10 tx=0x7f6ea000f790 comp rx=0 tx=0).stop 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3745679940 shutdown_connections 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x563837783f60 0x563837b834f0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377c5390 0x563837b7cb70 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377a46b0 0x563837b75c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6e86ffd640 0 client.16233 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3745679940 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3745679940 >> 172.21.15.98:0/3745679940 conn(0x5638377b94e0 msgr2=0x563837b62040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3745679940 shutdown_connections 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3745679940 wait complete. 2024-04-25T16:34:50.425 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.418+0000 7f6eb47cf740 20 client.16233 trim_cache size 0 max 16384 2024-04-25T16:34:50.440 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.431+0000 7f6eb47cf740 1 Processor -- start 2024-04-25T16:34:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.431+0000 7f6eb47cf740 1 -- start start 2024-04-25T16:34:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b75c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81aa7b0 con 0x5638377a46b0 2024-04-25T16:34:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81ac3b0 con 0x56383779c4d0 2024-04-25T16:34:50.441 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81aae80 con 0x5638377c5390 2024-04-25T16:34:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50704/0 (socket says 172.21.15.98:50704) 2024-04-25T16:34:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 -- 172.21.15.98:0/3465900043 learned_addr learned my addr 172.21.15.98:0/3465900043 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea77fe640 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b75c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 -- 172.21.15.98:0/3465900043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 msgr2=0x563837b75c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b75c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 -- 172.21.15.98:0/3465900043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56383779c4d0 msgr2=0x563837b75740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 -- 172.21.15.98:0/3465900043 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563837b71050 con 0x5638377a46b0 2024-04-25T16:34:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.432+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7ca90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009730 tx=0x7f6e98014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e4f62d9c28a3ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6e9d7fa640 1 -- 172.21.15.98:0/3465900043 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e98005020 con 0x5638377a46b0 2024-04-25T16:34:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6e9d7fa640 1 -- 172.21.15.98:0/3465900043 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e980047a0 con 0x5638377a46b0 2024-04-25T16:34:50.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6e9d7fa640 1 -- 172.21.15.98:0/3465900043 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e9801d730 con 0x5638377a46b0 2024-04-25T16:34:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3465900043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 msgr2=0x563837b7ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7ca90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009730 tx=0x7f6e98014000 comp rx=0 tx=0).stop 2024-04-25T16:34:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3465900043 shutdown_connections 2024-04-25T16:34:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x5638377c5390 0x563837b75c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7ca90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.433+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/3465900043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56383779c4d0 0x563837b75740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3465900043 >> 172.21.15.98:0/3465900043 conn(0x5638377b94e0 msgr2=0x563837b73d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3465900043 shutdown_connections 2024-04-25T16:34:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 -- 172.21.15.98:0/3465900043 wait complete. 2024-04-25T16:34:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 -- start start 2024-04-25T16:34:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 Processor -- start 2024-04-25T16:34:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b7a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377c5390 0x563837b7b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81abc20 con 0x5638377a46b0 2024-04-25T16:34:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81ab550 con 0x5638377c5390 2024-04-25T16:34:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea81a8790 con 0x56383779c4d0 2024-04-25T16:34:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6eb47cf740 10 client.0 _ll_register_callbacks cb 0x7f6ea0009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.434+0000 7f6ea6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b7a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea67fc640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b7a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40294/0 (socket says 172.21.15.98:40294) 2024-04-25T16:34:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/2953082021 learned_addr learned my addr 172.21.15.98:0/2953082021 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea77fe640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377c5390 0x563837b7b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/2953082021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377c5390 msgr2=0x563837b7b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377c5390 0x563837b7b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/2953082021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 msgr2=0x563837b7aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea6ffd640 1 -- 172.21.15.98:0/2953082021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563837b7cfd0 con 0x56383779c4d0 2024-04-25T16:34:50.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea77fe640 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377c5390 0x563837b7b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6ea6ffd640 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b7a550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6e900c7170 tx=0x7f6e900c3330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b88f1d1a954cb03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6e86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:34:50.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6e86ffd640 1 -- 172.21.15.98:0/2953082021 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e90049040 con 0x56383779c4d0 2024-04-25T16:34:50.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6e86ffd640 1 -- 172.21.15.98:0/2953082021 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f6e90003c80 con 0x56383779c4d0 2024-04-25T16:34:50.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6eb47cf740 1 -- 172.21.15.98:0/2953082021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e980047a0 con 0x56383779c4d0 2024-04-25T16:34:50.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6e86ffd640 1 -- 172.21.15.98:0/2953082021 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f6e900a9050 con 0x56383779c4d0 2024-04-25T16:34:50.448 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6eb47cf740 10 client.25675 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.435+0000 7f6eb47cf740 1 -- 172.21.15.98:0/2953082021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x563837beb970 con 0x56383779c4d0 2024-04-25T16:34:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6e86ffd640 1 -- 172.21.15.98:0/2953082021 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6e980047a0 con 0x56383779c4d0 2024-04-25T16:34:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6e86ffd640 1 -- 172.21.15.98:0/2953082021 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f6e900c76d0 con 0x56383779c4d0 2024-04-25T16:34:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6eb47cf740 10 client.25675 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6eb47cf740 10 client.25675 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-25T16:34:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6eb47cf740 20 client.25675 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.449 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5638377593a0 0x563837bee1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6eb47cf740 4 client.25675 mds_command: new command op to 24484 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6eb47cf740 1 -- 172.21.15.98:0/2953082021 --> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563837ad3800 con 0x5638377593a0 2024-04-25T16:34:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5638377593a0 0x563837bee1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.436+0000 7f6e86ffd640 10 client.25675 ms_handle_connect on v2:172.21.15.203:6838/2982027693 2024-04-25T16:34:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6ea67fc640 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5638377593a0 0x563837bee1d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=210458e144bd7934 server_cookie=2c9c97b506e62a61 in_seq=0 out_seq=0 2024-04-25T16:34:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6e86ffd640 1 -- 172.21.15.98:0/2953082021 <== mds.2 v2:172.21.15.203:6838/2982027693 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563837ad3800 con 0x5638377593a0 2024-04-25T16:34:50.450 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6e86ffd640 10 client.25675 handle_command_reply: tid=0 2024-04-25T16:34:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 client.25675 shutdown 2024-04-25T16:34:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 -- 172.21.15.98:0/2953082021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 msgr2=0x563837b7a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b7a550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6e900c7170 tx=0x7f6e900c3330 comp rx=0 tx=0).stop 2024-04-25T16:34:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 -- 172.21.15.98:0/2953082021 shutdown_connections 2024-04-25T16:34:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.203:6838/2982027693,v1:172.21.15.203:6839/2982027693] conn(0x5638377593a0 0x563837bee1d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5638377c5390 0x563837b7b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.451 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x5638377a46b0 0x563837b7aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6e86ffd640 0 client.25675 ms_handle_reset on v2:172.21.15.203:6838/2982027693 2024-04-25T16:34:50.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 --2- 172.21.15.98:0/2953082021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56383779c4d0 0x563837b7a550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 -- 172.21.15.98:0/2953082021 >> 172.21.15.98:0/2953082021 conn(0x5638377b94e0 msgr2=0x563837b75720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 -- 172.21.15.98:0/2953082021 shutdown_connections 2024-04-25T16:34:50.452 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 1 -- 172.21.15.98:0/2953082021 wait complete. 2024-04-25T16:34:50.453 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.437+0000 7f6eb47cf740 20 client.25675 trim_cache size 0 max 16384 2024-04-25T16:34:50.453 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 -- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e940afaf0 msgr2=0x7f6e940b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e940afaf0 0x7f6e940b2000 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f6ea819b220 tx=0x7f6e98005f90 comp rx=0 tx=0).stop 2024-04-25T16:34:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 -- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 msgr2=0x7f6ea819a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.455 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 0x7f6ea819a180 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6e90005760 tx=0x7f6e90005e20 comp rx=0 tx=0).stop 2024-04-25T16:34:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 -- 172.21.15.98:0/406187247 shutdown_connections 2024-04-25T16:34:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f6e940afaf0 0x7f6e940b2000 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea8109a20 0x7f6ea81a1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ea8104d20 0x7f6ea819a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 --2- 172.21.15.98:0/406187247 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6ea8103ae0 0x7f6ea819a180 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 -- 172.21.15.98:0/406187247 >> 172.21.15.98:0/406187247 conn(0x7f6ea80fecd0 msgr2=0x7f6ea8105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 -- 172.21.15.98:0/406187247 shutdown_connections 2024-04-25T16:34:50.456 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.438+0000 7f6ead842640 1 -- 172.21.15.98:0/406187247 wait complete. 2024-04-25T16:34:50.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:34:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 -- 172.21.15.98:0/3837631521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 msgr2=0x7f865010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 --2- 172.21.15.98:0/3837631521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 0x7f865010be40 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f8648009ca0 tx=0x7f864802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 -- 172.21.15.98:0/3837631521 shutdown_connections 2024-04-25T16:34:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 --2- 172.21.15.98:0/3837631521 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 0x7f865010be40 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 --2- 172.21.15.98:0/3837631521 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8650104d20 0x7f8650105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 --2- 172.21.15.98:0/3837631521 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8650103ae0 0x7f86501047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 -- 172.21.15.98:0/3837631521 >> 172.21.15.98:0/3837631521 conn(0x7f86500fecd0 msgr2=0x7f8650101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.599 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 -- 172.21.15.98:0/3837631521 shutdown_connections 2024-04-25T16:34:50.600 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.600+0000 7f8654884640 1 -- 172.21.15.98:0/3837631521 wait complete. 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.601+0000 7f8654884640 1 Processor -- start 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.601+0000 7f8654884640 1 -- start start 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.601+0000 7f8654884640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8650103ae0 0x7f865019a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.601+0000 7f8654884640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8650104d20 0x7f865019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.601+0000 7f8654884640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 0x7f86501a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.601+0000 7f8654884640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f865010f4f0 con 0x7f8650109a20 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.601+0000 7f8654884640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f865010f370 con 0x7f8650103ae0 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.601+0000 7f8654884640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f865010f670 con 0x7f8650104d20 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8650103ae0 0x7f865019a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8650103ae0 0x7f865019a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49686/0 (socket says 172.21.15.98:49686) 2024-04-25T16:34:50.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 -- 172.21.15.98:0/1196020817 learned_addr learned my addr 172.21.15.98:0/1196020817 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864ed9d640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 0x7f86501a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864dd9b640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8650104d20 0x7f865019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 -- 172.21.15.98:0/1196020817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8650104d20 msgr2=0x7f865019a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8650104d20 0x7f865019a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 -- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 msgr2=0x7f86501a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 0x7f86501a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 -- 172.21.15.98:0/1196020817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86501a1c30 con 0x7f8650103ae0 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864ed9d640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 0x7f86501a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.602+0000 7f864e59c640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8650103ae0 0x7f865019a0f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f864000e9f0 tx=0x7f864000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.603+0000 7f863f7fe640 1 -- 172.21.15.98:0/1196020817 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864000ccc0 con 0x7f8650103ae0 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.603+0000 7f863f7fe640 1 -- 172.21.15.98:0/1196020817 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86400044b0 con 0x7f8650103ae0 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.603+0000 7f863f7fe640 1 -- 172.21.15.98:0/1196020817 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640010430 con 0x7f8650103ae0 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.603+0000 7f8654884640 1 -- 172.21.15.98:0/1196020817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86501a1f20 con 0x7f8650103ae0 2024-04-25T16:34:50.603 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.603+0000 7f8654884640 1 -- 172.21.15.98:0/1196020817 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86501a23e0 con 0x7f8650103ae0 2024-04-25T16:34:50.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.606+0000 7f863f7fe640 1 -- 172.21.15.98:0/1196020817 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f8640003f90 con 0x7f8650103ae0 2024-04-25T16:34:50.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.606+0000 7f865b807740 1 Processor -- start 2024-04-25T16:34:50.605 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.606+0000 7f865b807740 1 -- start start 2024-04-25T16:34:50.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f865b807740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa04714390 0x55fa04ab6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f865b807740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046eb4d0 0x55fa04ab6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.607 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f865b807740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046f36b0 0x55fa04ab7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f865b807740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fa04a1ad80 con 0x55fa046eb4d0 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f865b807740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fa049d4a20 con 0x55fa046f36b0 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f865b807740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fa04a16d40 con 0x55fa04714390 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046eb4d0 0x55fa04ab6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864dd9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046f36b0 0x55fa04ab7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864dd9b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046f36b0 0x55fa04ab7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49690/0 (socket says 172.21.15.98:49690) 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864dd9b640 1 -- 172.21.15.98:0/3931353848 learned_addr learned my addr 172.21.15.98:0/3931353848 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864e59c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046eb4d0 0x55fa04ab6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50740/0 (socket says 172.21.15.98:50740) 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864ed9d640 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa04714390 0x55fa04ab6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f863f7fe640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86380afbf0 0x7f86380b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.608 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864dd9b640 1 -- 172.21.15.98:0/3931353848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa04714390 msgr2=0x55fa04ab6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864dd9b640 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa04714390 0x55fa04ab6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864dd9b640 1 -- 172.21.15.98:0/3931353848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046eb4d0 msgr2=0x55fa04ab6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864ed9d640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86380afbf0 0x7f86380b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.607+0000 7f864dd9b640 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046eb4d0 0x55fa04ab6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.608+0000 7f864dd9b640 1 -- 172.21.15.98:0/3931353848 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa049dfc30 con 0x55fa046f36b0 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.608+0000 7f864e59c640 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046eb4d0 0x55fa04ab6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.608+0000 7f863f7fe640 1 -- 172.21.15.98:0/1196020817 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f86400df290 con 0x7f8650103ae0 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.608+0000 7f864dd9b640 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046f36b0 0x55fa04ab7a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f864400bcb0 tx=0x7f86440107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d20709c6501f873 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.608+0000 7f864ed9d640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86380afbf0 0x7f86380b2100 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55fa04ab7860 tx=0x7f86480095c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.608+0000 7f863cff9640 1 -- 172.21.15.98:0/3931353848 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864401c480 con 0x55fa046f36b0 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.608+0000 7f863cff9640 1 -- 172.21.15.98:0/3931353848 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f864401cab0 con 0x55fa046f36b0 2024-04-25T16:34:50.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.608+0000 7f863cff9640 1 -- 172.21.15.98:0/3931353848 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8644011bd0 con 0x55fa046f36b0 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.609+0000 7f865b807740 1 -- 172.21.15.98:0/3931353848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046f36b0 msgr2=0x55fa04ab7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.609+0000 7f865b807740 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046f36b0 0x55fa04ab7a20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f864400bcb0 tx=0x7f86440107a0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.609+0000 7f865b807740 1 -- 172.21.15.98:0/3931353848 shutdown_connections 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.609+0000 7f865b807740 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046f36b0 0x55fa04ab7a20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.609+0000 7f865b807740 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046eb4d0 0x55fa04ab6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.609+0000 7f865b807740 1 --2- 172.21.15.98:0/3931353848 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa04714390 0x55fa04ab6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.609+0000 7f865b807740 1 -- 172.21.15.98:0/3931353848 >> 172.21.15.98:0/3931353848 conn(0x55fa047084e0 msgr2=0x55fa04ab2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.610+0000 7f865b807740 1 -- 172.21.15.98:0/3931353848 shutdown_connections 2024-04-25T16:34:50.614 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.610+0000 7f865b807740 1 -- 172.21.15.98:0/3931353848 wait complete. 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.610+0000 7f865b807740 1 -- start start 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.610+0000 7f865b807740 1 Processor -- start 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.610+0000 7f865b807740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046eb4d0 0x55fa04ac4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.610+0000 7f865b807740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046f36b0 0x55fa04ac4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.610+0000 7f865b807740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa04714390 0x55fa04acbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046f36b0 0x55fa04ac4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046eb4d0 0x55fa04ac4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864dd9b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046f36b0 0x55fa04ac4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50744/0 (socket says 172.21.15.98:50744) 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864e59c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046eb4d0 0x55fa04ac4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40354/0 (socket says 172.21.15.98:40354) 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f865b807740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55fa04ab8530 con 0x55fa046f36b0 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f865b807740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55fa049a1dc0 con 0x55fa04714390 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f865b807740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55fa04ab86b0 con 0x55fa046eb4d0 2024-04-25T16:34:50.615 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864ed9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa04714390 0x55fa04acbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864ed9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa04714390 0x55fa04acbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49702/0 (socket says 172.21.15.98:49702) 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864ed9d640 1 -- 172.21.15.98:0/1484955808 learned_addr learned my addr 172.21.15.98:0/1484955808 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f865b807740 10 client.0 _ll_register_callbacks cb 0x55fa049bd600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864dd9b640 1 -- 172.21.15.98:0/1484955808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046eb4d0 msgr2=0x55fa04ac4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864dd9b640 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046eb4d0 0x55fa04ac4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864dd9b640 1 -- 172.21.15.98:0/1484955808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa04714390 msgr2=0x55fa04acbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864dd9b640 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa04714390 0x55fa04acbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864dd9b640 1 -- 172.21.15.98:0/1484955808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa0472c3f0 con 0x55fa046f36b0 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864ed9d640 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa04714390 0x55fa04acbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864e59c640 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046eb4d0 0x55fa04ac4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f864dd9b640 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046f36b0 0x55fa04ac4d50 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f864400bcb0 tx=0x7f864401ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd92ec8cfbbe383 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f8635ffb640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:50.616 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f8635ffb640 1 -- 172.21.15.98:0/1484955808 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864402b060 con 0x55fa046f36b0 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f865b807740 1 -- 172.21.15.98:0/1484955808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f864401cab0 con 0x55fa046f36b0 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.611+0000 7f8635ffb640 1 -- 172.21.15.98:0/1484955808 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f864401c690 con 0x55fa046f36b0 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.612+0000 7f8635ffb640 1 -- 172.21.15.98:0/1484955808 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f8644032050 con 0x55fa046f36b0 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.612+0000 7f865b807740 10 client.16251 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.612+0000 7f865b807740 1 -- 172.21.15.98:0/1484955808 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fa047f67d0 con 0x55fa046f36b0 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.612+0000 7f8635ffb640 1 -- 172.21.15.98:0/1484955808 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f864401cab0 con 0x55fa046f36b0 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.613+0000 7f8635ffb640 1 -- 172.21.15.98:0/1484955808 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f8644005160 con 0x55fa046f36b0 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.613+0000 7f865b807740 10 client.16251 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.613+0000 7f865b807740 10 client.16251 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.613+0000 7f865b807740 20 client.16251 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.613+0000 7f865b807740 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55fa046d2f60 0x55fa04ad2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.617 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.613+0000 7f865b807740 4 client.16251 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.613+0000 7f865b807740 1 -- 172.21.15.98:0/1484955808 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fa04a28990 con 0x55fa046d2f60 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.613+0000 7f864e59c640 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55fa046d2f60 0x55fa04ad2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.614+0000 7f864e59c640 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55fa046d2f60 0x55fa04ad2820 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=13e400946c347e7b server_cookie=d531b74b912e097f in_seq=0 out_seq=0 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.614+0000 7f8635ffb640 10 client.16251 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f8635ffb640 1 -- 172.21.15.98:0/1484955808 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fa04a28990 con 0x55fa046d2f60 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f8635ffb640 10 client.16251 handle_command_reply: tid=0 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 client.16251 shutdown 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 -- 172.21.15.98:0/1484955808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046f36b0 msgr2=0x55fa04ac4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046f36b0 0x55fa04ac4d50 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f864400bcb0 tx=0x7f864401ce90 comp rx=0 tx=0).stop 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 -- 172.21.15.98:0/1484955808 shutdown_connections 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55fa046d2f60 0x55fa04ad2820 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa04714390 0x55fa04acbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f8635ffb640 0 client.16251 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:34:50.618 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa046f36b0 0x55fa04ac4d50 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 --2- 172.21.15.98:0/1484955808 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046eb4d0 0x55fa04ac4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 -- 172.21.15.98:0/1484955808 >> 172.21.15.98:0/1484955808 conn(0x55fa047084e0 msgr2=0x55fa04ab1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 -- 172.21.15.98:0/1484955808 shutdown_connections 2024-04-25T16:34:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 1 -- 172.21.15.98:0/1484955808 wait complete. 2024-04-25T16:34:50.619 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.615+0000 7f865b807740 20 client.16251 trim_cache size 0 max 16384 2024-04-25T16:34:50.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f865b807740 1 Processor -- start 2024-04-25T16:34:50.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f865b807740 1 -- start start 2024-04-25T16:34:50.631 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f865b807740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f865b807740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04acbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f865b807740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04ac3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.633 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f864e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f864e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49716/0 (socket says 172.21.15.98:49716) 2024-04-25T16:34:50.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f865b807740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86501aa7c0 con 0x55fa04714390 2024-04-25T16:34:50.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f865b807740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86501ac3c0 con 0x55fa046eb4d0 2024-04-25T16:34:50.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.632+0000 7f865b807740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86501aae90 con 0x55fa046f36b0 2024-04-25T16:34:50.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864e59c640 1 -- 172.21.15.98:0/2562318820 learned_addr learned my addr 172.21.15.98:0/2562318820 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.634 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864dd9b640 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04acbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864ed9d640 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04ac3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864e59c640 1 -- 172.21.15.98:0/2562318820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 msgr2=0x55fa04acbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864e59c640 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04acbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864e59c640 1 -- 172.21.15.98:0/2562318820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 msgr2=0x55fa04ac3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864e59c640 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04ac3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.635 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864e59c640 1 -- 172.21.15.98:0/2562318820 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fa04acc110 con 0x55fa046eb4d0 2024-04-25T16:34:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864ed9d640 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04ac3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.633+0000 7f864dd9b640 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04acbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.634+0000 7f864e59c640 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac30c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f86400a33e0 tx=0x7f86400035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=467847356b3644d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.636 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.634+0000 7f8637fff640 1 -- 172.21.15.98:0/2562318820 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864000ce60 con 0x55fa046eb4d0 2024-04-25T16:34:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.634+0000 7f8637fff640 1 -- 172.21.15.98:0/2562318820 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f86400aea90 con 0x55fa046eb4d0 2024-04-25T16:34:50.637 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.634+0000 7f8637fff640 1 -- 172.21.15.98:0/2562318820 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86400a3220 con 0x55fa046eb4d0 2024-04-25T16:34:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 -- 172.21.15.98:0/2562318820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 msgr2=0x55fa04ac30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac30c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f86400a33e0 tx=0x7f86400035f0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 -- 172.21.15.98:0/2562318820 shutdown_connections 2024-04-25T16:34:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04ac3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.644 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04acbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 --2- 172.21.15.98:0/2562318820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac30c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 -- 172.21.15.98:0/2562318820 >> 172.21.15.98:0/2562318820 conn(0x55fa047084e0 msgr2=0x55fa04abb190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 -- 172.21.15.98:0/2562318820 shutdown_connections 2024-04-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.635+0000 7f865b807740 1 -- 172.21.15.98:0/2562318820 wait complete. 2024-04-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 1 -- start start 2024-04-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 1 Processor -- start 2024-04-25T16:34:50.645 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04ac81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04acb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f86501abc30 con 0x55fa04714390 2024-04-25T16:34:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f86501ab560 con 0x55fa046eb4d0 2024-04-25T16:34:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f86501a87a0 con 0x55fa046f36b0 2024-04-25T16:34:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f865b807740 10 client.0 _ll_register_callbacks cb 0x7f86400e2b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.646 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f864ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04acb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f864e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f864dd9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04ac81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f864e59c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49722/0 (socket says 172.21.15.98:49722) 2024-04-25T16:34:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f864dd9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04ac81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40368/0 (socket says 172.21.15.98:40368) 2024-04-25T16:34:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.636+0000 7f864e59c640 1 -- 172.21.15.98:0/565346751 learned_addr learned my addr 172.21.15.98:0/565346751 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f864dd9b640 1 -- 172.21.15.98:0/565346751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 msgr2=0x55fa04ac7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f864dd9b640 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.647 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f864dd9b640 1 -- 172.21.15.98:0/565346751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 msgr2=0x55fa04acb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f864dd9b640 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04acb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f864dd9b640 1 -- 172.21.15.98:0/565346751 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fa04acb6c0 con 0x55fa046f36b0 2024-04-25T16:34:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f864ed9d640 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04acb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f864dd9b640 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04ac81c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f864400b3d0 tx=0x7f864401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b2f12bef2348c57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f8635ffb640 10 client.0 ms_handle_connect on v2:172.21.15.203:3300/0 2024-04-25T16:34:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f8635ffb640 1 -- 172.21.15.98:0/565346751 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f864402b040 con 0x55fa046f36b0 2024-04-25T16:34:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f865b807740 1 -- 172.21.15.98:0/565346751 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55fa04a22800 con 0x55fa046f36b0 2024-04-25T16:34:50.648 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f8635ffb640 1 -- 172.21.15.98:0/565346751 <== mon.1 v2:172.21.15.203:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f864401ca50 con 0x55fa046f36b0 2024-04-25T16:34:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f8635ffb640 1 -- 172.21.15.98:0/565346751 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f864403f050 con 0x55fa046f36b0 2024-04-25T16:34:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f865b807740 10 client.25693 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.637+0000 7f865b807740 1 -- 172.21.15.98:0/565346751 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55fa04b39400 con 0x55fa046f36b0 2024-04-25T16:34:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f8635ffb640 1 -- 172.21.15.98:0/565346751 <== mon.1 v2:172.21.15.203:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x55fa04a22800 con 0x55fa046f36b0 2024-04-25T16:34:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f8635ffb640 1 -- 172.21.15.98:0/565346751 <== mon.1 v2:172.21.15.203:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f8644003880 con 0x55fa046f36b0 2024-04-25T16:34:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f865b807740 10 client.25693 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.649 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f865b807740 10 client.25693 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-04-25T16:34:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f865b807740 20 client.25693 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f865b807740 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55fa046a83a0 0x55fa04b3bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f865b807740 4 client.25693 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f865b807740 1 -- 172.21.15.98:0/565346751 --> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fa0499d610 con 0x55fa046a83a0 2024-04-25T16:34:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.638+0000 7f864e59c640 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55fa046a83a0 0x55fa04b3bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.650 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f8635ffb640 10 client.25693 ms_handle_connect on v2:172.21.15.173:6836/311453164 2024-04-25T16:34:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f864e59c640 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55fa046a83a0 0x55fa04b3bd60 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=757fc06256634eb server_cookie=79f12faeab201aea in_seq=0 out_seq=0 2024-04-25T16:34:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f8635ffb640 1 -- 172.21.15.98:0/565346751 <== mds.1 v2:172.21.15.173:6836/311453164 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fa0499d610 con 0x55fa046a83a0 2024-04-25T16:34:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f8635ffb640 10 client.25693 handle_command_reply: tid=0 2024-04-25T16:34:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f865b807740 1 client.25693 shutdown 2024-04-25T16:34:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f865b807740 1 -- 172.21.15.98:0/565346751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 msgr2=0x55fa04ac81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f865b807740 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04ac81c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f864400b3d0 tx=0x7f864401c940 comp rx=0 tx=0).stop 2024-04-25T16:34:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f865b807740 1 -- 172.21.15.98:0/565346751 shutdown_connections 2024-04-25T16:34:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f865b807740 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.173:6836/311453164,v1:172.21.15.173:6837/311453164] conn(0x55fa046a83a0 0x55fa04b3bd60 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f865b807740 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55fa04714390 0x55fa04acb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f8635ffb640 0 client.25693 ms_handle_reset on v2:172.21.15.173:6836/311453164 2024-04-25T16:34:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.639+0000 7f865b807740 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55fa046f36b0 0x55fa04ac81c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f865b807740 1 --2- 172.21.15.98:0/565346751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55fa046eb4d0 0x55fa04ac7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.652 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f865b807740 1 -- 172.21.15.98:0/565346751 >> 172.21.15.98:0/565346751 conn(0x55fa047084e0 msgr2=0x55fa04ac30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f865b807740 1 -- 172.21.15.98:0/565346751 shutdown_connections 2024-04-25T16:34:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f865b807740 1 -- 172.21.15.98:0/565346751 wait complete. 2024-04-25T16:34:50.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f865b807740 20 client.25693 trim_cache size 0 max 16384 2024-04-25T16:34:50.653 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f8654884640 1 -- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86380afbf0 msgr2=0x7f86380b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f8654884640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86380afbf0 0x7f86380b2100 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55fa04ab7860 tx=0x7f86480095c0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f8654884640 1 -- 172.21.15.98:0/1196020817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8650103ae0 msgr2=0x7f865019a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.640+0000 7f8654884640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8650103ae0 0x7f865019a0f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f864000e9f0 tx=0x7f864000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.641+0000 7f8654884640 1 -- 172.21.15.98:0/1196020817 shutdown_connections 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.641+0000 7f8654884640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f86380afbf0 0x7f86380b2100 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.641+0000 7f8654884640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f8650109a20 0x7f86501a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.641+0000 7f8654884640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8650104d20 0x7f865019a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.641+0000 7f8654884640 1 --2- 172.21.15.98:0/1196020817 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8650103ae0 0x7f865019a0f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.641+0000 7f8654884640 1 -- 172.21.15.98:0/1196020817 >> 172.21.15.98:0/1196020817 conn(0x7f86500fecd0 msgr2=0x7f8650105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.641+0000 7f8654884640 1 -- 172.21.15.98:0/1196020817 shutdown_connections 2024-04-25T16:34:50.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.641+0000 7f8654884640 1 -- 172.21.15.98:0/1196020817 wait complete. 2024-04-25T16:34:50.657 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:34:50.802 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 -- 172.21.15.98:0/4182754984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bbc106c50 msgr2=0x7f7bbc1090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/4182754984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bbc106c50 0x7f7bbc1090f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009900 tx=0x7f7bac02a410 comp rx=0 tx=0).stop 2024-04-25T16:34:50.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 -- 172.21.15.98:0/4182754984 shutdown_connections 2024-04-25T16:34:50.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/4182754984 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc109780 0x7f7bbc10bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/4182754984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bbc106c50 0x7f7bbc1090f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/4182754984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bbc104260 0x7f7bbc106680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.803 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 -- 172.21.15.98:0/4182754984 >> 172.21.15.98:0/4182754984 conn(0x7f7bbc0fdc00 msgr2=0x7f7bbc100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 -- 172.21.15.98:0/4182754984 shutdown_connections 2024-04-25T16:34:50.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.800+0000 7f7bc3d96640 1 -- 172.21.15.98:0/4182754984 wait complete. 2024-04-25T16:34:50.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc3d96640 1 Processor -- start 2024-04-25T16:34:50.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc3d96640 1 -- start start 2024-04-25T16:34:50.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc3d96640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc104260 0x7f7bbc19a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc3d96640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bbc106c50 0x7f7bbc19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc3d96640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bbc109780 0x7f7bbc1a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc104260 0x7f7bbc19a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc104260 0x7f7bbc19a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50786/0 (socket says 172.21.15.98:50786) 2024-04-25T16:34:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc1b32640 1 -- 172.21.15.98:0/30325095 learned_addr learned my addr 172.21.15.98:0/30325095 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10f510 con 0x7f7bbc104260 2024-04-25T16:34:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10f390 con 0x7f7bbc109780 2024-04-25T16:34:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc10f690 con 0x7f7bbc106c50 2024-04-25T16:34:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc2333640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bbc109780 0x7f7bbc1a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.805 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.801+0000 7f7bc1331640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bbc106c50 0x7f7bbc19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.802+0000 7f7bc1b32640 1 -- 172.21.15.98:0/30325095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bbc106c50 msgr2=0x7f7bbc19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.802+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bbc106c50 0x7f7bbc19a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.802+0000 7f7bc1b32640 1 -- 172.21.15.98:0/30325095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bbc109780 msgr2=0x7f7bbc1a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.802+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bbc109780 0x7f7bbc1a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.802+0000 7f7bc1b32640 1 -- 172.21.15.98:0/30325095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bbc1a1d10 con 0x7f7bbc104260 2024-04-25T16:34:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.802+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc104260 0x7f7bbc19a1d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8009be0 tx=0x7f7bb800b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.806 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.802+0000 7f7bb2ffd640 1 -- 172.21.15.98:0/30325095 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb800bd00 con 0x7f7bbc104260 2024-04-25T16:34:50.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.803+0000 7f7bb2ffd640 1 -- 172.21.15.98:0/30325095 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7bb80054b0 con 0x7f7bbc104260 2024-04-25T16:34:50.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.803+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bbc1020b0 con 0x7f7bbc104260 2024-04-25T16:34:50.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.803+0000 7f7bb2ffd640 1 -- 172.21.15.98:0/30325095 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb8019410 con 0x7f7bbc104260 2024-04-25T16:34:50.807 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.803+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bbc102600 con 0x7f7bbc104260 2024-04-25T16:34:50.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.807+0000 7f7bb2ffd640 1 -- 172.21.15.98:0/30325095 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f7bb8004f80 con 0x7f7bbc104260 2024-04-25T16:34:50.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bcac48740 1 Processor -- start 2024-04-25T16:34:50.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bb2ffd640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7ba80afc40 0x7f7ba80b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.817 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bb2ffd640 1 -- 172.21.15.98:0/30325095 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7bb80e2980 con 0x7f7bbc104260 2024-04-25T16:34:50.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bc2333640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7ba80afc40 0x7f7ba80b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bcac48740 1 -- start start 2024-04-25T16:34:50.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa4f390 0x562ebfdfa020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa264d0 0x562ebfdfa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.818 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdfb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bcac48740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ebfd55d80 con 0x562ebfa2e6b0 2024-04-25T16:34:50.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bcac48740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562ebfd0fa20 con 0x562ebfa4f390 2024-04-25T16:34:50.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bcac48740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562ebfd51d40 con 0x562ebfa264d0 2024-04-25T16:34:50.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa4f390 0x562ebfdfa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa4f390 0x562ebfdfa020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49760/0 (socket says 172.21.15.98:49760) 2024-04-25T16:34:50.819 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.808+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2789828931 learned_addr learned my addr 172.21.15.98:0/2789828931 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.820 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc2333640 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdfb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa264d0 0x562ebfdfa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2789828931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa264d0 msgr2=0x562ebfdfa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa264d0 0x562ebfdfa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc2333640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7ba80afc40 0x7f7ba80b2150 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005fd0 tx=0x7f7bb4009500 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2789828931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 msgr2=0x562ebfdfb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdfb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.821 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2789828931 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ebfd1ac30 con 0x562ebfa4f390 2024-04-25T16:34:50.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa4f390 0x562ebfdfa020 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80a6b30 tx=0x7f7bb80b3310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3926ba8bd6a84e64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7ba7fff640 1 -- 172.21.15.98:0/2789828931 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb80aaab0 con 0x562ebfa4f390 2024-04-25T16:34:50.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7ba7fff640 1 -- 172.21.15.98:0/2789828931 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7bb80aee60 con 0x562ebfa4f390 2024-04-25T16:34:50.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.809+0000 7f7ba7fff640 1 -- 172.21.15.98:0/2789828931 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb80a6d40 con 0x562ebfa4f390 2024-04-25T16:34:50.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.810+0000 7f7bcac48740 1 -- 172.21.15.98:0/2789828931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa4f390 msgr2=0x562ebfdfa020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.810+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa4f390 0x562ebfdfa020 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80a6b30 tx=0x7f7bb80b3310 comp rx=0 tx=0).stop 2024-04-25T16:34:50.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.810+0000 7f7bcac48740 1 -- 172.21.15.98:0/2789828931 shutdown_connections 2024-04-25T16:34:50.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.810+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdfb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.810+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa264d0 0x562ebfdfa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.810+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2789828931 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa4f390 0x562ebfdfa020 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.810+0000 7f7bcac48740 1 -- 172.21.15.98:0/2789828931 >> 172.21.15.98:0/2789828931 conn(0x562ebfa434e0 msgr2=0x562ebfded020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.810+0000 7f7bcac48740 1 -- 172.21.15.98:0/2789828931 shutdown_connections 2024-04-25T16:34:50.823 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 -- 172.21.15.98:0/2789828931 wait complete. 2024-04-25T16:34:50.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 -- start start 2024-04-25T16:34:50.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 Processor -- start 2024-04-25T16:34:50.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa264d0 0x562ebfe03d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa2e6b0 0x562ebfe04280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfe0b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x562ebfdfbd40 con 0x562ebfa264d0 2024-04-25T16:34:50.824 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x562ebfcdcdc0 con 0x562ebfa2e6b0 2024-04-25T16:34:50.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x562ebfdfbec0 con 0x562ebfa4f390 2024-04-25T16:34:50.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.811+0000 7f7bcac48740 10 client.0 _ll_register_callbacks cb 0x562ebfcf8600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa264d0 0x562ebfe03d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc2333640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfe0b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc2333640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfe0b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40402/0 (socket says 172.21.15.98:40402) 2024-04-25T16:34:50.825 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc2333640 1 -- 172.21.15.98:0/764248182 learned_addr learned my addr 172.21.15.98:0/764248182 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc1331640 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa2e6b0 0x562ebfe04280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc1331640 1 -- 172.21.15.98:0/764248182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 msgr2=0x562ebfe0b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc1331640 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfe0b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc1331640 1 -- 172.21.15.98:0/764248182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa264d0 msgr2=0x562ebfe03d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.826 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc1331640 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa264d0 0x562ebfe03d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc1331640 1 -- 172.21.15.98:0/764248182 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ebfa673f0 con 0x562ebfa2e6b0 2024-04-25T16:34:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc2333640 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfe0b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.812+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa264d0 0x562ebfe03d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:34:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7bc1331640 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa2e6b0 0x562ebfe04280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7bac02a930 tx=0x7f7bac004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26bc90b83090a838 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.827 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7ba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7ba57fa640 1 -- 172.21.15.98:0/764248182 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac035030 con 0x562ebfa2e6b0 2024-04-25T16:34:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7ba57fa640 1 -- 172.21.15.98:0/764248182 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7bac0037f0 con 0x562ebfa2e6b0 2024-04-25T16:34:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7bcac48740 1 -- 172.21.15.98:0/764248182 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bb80aee60 con 0x562ebfa2e6b0 2024-04-25T16:34:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7ba57fa640 1 -- 172.21.15.98:0/764248182 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7bac0039f0 con 0x562ebfa2e6b0 2024-04-25T16:34:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7bcac48740 10 client.25784 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7bcac48740 1 -- 172.21.15.98:0/764248182 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562ebfb317d0 con 0x562ebfa2e6b0 2024-04-25T16:34:50.828 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.813+0000 7f7ba57fa640 1 -- 172.21.15.98:0/764248182 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7bb80aee60 con 0x562ebfa2e6b0 2024-04-25T16:34:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7ba57fa640 1 -- 172.21.15.98:0/764248182 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f7bac03c440 con 0x562ebfa2e6b0 2024-04-25T16:34:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7bcac48740 10 client.25784 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7bcac48740 10 client.25784 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7bcac48740 20 client.25784 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7bcac48740 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562ebfa0df60 0x562ebfe11ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.829 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7bcac48740 4 client.25784 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7bcac48740 1 -- 172.21.15.98:0/764248182 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ebfd63990 con 0x562ebfa0df60 2024-04-25T16:34:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562ebfa0df60 0x562ebfe11ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562ebfa0df60 0x562ebfe11ae0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d24ebf5d16d83d8 server_cookie=b6f495ce0b5e8067 in_seq=0 out_seq=0 2024-04-25T16:34:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.814+0000 7f7ba57fa640 10 client.25784 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7ba57fa640 1 -- 172.21.15.98:0/764248182 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562ebfd63990 con 0x562ebfa0df60 2024-04-25T16:34:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7ba57fa640 10 client.25784 handle_command_reply: tid=0 2024-04-25T16:34:50.830 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7bcac48740 1 client.25784 shutdown 2024-04-25T16:34:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7bcac48740 1 -- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa2e6b0 msgr2=0x562ebfe04280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7bcac48740 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa2e6b0 0x562ebfe04280 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f7bac02a930 tx=0x7f7bac004250 comp rx=0 tx=0).stop 2024-04-25T16:34:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7bcac48740 1 -- 172.21.15.98:0/764248182 shutdown_connections 2024-04-25T16:34:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7bcac48740 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562ebfa0df60 0x562ebfe11ae0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7bcac48740 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfe0b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.831 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7bcac48740 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa2e6b0 0x562ebfe04280 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.815+0000 7f7bcac48740 1 --2- 172.21.15.98:0/764248182 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa264d0 0x562ebfe03d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.816+0000 7f7bcac48740 1 -- 172.21.15.98:0/764248182 >> 172.21.15.98:0/764248182 conn(0x562ebfa434e0 msgr2=0x562ebfdeff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.816+0000 7f7ba57fa640 0 client.25784 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.816+0000 7f7bcac48740 1 -- 172.21.15.98:0/764248182 shutdown_connections 2024-04-25T16:34:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.816+0000 7f7bcac48740 1 -- 172.21.15.98:0/764248182 wait complete. 2024-04-25T16:34:50.832 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.816+0000 7f7bcac48740 20 client.25784 trim_cache size 0 max 16384 2024-04-25T16:34:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.832+0000 7f7bcac48740 1 Processor -- start 2024-04-25T16:34:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.832+0000 7f7bcac48740 1 -- start start 2024-04-25T16:34:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfde7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdeca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfdecf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bcac48740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1aa6d0 con 0x562ebfa2e6b0 2024-04-25T16:34:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bcac48740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1ac2d0 con 0x562ebfa264d0 2024-04-25T16:34:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bcac48740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1aada0 con 0x562ebfa4f390 2024-04-25T16:34:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfde7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfde7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49782/0 (socket says 172.21.15.98:49782) 2024-04-25T16:34:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2129030747 learned_addr learned my addr 172.21.15.98:0/2129030747 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdeca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc2333640 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfdecf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1331640 1 -- 172.21.15.98:0/2129030747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 msgr2=0x562ebfdecf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfdecf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1331640 1 -- 172.21.15.98:0/2129030747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 msgr2=0x562ebfde7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfde7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1331640 1 -- 172.21.15.98:0/2129030747 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ebfdf9ab0 con 0x562ebfa2e6b0 2024-04-25T16:34:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.833+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfde7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.834+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdeca50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7bac004650 tx=0x7f7bac001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5fed376820ca09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.834+0000 7f7ba77fe640 1 -- 172.21.15.98:0/2129030747 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac035070 con 0x562ebfa2e6b0 2024-04-25T16:34:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.834+0000 7f7ba77fe640 1 -- 172.21.15.98:0/2129030747 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7bac040070 con 0x562ebfa2e6b0 2024-04-25T16:34:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.834+0000 7f7ba77fe640 1 -- 172.21.15.98:0/2129030747 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bac031760 con 0x562ebfa2e6b0 2024-04-25T16:34:50.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.835+0000 7f7bcac48740 1 -- 172.21.15.98:0/2129030747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 msgr2=0x562ebfdeca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.835+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdeca50 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7bac004650 tx=0x7f7bac001e70 comp rx=0 tx=0).stop 2024-04-25T16:34:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.835+0000 7f7bcac48740 1 -- 172.21.15.98:0/2129030747 shutdown_connections 2024-04-25T16:34:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.835+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa4f390 0x562ebfdecf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.835+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa2e6b0 0x562ebfdeca50 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.835+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2129030747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfde7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.835+0000 7f7bcac48740 1 -- 172.21.15.98:0/2129030747 >> 172.21.15.98:0/2129030747 conn(0x562ebfa434e0 msgr2=0x562ebfdfeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.835+0000 7f7bcac48740 1 -- 172.21.15.98:0/2129030747 shutdown_connections 2024-04-25T16:34:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.836+0000 7f7bcac48740 1 -- 172.21.15.98:0/2129030747 wait complete. 2024-04-25T16:34:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.836+0000 7f7bcac48740 1 -- start start 2024-04-25T16:34:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.836+0000 7f7bcac48740 1 Processor -- start 2024-04-25T16:34:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfe07d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa2e6b0 0x562ebfe08240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bcac48740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa4f390 0x562ebfe09090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bcac48740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1abb40 con 0x562ebfa4f390 2024-04-25T16:34:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bcac48740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1ab470 con 0x562ebfa264d0 2024-04-25T16:34:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bcac48740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7bbc1a86b0 con 0x562ebfa2e6b0 2024-04-25T16:34:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfe07d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bcac48740 10 client.0 _ll_register_callbacks cb 0x7f7bb40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:50.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bc1b32640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfe07d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.98:49792/0 (socket says 172.21.15.98:49792) 2024-04-25T16:34:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2230927633 learned_addr learned my addr 172.21.15.98:0/2230927633 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bc2333640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa4f390 0x562ebfe09090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.837+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa2e6b0 0x562ebfe08240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2230927633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa2e6b0 msgr2=0x562ebfe08240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa2e6b0 0x562ebfe08240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2230927633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa4f390 msgr2=0x562ebfe09090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:50.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa4f390 0x562ebfe09090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bc1b32640 1 -- 172.21.15.98:0/2230927633 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ebfe09650 con 0x562ebfa264d0 2024-04-25T16:34:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa2e6b0 0x562ebfe08240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bc2333640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa4f390 0x562ebfe09090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bc1b32640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfe07d00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80ab360 tx=0x7f7bb80aac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f3d20250f20461a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7ba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:50.852 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7ba57fa640 1 -- 172.21.15.98:0/2230927633 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb80b3910 con 0x562ebfa264d0 2024-04-25T16:34:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7ba57fa640 1 -- 172.21.15.98:0/2230927633 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f7bb8005270 con 0x562ebfa264d0 2024-04-25T16:34:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.838+0000 7f7bcac48740 1 -- 172.21.15.98:0/2230927633 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7bac040070 con 0x562ebfa264d0 2024-04-25T16:34:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.839+0000 7f7ba57fa640 1 -- 172.21.15.98:0/2230927633 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f7bb80aee60 con 0x562ebfa264d0 2024-04-25T16:34:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.839+0000 7f7bcac48740 10 client.25790 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.839+0000 7f7bcac48740 1 -- 172.21.15.98:0/2230927633 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x562ebfe0a9a0 con 0x562ebfa264d0 2024-04-25T16:34:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.839+0000 7f7ba57fa640 1 -- 172.21.15.98:0/2230927633 <== mon.2 v2:172.21.15.173:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7bac040070 con 0x562ebfa264d0 2024-04-25T16:34:50.853 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.839+0000 7f7ba57fa640 1 -- 172.21.15.98:0/2230927633 <== mon.2 v2:172.21.15.173:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f7bb80a9d10 con 0x562ebfa264d0 2024-04-25T16:34:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7bcac48740 10 client.25790 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7bcac48740 10 client.25790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7bcac48740 20 client.25790 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562ebf9e33a0 0x562ebfe7c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7bcac48740 4 client.25790 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:34:50.854 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7bcac48740 1 -- 172.21.15.98:0/2230927633 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562ebfd5d800 con 0x562ebf9e33a0 2024-04-25T16:34:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562ebf9e33a0 0x562ebfe7c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7bc1331640 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562ebf9e33a0 0x562ebfe7c130 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae423cf5c459ff66 server_cookie=8d581018ebb3cec1 in_seq=0 out_seq=0 2024-04-25T16:34:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.840+0000 7f7ba57fa640 10 client.25790 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7ba57fa640 1 -- 172.21.15.98:0/2230927633 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562ebfd5d800 con 0x562ebf9e33a0 2024-04-25T16:34:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7ba57fa640 10 client.25790 handle_command_reply: tid=0 2024-04-25T16:34:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7bcac48740 1 client.25790 shutdown 2024-04-25T16:34:50.855 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7bcac48740 1 -- 172.21.15.98:0/2230927633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 msgr2=0x562ebfe07d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfe07d00 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7bb80ab360 tx=0x7f7bb80aac00 comp rx=0 tx=0).stop 2024-04-25T16:34:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7bcac48740 1 -- 172.21.15.98:0/2230927633 shutdown_connections 2024-04-25T16:34:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x562ebf9e33a0 0x562ebfe7c130 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x562ebfa4f390 0x562ebfe09090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.843+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x562ebfa2e6b0 0x562ebfe08240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.856 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.843+0000 7f7bcac48740 1 --2- 172.21.15.98:0/2230927633 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x562ebfa264d0 0x562ebfe07d00 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.843+0000 7f7bcac48740 1 -- 172.21.15.98:0/2230927633 >> 172.21.15.98:0/2230927633 conn(0x562ebfa434e0 msgr2=0x562ebfdec9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.842+0000 7f7ba57fa640 0 client.25790 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.843+0000 7f7bcac48740 1 -- 172.21.15.98:0/2230927633 shutdown_connections 2024-04-25T16:34:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.843+0000 7f7bcac48740 1 -- 172.21.15.98:0/2230927633 wait complete. 2024-04-25T16:34:50.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.843+0000 7f7bcac48740 20 client.25790 trim_cache size 0 max 16384 2024-04-25T16:34:50.858 INFO:teuthology.orchestra.run.smithi098.stdout:[] 2024-04-25T16:34:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.843+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7ba80afc40 msgr2=0x7f7ba80b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.843+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7ba80afc40 0x7f7ba80b2150 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4005fd0 tx=0x7f7bb4009500 comp rx=0 tx=0).stop 2024-04-25T16:34:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc104260 msgr2=0x7f7bbc19a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc104260 0x7f7bbc19a1d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7bb8009be0 tx=0x7f7bb800b2c0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 shutdown_connections 2024-04-25T16:34:50.861 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7ba80afc40 0x7f7ba80b2150 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7bbc109780 0x7f7bbc1a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bbc106c50 0x7f7bbc19a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 --2- 172.21.15.98:0/30325095 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7bbc104260 0x7f7bbc19a1d0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 >> 172.21.15.98:0/30325095 conn(0x7f7bbc0fdc00 msgr2=0x7f7bbc104e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 shutdown_connections 2024-04-25T16:34:50.862 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:50.844+0000 7f7bc3d96640 1 -- 172.21.15.98:0/30325095 wait complete. 2024-04-25T16:34:51.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:50 smithi203 ceph-mon[33115]: pgmap v426: 129 pgs: 129 active+clean; 206 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 296 op/s 2024-04-25T16:34:51.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:50 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:34:51.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:50 smithi098 ceph-mon[29162]: pgmap v426: 129 pgs: 129 active+clean; 206 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 296 op/s 2024-04-25T16:34:51.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:50 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:34:51.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:50 smithi173 ceph-mon[32556]: pgmap v426: 129 pgs: 129 active+clean; 206 MiB data, 966 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 296 op/s 2024-04-25T16:34:51.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:50 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:34:51.864 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:34:51.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:34:51.864 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:34:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 -- 172.21.15.98:0/1136933132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8104d40 msgr2=0x7fa6d8105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/1136933132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8104d40 0x7fa6d8105150 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009900 tx=0x7fa6c802a410 comp rx=0 tx=0).stop 2024-04-25T16:34:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 -- 172.21.15.98:0/1136933132 shutdown_connections 2024-04-25T16:34:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/1136933132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6d8109a40 0x7fa6d810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/1136933132 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8104d40 0x7fa6d8105150 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/1136933132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d8103b00 0x7fa6d8104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 -- 172.21.15.98:0/1136933132 >> 172.21.15.98:0/1136933132 conn(0x7fa6d80fecf0 msgr2=0x7fa6d81011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 -- 172.21.15.98:0/1136933132 shutdown_connections 2024-04-25T16:34:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.004+0000 7fa6e00e0640 1 -- 172.21.15.98:0/1136933132 wait complete. 2024-04-25T16:34:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6e00e0640 1 Processor -- start 2024-04-25T16:34:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6e00e0640 1 -- start start 2024-04-25T16:34:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6e00e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6d8103b00 0x7fa6d819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6e00e0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d8104d40 0x7fa6d819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6e00e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8109a40 0x7fa6d81a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6e00e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d810f4e0 con 0x7fa6d8109a40 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6e00e0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d810f360 con 0x7fa6d8104d40 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6e00e0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d810f660 con 0x7fa6d8103b00 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6de67d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8109a40 0x7fa6d81a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.005+0000 7fa6de67d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8109a40 0x7fa6d81a1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50816/0 (socket says 172.21.15.98:50816) 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6de67d640 1 -- 172.21.15.98:0/2068059009 learned_addr learned my addr 172.21.15.98:0/2068059009 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d8104d40 0x7fa6d819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6d8103b00 0x7fa6d819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6dd67b640 1 -- 172.21.15.98:0/2068059009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6d8103b00 msgr2=0x7fa6d819a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6d8103b00 0x7fa6d819a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6dd67b640 1 -- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8109a40 msgr2=0x7fa6d81a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8109a40 0x7fa6d81a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6dd67b640 1 -- 172.21.15.98:0/2068059009 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6d81a1c80 con 0x7fa6d8104d40 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6de67d640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8109a40 0x7fa6d81a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d8104d40 0x7fa6d819a680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8005760 tx=0x7fa6c8005e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6ceffd640 1 -- 172.21.15.98:0/2068059009 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c803f070 con 0x7fa6d8104d40 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6ceffd640 1 -- 172.21.15.98:0/2068059009 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6c8003a80 con 0x7fa6d8104d40 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6e00e0640 1 -- 172.21.15.98:0/2068059009 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6d81a1f10 con 0x7fa6d8104d40 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6ceffd640 1 -- 172.21.15.98:0/2068059009 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8035070 con 0x7fa6d8104d40 2024-04-25T16:34:52.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.006+0000 7fa6e00e0640 1 -- 172.21.15.98:0/2068059009 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6d81a2320 con 0x7fa6d8104d40 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6ceffd640 1 -- 172.21.15.98:0/2068059009 <== mon.2 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fa6c8043400 con 0x7fa6d8104d40 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6e6f8a740 1 Processor -- start 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6e6f8a740 1 -- start start 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aec4550 0x55f41b292e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aecc730 0x55f41b299d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f41b1f3b00 con 0x55f41aecc730 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f41b18f920 con 0x55f41aec4550 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.010+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f41b1efac0 con 0x55f41aeed410 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6de67d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6de67d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2928b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40438/0 (socket says 172.21.15.98:40438) 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6dd67b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aecc730 0x55f41b299d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6de67d640 1 -- 172.21.15.98:0/1142608391 learned_addr learned my addr 172.21.15.98:0/1142608391 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aec4550 0x55f41b292e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6ceffd640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6c40afaf0 0x7fa6c40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6dde7c640 1 -- 172.21.15.98:0/1142608391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 msgr2=0x55f41b2928b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2928b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6dde7c640 1 -- 172.21.15.98:0/1142608391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aecc730 msgr2=0x55f41b299d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aecc730 0x55f41b299d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6dde7c640 1 -- 172.21.15.98:0/1142608391 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f41afcd7e0 con 0x55f41aec4550 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6de67d640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6c40afaf0 0x7fa6c40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.011+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aecc730 0x55f41b299d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.012+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aec4550 0x55f41b292e30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa6d81033d0 tx=0x7fa6d4011440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a980093c83bf5c0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.012+0000 7fa6ceffd640 1 -- 172.21.15.98:0/2068059009 <== mon.2 v2:172.21.15.173:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa6c80ff0b0 con 0x7fa6d8104d40 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.012+0000 7fa6c3fff640 1 -- 172.21.15.98:0/1142608391 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4004230 con 0x55f41aec4550 2024-04-25T16:34:52.019 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.012+0000 7fa6de67d640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6c40afaf0 0x7fa6c40b2000 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55f41b293a80 tx=0x7fa6d0004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.012+0000 7fa6c3fff640 1 -- 172.21.15.98:0/1142608391 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6d40043d0 con 0x55f41aec4550 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.012+0000 7fa6c3fff640 1 -- 172.21.15.98:0/1142608391 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4004e50 con 0x55f41aec4550 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1142608391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aec4550 msgr2=0x55f41b292e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aec4550 0x55f41b292e30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa6d81033d0 tx=0x7fa6d4011440 comp rx=0 tx=0).stop 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1142608391 shutdown_connections 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aecc730 0x55f41b299d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aec4550 0x55f41b292e30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1142608391 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2928b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1142608391 >> 172.21.15.98:0/1142608391 conn(0x55f41aee1560 msgr2=0x55f41b289e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1142608391 shutdown_connections 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.013+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1142608391 wait complete. 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.014+0000 7fa6e6f8a740 1 -- start start 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.014+0000 7fa6e6f8a740 1 Processor -- start 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.014+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b29c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.020 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.014+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b29cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.014+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.014+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55f41b29a780 con 0x55f41aec4550 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.014+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55f41b18d5d0 con 0x55f41aecc730 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55f41b29a900 con 0x55f41aeed410 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b29c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b29c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50834/0 (socket says 172.21.15.98:50834) 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 -- 172.21.15.98:0/1339719722 learned_addr learned my addr 172.21.15.98:0/1339719722 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6e6f8a740 10 client.0 _ll_register_callbacks cb 0x55f41b17b080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:52.021 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6de67d640 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b29cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 -- 172.21.15.98:0/1339719722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 msgr2=0x55f41b2a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 -- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 msgr2=0x55f41b29cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b29cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 -- 172.21.15.98:0/1339719722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f41b02df70 con 0x55f41aec4550 2024-04-25T16:34:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b29cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:52.022 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b29c700 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4004a20 tx=0x7fa6d400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e883d1c8aa9ef8b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.015+0000 7fa6c17fa640 1 -- 172.21.15.98:0/1339719722 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4028020 con 0x55f41aec4550 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1339719722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6d40043d0 con 0x55f41aec4550 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6c17fa640 1 -- 172.21.15.98:0/1339719722 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa6d4019640 con 0x55f41aec4550 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6c17fa640 1 -- 172.21.15.98:0/1339719722 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa6d4033050 con 0x55f41aec4550 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 10 client.16287 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1339719722 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55f41aec5170 con 0x55f41aec4550 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6c17fa640 1 -- 172.21.15.98:0/1339719722 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6d40043d0 con 0x55f41aec4550 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6c17fa640 1 -- 172.21.15.98:0/1339719722 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fa6d4024530 con 0x55f41aec4550 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 10 client.16287 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 10 client.16287 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 20 client.16287 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55f41aeac1f0 0x55f41b2aa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 4 client.16287 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.016+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1339719722 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f41b201710 con 0x55f41aeac1f0 2024-04-25T16:34:52.023 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.017+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55f41aeac1f0 0x55f41b2aa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.017+0000 7fa6c17fa640 10 client.16287 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.017+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55f41aeac1f0 0x55f41b2aa4a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2505a91c192cdf5e server_cookie=59a52d00efc7e7d7 in_seq=0 out_seq=0 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.018+0000 7fa6c17fa640 1 -- 172.21.15.98:0/1339719722 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f41b201710 con 0x55f41aeac1f0 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.018+0000 7fa6c17fa640 10 client.16287 handle_command_reply: tid=0 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.018+0000 7fa6e6f8a740 1 client.16287 shutdown 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.018+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1339719722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 msgr2=0x55f41b29c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.018+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b29c700 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fa6d4004a20 tx=0x7fa6d400ec20 comp rx=0 tx=0).stop 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1339719722 shutdown_connections 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55f41aeac1f0 0x55f41b2aa4a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeed410 0x55f41b2a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b29cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6c17fa640 0 client.16287 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/1339719722 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b29c700 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.024 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1339719722 >> 172.21.15.98:0/1339719722 conn(0x55f41aee1560 msgr2=0x55f41b288ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1339719722 shutdown_connections 2024-04-25T16:34:52.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/1339719722 wait complete. 2024-04-25T16:34:52.025 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.019+0000 7fa6e6f8a740 20 client.16287 trim_cache size 0 max 16384 2024-04-25T16:34:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6e6f8a740 1 Processor -- start 2024-04-25T16:34:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6e6f8a740 1 -- start start 2024-04-25T16:34:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeac1f0 0x55f41b29c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b2a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.043 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81aa7a0 con 0x55f41aec4550 2024-04-25T16:34:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81ac3a0 con 0x55f41aecc730 2024-04-25T16:34:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81aae70 con 0x55f41aeac1f0 2024-04-25T16:34:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6dd67b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b2a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6dde7c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeac1f0 0x55f41b29c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6dd67b640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b2a3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50844/0 (socket says 172.21.15.98:50844) 2024-04-25T16:34:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6de67d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.044 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6dd67b640 1 -- 172.21.15.98:0/3466845707 learned_addr learned my addr 172.21.15.98:0/3466845707 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.042+0000 7fa6dde7c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeac1f0 0x55f41b29c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40454/0 (socket says 172.21.15.98:40454) 2024-04-25T16:34:52.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6de67d640 1 -- 172.21.15.98:0/3466845707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeac1f0 msgr2=0x55f41b29c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6de67d640 1 --2- 172.21.15.98:0/3466845707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeac1f0 0x55f41b29c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6de67d640 1 -- 172.21.15.98:0/3466845707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 msgr2=0x55f41b2a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6de67d640 1 --2- 172.21.15.98:0/3466845707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b2a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.045 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6de67d640 1 -- 172.21.15.98:0/3466845707 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f41b29d300 con 0x55f41aecc730 2024-04-25T16:34:52.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/3466845707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b2a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:52.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6de67d640 1 --2- 172.21.15.98:0/3466845707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a4000 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000f7b0 tx=0x7fa6d001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2580a55dabcf4f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6c37fe640 1 -- 172.21.15.98:0/3466845707 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d001ce30 con 0x55f41aecc730 2024-04-25T16:34:52.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6c37fe640 1 -- 172.21.15.98:0/3466845707 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa6d002d070 con 0x55f41aecc730 2024-04-25T16:34:52.046 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.043+0000 7fa6c37fe640 1 -- 172.21.15.98:0/3466845707 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d000e610 con 0x55f41aecc730 2024-04-25T16:34:52.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.044+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/3466845707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 msgr2=0x55f41b2a4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.051 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.044+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/3466845707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a4000 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa6d000f7b0 tx=0x7fa6d001a000 comp rx=0 tx=0).stop 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.044+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/3466845707 shutdown_connections 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.044+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/3466845707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a4000 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.044+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/3466845707 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aec4550 0x55f41b2a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.044+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/3466845707 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aeac1f0 0x55f41b29c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.044+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/3466845707 >> 172.21.15.98:0/3466845707 conn(0x55f41aee1560 msgr2=0x55f41b29c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.044+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/3466845707 shutdown_connections 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/3466845707 wait complete. 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 -- start start 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 Processor -- start 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aeac1f0 0x55f41b2a43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aec4550 0x55f41b2a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.052 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81abc10 con 0x55f41aeac1f0 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81ab540 con 0x55f41aecc730 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6e6f8a740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fa6d81a8780 con 0x55f41aec4550 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6dde7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aeac1f0 0x55f41b2a43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.045+0000 7fa6dde7c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aeac1f0 0x55f41b2a43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50858/0 (socket says 172.21.15.98:50858) 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6dde7c640 1 -- 172.21.15.98:0/368050347 learned_addr learned my addr 172.21.15.98:0/368050347 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6de67d640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6de67d640 1 -- 172.21.15.98:0/368050347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aec4550 msgr2=0x55f41b2a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6e6f8a740 10 client.0 _ll_register_callbacks cb 0x7fa6d400e830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aec4550 0x55f41b2a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6de67d640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aec4550 0x55f41b2a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6de67d640 1 -- 172.21.15.98:0/368050347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aeac1f0 msgr2=0x55f41b2a43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6de67d640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aeac1f0 0x55f41b2a43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6de67d640 1 -- 172.21.15.98:0/368050347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f41b310870 con 0x55f41aecc730 2024-04-25T16:34:52.053 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aeac1f0 0x55f41b2a43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6dd67b640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aec4550 0x55f41b2a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/368050347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6d002d070 con 0x55f41aecc730 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6de67d640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a1590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00049d0 tx=0x7fa6d0009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57c0d11f88c0f237 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.046+0000 7fa6c17fa640 10 client.25808 ms_handle_connect on v2:172.21.15.173:3300/0 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6c17fa640 1 -- 172.21.15.98:0/368050347 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d001ce30 con 0x55f41aecc730 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6c17fa640 1 -- 172.21.15.98:0/368050347 <== mon.2 v2:172.21.15.173:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7fa6d0031620 con 0x55f41aecc730 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6c17fa640 1 -- 172.21.15.98:0/368050347 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fa6d0031820 con 0x55f41aecc730 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6e6f8a740 10 client.25808 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/368050347 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({fsmap=31}) -- 0x55f41b311a60 con 0x55f41aecc730 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6c17fa640 1 -- 172.21.15.98:0/368050347 <== mon.2 v2:172.21.15.173:3300/0 4 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7fa6d00319c0 con 0x55f41aecc730 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6e6f8a740 10 client.25808 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6e6f8a740 10 client.25808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6e6f8a740 20 client.25808 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:52.054 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.047+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55f41aeed410 0x55f41b3143c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.048+0000 7fa6e6f8a740 4 client.25808 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.048+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/368050347 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f41b2aaa90 con 0x55f41aeed410 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.048+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55f41aeed410 0x55f41b3143c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.048+0000 7fa6dde7c640 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55f41aeed410 0x55f41b3143c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d6b313c5b309e01 server_cookie=c822b07b9a5993fa in_seq=0 out_seq=0 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.048+0000 7fa6c17fa640 10 client.25808 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.050+0000 7fa6c17fa640 1 -- 172.21.15.98:0/368050347 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f41b2aaa90 con 0x55f41aeed410 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.050+0000 7fa6c17fa640 10 client.25808 handle_command_reply: tid=0 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.050+0000 7fa6e6f8a740 1 client.25808 shutdown 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.050+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 msgr2=0x55f41b2a1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.050+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a1590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6d00049d0 tx=0x7fa6d0009500 comp rx=0 tx=0).stop 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/368050347 shutdown_connections 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x55f41aeed410 0x55f41b3143c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55f41aecc730 0x55f41b2a1590 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.055 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55f41aec4550 0x55f41b2a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 1 --2- 172.21.15.98:0/368050347 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55f41aeac1f0 0x55f41b2a43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/368050347 >> 172.21.15.98:0/368050347 conn(0x55f41aee1560 msgr2=0x55f41b29ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6c17fa640 0 client.25808 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/368050347 shutdown_connections 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 1 -- 172.21.15.98:0/368050347 wait complete. 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.051+0000 7fa6e6f8a740 20 client.25808 trim_cache size 0 max 16384 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stdout: "return_code": 0, 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_tag": "e3b4b887-ca51-4525-a19b-e4abf84e9995", 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": "asynchronous" 2024-04-25T16:34:52.056 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:34:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 -- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6c40afaf0 msgr2=0x7fa6c40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6c40afaf0 0x7fa6c40b2000 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55f41b293a80 tx=0x7fa6d0004a40 comp rx=0 tx=0).stop 2024-04-25T16:34:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 -- 172.21.15.98:0/2068059009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d8104d40 msgr2=0x7fa6d819a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d8104d40 0x7fa6d819a680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8005760 tx=0x7fa6c8005e20 comp rx=0 tx=0).stop 2024-04-25T16:34:52.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 -- 172.21.15.98:0/2068059009 shutdown_connections 2024-04-25T16:34:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fa6c40afaf0 0x7fa6c40b2000 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fa6d8109a40 0x7fa6d81a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6d8104d40 0x7fa6d819a680 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 --2- 172.21.15.98:0/2068059009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa6d8103b00 0x7fa6d819a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 -- 172.21.15.98:0/2068059009 >> 172.21.15.98:0/2068059009 conn(0x7fa6d80fecf0 msgr2=0x7fa6d8105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 -- 172.21.15.98:0/2068059009 shutdown_connections 2024-04-25T16:34:52.058 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.052+0000 7fa6e00e0640 1 -- 172.21.15.98:0/2068059009 wait complete. 2024-04-25T16:34:52.066 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:34:52.067 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:34:52.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:51 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:34:52.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:51 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:34:52.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:51 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:34:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 -- 172.21.15.98:0/1991240981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538103c40 msgr2=0x7f5538104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 --2- 172.21.15.98:0/1991240981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538103c40 0x7f5538104050 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f552c009900 tx=0x7f552c02a410 comp rx=0 tx=0).stop 2024-04-25T16:34:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 -- 172.21.15.98:0/1991240981 shutdown_connections 2024-04-25T16:34:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 --2- 172.21.15.98:0/1991240981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5538108940 0x7f553810ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 --2- 172.21.15.98:0/1991240981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538103c40 0x7f5538104050 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 --2- 172.21.15.98:0/1991240981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5538102a00 0x7f5538103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 -- 172.21.15.98:0/1991240981 >> 172.21.15.98:0/1991240981 conn(0x7f55380fdbf0 msgr2=0x7f55381000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.204 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 -- 172.21.15.98:0/1991240981 shutdown_connections 2024-04-25T16:34:52.205 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.205+0000 7f553e4d8640 1 -- 172.21.15.98:0/1991240981 wait complete. 2024-04-25T16:34:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553e4d8640 1 Processor -- start 2024-04-25T16:34:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553e4d8640 1 -- start start 2024-04-25T16:34:52.207 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553e4d8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5538102a00 0x7f5538197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553e4d8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5538103c40 0x7f55381983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553e4d8640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538108940 0x7f553819f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553e4d8640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f553810f530 con 0x7f5538108940 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553e4d8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f553810f3b0 con 0x7f5538102a00 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553e4d8640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f553810f6b0 con 0x7f5538103c40 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553ca75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538108940 0x7f553819f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f55377fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5538103c40 0x7f55381983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f5537fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5538102a00 0x7f5538197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553ca75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538108940 0x7f553819f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50884/0 (socket says 172.21.15.98:50884) 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f55377fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5538103c40 0x7f55381983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40494/0 (socket says 172.21.15.98:40494) 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.206+0000 7f553ca75640 1 -- 172.21.15.98:0/3757329893 learned_addr learned my addr 172.21.15.98:0/3757329893 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55377fe640 1 -- 172.21.15.98:0/3757329893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5538102a00 msgr2=0x7f5538197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55377fe640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5538102a00 0x7f5538197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.208 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55377fe640 1 -- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538108940 msgr2=0x7f553819f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55377fe640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538108940 0x7f553819f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55377fe640 1 -- 172.21.15.98:0/3757329893 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553819f8f0 con 0x7f5538103c40 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f5537fff640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5538102a00 0x7f5538197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f553ca75640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538108940 0x7f553819f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55377fe640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5538103c40 0x7f55381983b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f552c02a930 tx=0x7f552c003ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55357fa640 1 -- 172.21.15.98:0/3757329893 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c03f070 con 0x7f5538103c40 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f553e4d8640 1 -- 172.21.15.98:0/3757329893 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f553819fb80 con 0x7f5538103c40 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55357fa640 1 -- 172.21.15.98:0/3757329893 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f552c03bd00 con 0x7f5538103c40 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f55357fa640 1 -- 172.21.15.98:0/3757329893 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c035030 con 0x7f5538103c40 2024-04-25T16:34:52.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.207+0000 7f553e4d8640 1 -- 172.21.15.98:0/3757329893 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55381a0090 con 0x7f5538103c40 2024-04-25T16:34:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.210+0000 7f55357fa640 1 -- 172.21.15.98:0/3757329893 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f552c042070 con 0x7f5538103c40 2024-04-25T16:34:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.210+0000 7f5545389740 1 Processor -- start 2024-04-25T16:34:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5545389740 1 -- start start 2024-04-25T16:34:52.211 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5545389740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a3c390 0x560ea9ddda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5545389740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a134d0 0x560ea9dddfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5545389740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a1b6b0 0x560ea9ddec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5545389740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560ea9d42d80 con 0x560ea9a3c390 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5545389740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560ea9cfc8b0 con 0x560ea9a134d0 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5545389740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560ea9d3ed40 con 0x560ea9a1b6b0 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f553ca75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a3c390 0x560ea9ddda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5537fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a134d0 0x560ea9dddfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f553ca75640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a3c390 0x560ea9ddda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50890/0 (socket says 172.21.15.98:50890) 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f553ca75640 1 -- 172.21.15.98:0/75281252 learned_addr learned my addr 172.21.15.98:0/75281252 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f55377fe640 1 --2- 172.21.15.98:0/75281252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a1b6b0 0x560ea9ddec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5537fff640 1 -- 172.21.15.98:0/75281252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a1b6b0 msgr2=0x560ea9ddec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5537fff640 1 --2- 172.21.15.98:0/75281252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a1b6b0 0x560ea9ddec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.212 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f55357fa640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55200afaf0 0x7f55200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5537fff640 1 -- 172.21.15.98:0/75281252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a3c390 msgr2=0x560ea9ddda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5537fff640 1 --2- 172.21.15.98:0/75281252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a3c390 0x560ea9ddda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f5537fff640 1 -- 172.21.15.98:0/75281252 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ea9ccda40 con 0x560ea9a134d0 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.211+0000 7f553ca75640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55200afaf0 0x7f55200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.212+0000 7f55357fa640 1 -- 172.21.15.98:0/3757329893 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f552c0ff940 con 0x7f5538103c40 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.212+0000 7f5537fff640 1 --2- 172.21.15.98:0/75281252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a134d0 0x560ea9dddfe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f55381022d0 tx=0x7f5528011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b7f62d7226736a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.212+0000 7f551e7fc640 1 -- 172.21.15.98:0/75281252 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528004310 con 0x560ea9a134d0 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.212+0000 7f553ca75640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55200afaf0 0x7f55200b2000 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x560ea9ddeab0 tx=0x7f5530004a40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.212+0000 7f551e7fc640 1 -- 172.21.15.98:0/75281252 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55280044b0 con 0x560ea9a134d0 2024-04-25T16:34:52.213 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.212+0000 7f551e7fc640 1 -- 172.21.15.98:0/75281252 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528004e60 con 0x560ea9a134d0 2024-04-25T16:34:52.216 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 -- 172.21.15.98:0/75281252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a134d0 msgr2=0x560ea9dddfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 --2- 172.21.15.98:0/75281252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a134d0 0x560ea9dddfe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f55381022d0 tx=0x7f5528011810 comp rx=0 tx=0).stop 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 -- 172.21.15.98:0/75281252 shutdown_connections 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 --2- 172.21.15.98:0/75281252 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a1b6b0 0x560ea9ddec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 --2- 172.21.15.98:0/75281252 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a134d0 0x560ea9dddfe0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 --2- 172.21.15.98:0/75281252 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a3c390 0x560ea9ddda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 -- 172.21.15.98:0/75281252 >> 172.21.15.98:0/75281252 conn(0x560ea9a304e0 msgr2=0x560ea9dd9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 -- 172.21.15.98:0/75281252 shutdown_connections 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.213+0000 7f5545389740 1 -- 172.21.15.98:0/75281252 wait complete. 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 1 -- start start 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 1 Processor -- start 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a134d0 0x560ea9de7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a1b6b0 0x560ea9de7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a3c390 0x560ea9deeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x560ea9ddf780 con 0x560ea9a1b6b0 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x560ea9cc9d10 con 0x560ea9a3c390 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x560ea9ddf900 con 0x560ea9a134d0 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f55377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a1b6b0 0x560ea9de7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5537fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a134d0 0x560ea9de7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5545389740 10 client.0 _ll_register_callbacks cb 0x560ea9cf6210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f55377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a1b6b0 0x560ea9de7c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50894/0 (socket says 172.21.15.98:50894) 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f5537fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a134d0 0x560ea9de7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40518/0 (socket says 172.21.15.98:40518) 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.214+0000 7f55377fe640 1 -- 172.21.15.98:0/1921308274 learned_addr learned my addr 172.21.15.98:0/1921308274 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f553ca75640 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a3c390 0x560ea9deeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55377fe640 1 -- 172.21.15.98:0/1921308274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a134d0 msgr2=0x560ea9de7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55377fe640 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a134d0 0x560ea9de7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55377fe640 1 -- 172.21.15.98:0/1921308274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a3c390 msgr2=0x560ea9deeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55377fe640 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a3c390 0x560ea9deeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55377fe640 1 -- 172.21.15.98:0/1921308274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ea9b1e7d0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55377fe640 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a1b6b0 0x560ea9de7c40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f552c0c7ab0 tx=0x7f552c032f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f504649612c1a2d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55177fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55177fe640 1 -- 172.21.15.98:0/1921308274 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c049070 con 0x560ea9a1b6b0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55177fe640 1 -- 172.21.15.98:0/1921308274 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f552c031400 con 0x560ea9a1b6b0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f5545389740 1 -- 172.21.15.98:0/1921308274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55280044b0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f55177fe640 1 -- 172.21.15.98:0/1921308274 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f552c0c78f0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f5545389740 10 client.16305 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.215+0000 7f5545389740 1 -- 172.21.15.98:0/1921308274 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560ea9a543f0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f55177fe640 1 -- 172.21.15.98:0/1921308274 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f55280044b0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f55177fe640 1 -- 172.21.15.98:0/1921308274 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f552c0c7d80 con 0x560ea9a1b6b0 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f5545389740 10 client.16305 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f5545389740 10 client.16305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f5545389740 20 client.16305 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f5545389740 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560ea99faf60 0x560ea9df5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f5545389740 4 client.16305 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:34:52.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f5545389740 1 -- 172.21.15.98:0/1921308274 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ea9d50990 con 0x560ea99faf60 2024-04-25T16:34:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.216+0000 7f5537fff640 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560ea99faf60 0x560ea9df5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.217+0000 7f55177fe640 10 client.16305 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:52.220 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.217+0000 7f5537fff640 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560ea99faf60 0x560ea9df5140 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acc67f52848ac3c3 server_cookie=e712a72156edf1b2 in_seq=0 out_seq=0 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.219+0000 7f55177fe640 1 -- 172.21.15.98:0/1921308274 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560ea9d50990 con 0x560ea99faf60 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.219+0000 7f55177fe640 10 client.16305 handle_command_reply: tid=0 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.219+0000 7f5545389740 1 client.16305 shutdown 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 -- 172.21.15.98:0/1921308274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a1b6b0 msgr2=0x560ea9de7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a1b6b0 0x560ea9de7c40 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f552c0c7ab0 tx=0x7f552c032f70 comp rx=0 tx=0).stop 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 -- 172.21.15.98:0/1921308274 shutdown_connections 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560ea99faf60 0x560ea9df5140 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a3c390 0x560ea9deeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f55177fe640 0 client.16305 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:52.221 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a1b6b0 0x560ea9de7c40 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 --2- 172.21.15.98:0/1921308274 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea9a134d0 0x560ea9de7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 -- 172.21.15.98:0/1921308274 >> 172.21.15.98:0/1921308274 conn(0x560ea9a304e0 msgr2=0x560ea9ddc180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 -- 172.21.15.98:0/1921308274 shutdown_connections 2024-04-25T16:34:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 1 -- 172.21.15.98:0/1921308274 wait complete. 2024-04-25T16:34:52.222 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.220+0000 7f5545389740 20 client.16305 trim_cache size 0 max 16384 2024-04-25T16:34:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5545389740 1 Processor -- start 2024-04-25T16:34:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5545389740 1 -- start start 2024-04-25T16:34:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5545389740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9de5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5545389740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9deeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5545389740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9dd8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5545389740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55381bb610 con 0x560ea9a134d0 2024-04-25T16:34:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5545389740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55381bd210 con 0x560ea9a1b6b0 2024-04-25T16:34:52.244 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5545389740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55381bbce0 con 0x560ea99faf60 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f55377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9deeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5537fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9de5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5537fff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9de5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.98:40530/0 (socket says 172.21.15.98:40530) 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f553ca75640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9dd8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f5537fff640 1 -- 172.21.15.98:0/3071221575 learned_addr learned my addr 172.21.15.98:0/3071221575 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.244+0000 7f55377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9deeac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50908/0 (socket says 172.21.15.98:50908) 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f553ca75640 1 -- 172.21.15.98:0/3071221575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 msgr2=0x560ea9de5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f553ca75640 1 --2- 172.21.15.98:0/3071221575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9de5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f553ca75640 1 -- 172.21.15.98:0/3071221575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 msgr2=0x560ea9deeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f553ca75640 1 --2- 172.21.15.98:0/3071221575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9deeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f553ca75640 1 -- 172.21.15.98:0/3071221575 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ea9dde0c0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f55377fe640 1 --2- 172.21.15.98:0/3071221575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9deeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f553ca75640 1 --2- 172.21.15.98:0/3071221575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9dd8c10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f553000f7b0 tx=0x7f553001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5095c0e76d0192ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.245 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f551dffb640 1 -- 172.21.15.98:0/3071221575 <== mon.2 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55300283f0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f551dffb640 1 -- 172.21.15.98:0/3071221575 <== mon.2 v2:172.21.15.173:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f553002d070 con 0x560ea9a1b6b0 2024-04-25T16:34:52.246 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.245+0000 7f551dffb640 1 -- 172.21.15.98:0/3071221575 <== mon.2 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f553000e6a0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.246+0000 7f5545389740 1 -- 172.21.15.98:0/3071221575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 msgr2=0x560ea9dd8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.246+0000 7f5545389740 1 --2- 172.21.15.98:0/3071221575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9dd8c10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f553000f7b0 tx=0x7f553001a000 comp rx=0 tx=0).stop 2024-04-25T16:34:52.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.246+0000 7f5545389740 1 -- 172.21.15.98:0/3071221575 shutdown_connections 2024-04-25T16:34:52.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.246+0000 7f5545389740 1 --2- 172.21.15.98:0/3071221575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9dd8c10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.249 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.246+0000 7f5545389740 1 --2- 172.21.15.98:0/3071221575 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9deeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.246+0000 7f5545389740 1 --2- 172.21.15.98:0/3071221575 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9de5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.246+0000 7f5545389740 1 -- 172.21.15.98:0/3071221575 >> 172.21.15.98:0/3071221575 conn(0x560ea9a304e0 msgr2=0x560ea9de7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.247+0000 7f5545389740 1 -- 172.21.15.98:0/3071221575 shutdown_connections 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.247+0000 7f5545389740 1 -- 172.21.15.98:0/3071221575 wait complete. 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.247+0000 7f5545389740 1 -- start start 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.247+0000 7f5545389740 1 Processor -- start 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.247+0000 7f5545389740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9debed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f5545389740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9dec410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f5545389740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9ded2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f5545389740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f55381bca80 con 0x560ea9a134d0 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f5545389740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f55381bc3b0 con 0x560ea9a1b6b0 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f5545389740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f55381b95f0 con 0x560ea99faf60 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f5545389740 10 client.0 _ll_register_callbacks cb 0x7f552c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:34:52.250 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9dec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9dec410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50924/0 (socket says 172.21.15.98:50924) 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 -- 172.21.15.98:0/1332572128 learned_addr learned my addr 172.21.15.98:0/1332572128 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f5537fff640 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9debed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f553ca75640 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9ded2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 -- 172.21.15.98:0/1332572128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 msgr2=0x560ea9debed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9debed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 -- 172.21.15.98:0/1332572128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 msgr2=0x560ea9ded2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9ded2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 -- 172.21.15.98:0/1332572128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ea9a140f0 con 0x560ea9a134d0 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55177fe640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.248+0000 7f55377fe640 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9dec410 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f552c0c7ab0 tx=0x7f552c0ffcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91a9d596836621f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f55177fe640 1 -- 172.21.15.98:0/1332572128 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f552c049040 con 0x560ea9a134d0 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f55177fe640 1 -- 172.21.15.98:0/1332572128 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9179+0+0 (secure 0 0 0) 0x7f552c0d0650 con 0x560ea9a134d0 2024-04-25T16:34:52.251 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 1 -- 172.21.15.98:0/1332572128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f553002d070 con 0x560ea9a134d0 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f55177fe640 1 -- 172.21.15.98:0/1332572128 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f552c0c7290 con 0x560ea9a134d0 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 10 client.16311 fetch_fsmap learned FSMap version 31 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 1 -- 172.21.15.98:0/1332572128 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x560ea9d034b0 con 0x560ea9a134d0 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f55177fe640 1 -- 172.21.15.98:0/1332572128 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f553002d070 con 0x560ea9a134d0 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f55177fe640 1 -- 172.21.15.98:0/1332572128 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 31) ==== 8378+0+0 (secure 0 0 0) 0x7f552c0ea0d0 con 0x560ea9a134d0 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 10 client.16311 fetch_fsmap finished waiting for FSMap version 31 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 20 client.16311 populate_metadata read hostname 'smithi098' 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560ea9a3c390 0x560ea9e5f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 4 client.16311 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.249+0000 7f5545389740 1 -- 172.21.15.98:0/1332572128 --> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560ea9d4a800 con 0x560ea9a3c390 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.250+0000 7f5537fff640 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560ea9a3c390 0x560ea9e5f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.250+0000 7f55177fe640 10 client.16311 ms_handle_connect on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:52.252 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.250+0000 7f5537fff640 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560ea9a3c390 0x560ea9e5f500 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ccd535144fdddca server_cookie=23c9fb42b3f89fa in_seq=0 out_seq=0 2024-04-25T16:34:52.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.252+0000 7f55177fe640 1 -- 172.21.15.98:0/1332572128 <== mds.0 v2:172.21.15.173:6838/2050448663 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x560ea9d4a800 con 0x560ea9a3c390 2024-04-25T16:34:52.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.252+0000 7f55177fe640 10 client.16311 handle_command_reply: tid=0 2024-04-25T16:34:52.253 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.252+0000 7f5545389740 1 client.16311 shutdown 2024-04-25T16:34:52.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:51 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:34:52.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:51 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:34:52.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:51 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 -- 172.21.15.98:0/1332572128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 msgr2=0x560ea9dec410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9dec410 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f552c0c7ab0 tx=0x7f552c0ffcc0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 -- 172.21.15.98:0/1332572128 shutdown_connections 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.173:6838/2050448663,v1:172.21.15.173:6839/2050448663] conn(0x560ea9a3c390 0x560ea9e5f500 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x560ea9a1b6b0 0x560ea9ded2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x560ea9a134d0 0x560ea9dec410 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 --2- 172.21.15.98:0/1332572128 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x560ea99faf60 0x560ea9debed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 -- 172.21.15.98:0/1332572128 >> 172.21.15.98:0/1332572128 conn(0x560ea9a304e0 msgr2=0x560ea9de7700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f55177fe640 0 client.16311 ms_handle_reset on v2:172.21.15.173:6838/2050448663 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 -- 172.21.15.98:0/1332572128 shutdown_connections 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 1 -- 172.21.15.98:0/1332572128 wait complete. 2024-04-25T16:34:52.255 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.253+0000 7f5545389740 20 client.16311 trim_cache size 0 max 16384 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout: "status": "scrub active (320 inodes in the stack)", 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout: "scrubs": { 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout: "e3b4b887-ca51-4525-a19b-e4abf84e9995": { 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout: "path": "/", 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tag": "e3b4b887-ca51-4525-a19b-e4abf84e9995", 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout: "options": "recursive,force" 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:34:52.256 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:34:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 -- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55200afaf0 msgr2=0x7f55200b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55200afaf0 0x7f55200b2000 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x560ea9ddeab0 tx=0x7f5530004a40 comp rx=0 tx=0).stop 2024-04-25T16:34:52.257 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 -- 172.21.15.98:0/3757329893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5538103c40 msgr2=0x7f55381983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5538103c40 0x7f55381983b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f552c02a930 tx=0x7f552c003ed0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 -- 172.21.15.98:0/3757329893 shutdown_connections 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f55200afaf0 0x7f55200b2000 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5538108940 0x7f553819f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5538103c40 0x7f55381983b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 --2- 172.21.15.98:0/3757329893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5538102a00 0x7f5538197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 -- 172.21.15.98:0/3757329893 >> 172.21.15.98:0/3757329893 conn(0x7f55380fdbf0 msgr2=0x7f55380fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.254+0000 7f553e4d8640 1 -- 172.21.15.98:0/3757329893 shutdown_connections 2024-04-25T16:34:52.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:34:52.255+0000 7f553e4d8640 1 -- 172.21.15.98:0/3757329893 wait complete. 2024-04-25T16:34:52.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:51 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:34:52.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:51 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:34:52.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:51 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:34:52.267 INFO:tasks.cephfs.filesystem:scrub status for tag:e3b4b887-ca51-4525-a19b-e4abf84e9995 - {'path': '/', 'tag': 'e3b4b887-ca51-4525-a19b-e4abf84e9995', 'options': 'recursive,force'} 2024-04-25T16:34:53.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: pgmap v427: 129 pgs: 129 active+clean; 206 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 250 op/s 2024-04-25T16:34:53.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub queued for path: / 2024-04-25T16:34:53.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:53.189 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:52 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: pgmap v427: 129 pgs: 129 active+clean; 206 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 250 op/s 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub queued for path: / 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:53.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:52 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: pgmap v427: 129 pgs: 129 active+clean; 206 MiB data, 963 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 250 op/s 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub queued for path: / 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:53.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:52 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.190 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.191 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.192 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.193 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:54.194 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:53 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:54.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:53 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.266 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.267 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:54.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:54.268 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:53 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.206 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:55.207 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.207 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:55.207 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.207 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:55.207 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.207 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:55.207 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: pgmap v428: 129 pgs: 129 active+clean; 219 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 334 op/s 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:55.208 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:54 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.209 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8d13541700 time 2024-04-25T16:34:55.142581+0000 2024-04-25T16:34:55.209 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d1dd75ba6] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d1dd75dd1] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56324ffac26b] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56324ff4b428] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56324ff4c8a9] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56324ff4e9c3] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56324fe1a85d] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56324fe1ce8e] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56324fe1d894] 2024-04-25T16:34:55.210 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56324fe0805b] 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d1def2ac6] 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 12: (DispatchQueue::entry()+0xb27) [0x7f8d1def00b9] 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d1dfdaccb] 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 14: (Thread::entry_wrapper()+0x43) [0x7f8d1dd4472d] 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8d1dd44749] 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8d1c8e81ca] 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 17: clone() 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.144+0000 7f8d13541700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8d13541700 time 2024-04-25T16:34:55.142581+0000 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d1dd75ba6] 2024-04-25T16:34:55.211 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d1dd75dd1] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56324ffac26b] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56324ff4b428] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56324ff4c8a9] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56324ff4e9c3] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56324fe1a85d] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56324fe1ce8e] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56324fe1d894] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56324fe0805b] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d1def2ac6] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 12: (DispatchQueue::entry()+0xb27) [0x7f8d1def00b9] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d1dfdaccb] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 14: (Thread::entry_wrapper()+0x43) [0x7f8d1dd4472d] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8d1dd44749] 2024-04-25T16:34:55.212 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8d1c8e81ca] 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 17: clone() 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: *** Caught signal (Aborted) ** 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: in thread 7f8d13541700 thread_name:ms_dispatch 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d1c8f2d20] 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2: gsignal() 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 3: abort() 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d1dd75ced] 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d1dd75dd1] 2024-04-25T16:34:55.213 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56324ffac26b] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56324ff4b428] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56324ff4c8a9] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56324ff4e9c3] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56324fe1a85d] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56324fe1ce8e] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56324fe1d894] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56324fe0805b] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d1def2ac6] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 15: (DispatchQueue::entry()+0xb27) [0x7f8d1def00b9] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d1dfdaccb] 2024-04-25T16:34:55.214 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 17: (Thread::entry_wrapper()+0x43) [0x7f8d1dd4472d] 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8d1dd44749] 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8d1c8e81ca] 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 20: clone() 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.146+0000 7f8d13541700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: in thread 7f8d13541700 thread_name:ms_dispatch 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d1c8f2d20] 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2: gsignal() 2024-04-25T16:34:55.215 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 3: abort() 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d1dd75ced] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d1dd75dd1] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56324ffac26b] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56324ff4b428] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56324ff4c8a9] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56324ff4e9c3] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56324fe1a85d] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56324fe1ce8e] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56324fe1d894] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56324fe0805b] 2024-04-25T16:34:55.216 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d1def2ac6] 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 15: (DispatchQueue::entry()+0xb27) [0x7f8d1def00b9] 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d1dfdaccb] 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 17: (Thread::entry_wrapper()+0x43) [0x7f8d1dd4472d] 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8d1dd44749] 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8d1c8e81ca] 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 20: clone() 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: -1> 2024-04-25T16:34:55.144+0000 7f8d13541700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8d13541700 time 2024-04-25T16:34:55.142581+0000 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.217 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d1dd75ba6] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d1dd75dd1] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56324ffac26b] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56324ff4b428] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56324ff4c8a9] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56324ff4e9c3] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56324fe1a85d] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56324fe1ce8e] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56324fe1d894] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56324fe0805b] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d1def2ac6] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 12: (DispatchQueue::entry()+0xb27) [0x7f8d1def00b9] 2024-04-25T16:34:55.218 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d1dfdaccb] 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 14: (Thread::entry_wrapper()+0x43) [0x7f8d1dd4472d] 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8d1dd44749] 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8d1c8e81ca] 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 17: clone() 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 0> 2024-04-25T16:34:55.146+0000 7f8d13541700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: in thread 7f8d13541700 thread_name:ms_dispatch 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d1c8f2d20] 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2: gsignal() 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 3: abort() 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d1dd75ced] 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d1dd75dd1] 2024-04-25T16:34:55.219 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56324ffac26b] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56324ff4b428] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56324ff4c8a9] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56324ff4e9c3] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56324fe1a85d] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56324fe1ce8e] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56324fe1d894] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56324fe0805b] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d1def2ac6] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 15: (DispatchQueue::entry()+0xb27) [0x7f8d1def00b9] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d1dfdaccb] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 17: (Thread::entry_wrapper()+0x43) [0x7f8d1dd4472d] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8d1dd44749] 2024-04-25T16:34:55.220 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8d1c8e81ca] 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 20: clone() 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: -1> 2024-04-25T16:34:55.144+0000 7f8d13541700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f8d13541700 time 2024-04-25T16:34:55.142581+0000 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f8d1dd75ba6] 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d1dd75dd1] 2024-04-25T16:34:55.221 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56324ffac26b] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56324ff4b428] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56324ff4c8a9] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56324ff4e9c3] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56324fe1a85d] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56324fe1ce8e] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56324fe1d894] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56324fe0805b] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d1def2ac6] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 12: (DispatchQueue::entry()+0xb27) [0x7f8d1def00b9] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d1dfdaccb] 2024-04-25T16:34:55.222 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 14: (Thread::entry_wrapper()+0x43) [0x7f8d1dd4472d] 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 15: (Thread::_entry_func(void*)+0xd) [0x7f8d1dd44749] 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8d1c8e81ca] 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 17: clone() 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 0> 2024-04-25T16:34:55.146+0000 7f8d13541700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: in thread 7f8d13541700 thread_name:ms_dispatch 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8d1c8f2d20] 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2: gsignal() 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 3: abort() 2024-04-25T16:34:55.223 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f8d1dd75ced] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8d1dd75dd1] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56324ffac26b] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56324ff4b428] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56324ff4c8a9] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56324ff4e9c3] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56324fe1a85d] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56324fe1ce8e] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56324fe1d894] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56324fe0805b] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f8d1def2ac6] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 15: (DispatchQueue::entry()+0xb27) [0x7f8d1def00b9] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8d1dfdaccb] 2024-04-25T16:34:55.224 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 17: (Thread::entry_wrapper()+0x43) [0x7f8d1dd4472d] 2024-04-25T16:34:55.225 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 18: (Thread::_entry_func(void*)+0xd) [0x7f8d1dd44749] 2024-04-25T16:34:55.225 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8d1c8e81ca] 2024-04-25T16:34:55.225 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 20: clone() 2024-04-25T16:34:55.225 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:34:55.225 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[62804]: 2024-04-25T16:34:55.225 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:55 smithi173 podman[64999]: 2024-04-25 16:34:55.208611371 +0000 UTC m=+0.020587862 container died 2d7b6bdfc8eb840845a042b20a4ab46713f429c27ea7ab15b9257b72e3a0423c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: pgmap v428: 129 pgs: 129 active+clean; 219 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 334 op/s 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:55.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:54 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:55.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:55.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: pgmap v428: 129 pgs: 129 active+clean; 219 MiB data, 991 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 334 op/s 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:55.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:54 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: active paths [/] 2024-04-25T16:34:56.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:55 smithi098 ceph-mon[29162]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: active paths [/] 2024-04-25T16:34:56.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:55 smithi173 ceph-mon[32556]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.338 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.339 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:56.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: active paths [/] 2024-04-25T16:34:56.340 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:55 smithi203 ceph-mon[33115]: scrub summary: idle+waiting paths [/] 2024-04-25T16:34:57.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:56 smithi098 ceph-mon[29162]: pgmap v429: 129 pgs: 129 active+clean; 221 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 274 op/s 2024-04-25T16:34:57.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:56 smithi173 ceph-mon[32556]: pgmap v429: 129 pgs: 129 active+clean; 221 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 274 op/s 2024-04-25T16:34:57.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:56 smithi203 ceph-mon[33115]: pgmap v429: 129 pgs: 129 active+clean; 221 MiB data, 1000 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.1 MiB/s wr, 274 op/s 2024-04-25T16:34:58.337 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:34:58 smithi173 ceph-mon[32556]: pgmap v430: 129 pgs: 129 active+clean; 225 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 293 op/s 2024-04-25T16:34:58.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:34:58 smithi203 ceph-mon[33115]: pgmap v430: 129 pgs: 129 active+clean; 225 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 293 op/s 2024-04-25T16:34:58.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:34:58 smithi098 ceph-mon[29162]: pgmap v430: 129 pgs: 129 active+clean; 225 MiB data, 1023 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 293 op/s 2024-04-25T16:34:59.263 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:58 smithi173 podman[64999]: 2024-04-25 16:34:58.773226498 +0000 UTC m=+3.585203023 container remove 2d7b6bdfc8eb840845a042b20a4ab46713f429c27ea7ab15b9257b72e3a0423c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T16:34:59.264 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:58 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:34:59.264 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:59 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-25T16:34:59.264 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:34:59 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 52.925s CPU time. 2024-04-25T16:35:00.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:00 smithi203 ceph-mon[33115]: pgmap v431: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.3 MiB/s wr, 230 op/s 2024-04-25T16:35:00.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:00 smithi098 ceph-mon[29162]: pgmap v431: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.3 MiB/s wr, 230 op/s 2024-04-25T16:35:00.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:00 smithi173 ceph-mon[32556]: pgmap v431: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.3 MiB/s wr, 230 op/s 2024-04-25T16:35:02.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:02 smithi203 ceph-mon[33115]: pgmap v432: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-25T16:35:02.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:02 smithi098 ceph-mon[29162]: pgmap v432: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-25T16:35:02.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:02 smithi173 ceph-mon[32556]: pgmap v432: 129 pgs: 129 active+clean; 226 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 1.8 MiB/s wr, 175 op/s 2024-04-25T16:35:04.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:04 smithi203 ceph-mon[33115]: pgmap v433: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-04-25T16:35:04.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:04 smithi098 ceph-mon[29162]: pgmap v433: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-04-25T16:35:04.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:04 smithi173 ceph-mon[32556]: pgmap v433: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-04-25T16:35:06.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:06 smithi203 ceph-mon[33115]: pgmap v434: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 716 KiB/s wr, 77 op/s 2024-04-25T16:35:06.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:06 smithi098 ceph-mon[29162]: pgmap v434: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 716 KiB/s wr, 77 op/s 2024-04-25T16:35:06.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:06 smithi173 ceph-mon[32556]: pgmap v434: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 716 KiB/s wr, 77 op/s 2024-04-25T16:35:08.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:08 smithi203 ceph-mon[33115]: pgmap v435: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 534 KiB/s wr, 59 op/s 2024-04-25T16:35:08.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:08 smithi098 ceph-mon[29162]: pgmap v435: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 534 KiB/s wr, 59 op/s 2024-04-25T16:35:08.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:08 smithi173 ceph-mon[32556]: pgmap v435: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 534 KiB/s wr, 59 op/s 2024-04-25T16:35:09.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:09 smithi173 ceph-mon[32556]: Replacing daemon mds.b as rank 0 with standby daemon mds.f 2024-04-25T16:35:09.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:09 smithi173 ceph-mon[32556]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-25T16:35:09.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:09 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:09.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:09 smithi173 ceph-mon[32556]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T16:35:09.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:09 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:09.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:35:09.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 systemd[1]: Stopped Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:09.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 52.925s CPU time. 2024-04-25T16:35:09.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 systemd[1]: Starting Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:35:09.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:09 smithi098 ceph-mon[29162]: Replacing daemon mds.b as rank 0 with standby daemon mds.f 2024-04-25T16:35:09.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:09 smithi098 ceph-mon[29162]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-25T16:35:09.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:09 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:09.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:09 smithi098 ceph-mon[29162]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T16:35:09.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:09 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:09.792 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 podman[65098]: 2024-04-25 16:35:09.420695358 +0000 UTC m=+0.024060645 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:35:09.793 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 podman[65098]: 2024-04-25 16:35:09.549022139 +0000 UTC m=+0.152387414 container create 123225c72bac0aaabf6152c344a11847189663fe099c6026e91e05d758d703e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T16:35:09.793 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 podman[65098]: 2024-04-25 16:35:09.759799008 +0000 UTC m=+0.363164282 container init 123225c72bac0aaabf6152c344a11847189663fe099c6026e91e05d758d703e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T16:35:09.793 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 podman[65098]: 2024-04-25 16:35:09.762316007 +0000 UTC m=+0.365681283 container start 123225c72bac0aaabf6152c344a11847189663fe099c6026e91e05d758d703e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T16:35:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:09 smithi203 ceph-mon[33115]: Replacing daemon mds.b as rank 0 with standby daemon mds.f 2024-04-25T16:35:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:09 smithi203 ceph-mon[33115]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-25T16:35:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:09 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:09 smithi203 ceph-mon[33115]: osdmap e86: 12 total, 12 up, 12 in 2024-04-25T16:35:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:09 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:10.263 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 ceph-mds[65117]: -- 172.21.15.173:0/624204029 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d6dfa21a40 con 0x55d6dfa69a80 2024-04-25T16:35:10.264 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: starting mds.b at 2024-04-25T16:35:10.264 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 bash[65098]: 123225c72bac0aaabf6152c344a11847189663fe099c6026e91e05d758d703e7 2024-04-25T16:35:10.264 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:09 smithi173 systemd[1]: Started Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:10.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:10 smithi098 ceph-mon[29162]: pgmap v437: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 109 KiB/s wr, 19 op/s 2024-04-25T16:35:10.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:10 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:resolve 2024-04-25T16:35:10.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:10 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:boot 2024-04-25T16:35:10.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:10 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:10.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:35:10.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:10 smithi173 ceph-mon[32556]: pgmap v437: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 109 KiB/s wr, 19 op/s 2024-04-25T16:35:10.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:10 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:resolve 2024-04-25T16:35:10.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:10 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:boot 2024-04-25T16:35:10.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:10 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:10.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:35:10.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:10 smithi203 ceph-mon[33115]: pgmap v437: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 109 KiB/s wr, 19 op/s 2024-04-25T16:35:10.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:10 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:resolve 2024-04-25T16:35:10.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:10 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:boot 2024-04-25T16:35:10.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:10 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:10.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:35:12.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:12 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:reconnect 2024-04-25T16:35:12.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:12 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:12.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:12 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0239994 2024-04-25T16:35:12.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:12 smithi098 ceph-mon[29162]: pgmap v438: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 109 KiB/s wr, 19 op/s 2024-04-25T16:35:12.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:12 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:reconnect 2024-04-25T16:35:12.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:12 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:12.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:12 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0239994 2024-04-25T16:35:12.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:12 smithi173 ceph-mon[32556]: pgmap v438: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 109 KiB/s wr, 19 op/s 2024-04-25T16:35:12.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:12 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:reconnect 2024-04-25T16:35:12.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:12 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:12.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:12 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0239994 2024-04-25T16:35:12.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:12 smithi203 ceph-mon[33115]: pgmap v438: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 109 KiB/s wr, 19 op/s 2024-04-25T16:35:13.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:13 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:rejoin 2024-04-25T16:35:13.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:13 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:13.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:13 smithi098 ceph-mon[29162]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:35:13.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:13 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:rejoin 2024-04-25T16:35:13.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:13 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:13.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:13 smithi173 ceph-mon[32556]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:35:13.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:13 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:rejoin 2024-04-25T16:35:13.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:13 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:13.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:13 smithi203 ceph-mon[33115]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:35:14.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:14.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:35:14.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:active 2024-04-25T16:35:14.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:14.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:14.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-mon[33115]: pgmap v439: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-04-25T16:35:14.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f13b73b8700 time 2024-04-25T16:35:14.416548+0000 2024-04-25T16:35:14.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:14.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:14.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f13c1becba6] 2024-04-25T16:35:14.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f13c1becdd1] 2024-04-25T16:35:14.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55f946d6c26b] 2024-04-25T16:35:14.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f946d0b428] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f946d0c8a9] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f946d0e9c3] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f946bda85d] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f946bdce8e] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f946bdd894] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f946bc805b] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13c1d69ac6] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 12: (DispatchQueue::entry()+0xb27) [0x7f13c1d670b9] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13c1e51ccb] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 14: (Thread::entry_wrapper()+0x43) [0x7f13c1bbb72d] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 15: (Thread::_entry_func(void*)+0xd) [0x7f13c1bbb749] 2024-04-25T16:35:14.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f13c075f1ca] 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 17: clone() 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: *** Caught signal (Aborted) ** 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: in thread 7f13b73b8700 thread_name:ms_dispatch 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.418+0000 7f13b73b8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f13b73b8700 time 2024-04-25T16:35:14.416548+0000 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f13c1becba6] 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f13c1becdd1] 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55f946d6c26b] 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f946d0b428] 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f946d0c8a9] 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f946d0e9c3] 2024-04-25T16:35:14.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f946bda85d] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f946bdce8e] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f946bdd894] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f946bc805b] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13c1d69ac6] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 12: (DispatchQueue::entry()+0xb27) [0x7f13c1d670b9] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13c1e51ccb] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 14: (Thread::entry_wrapper()+0x43) [0x7f13c1bbb72d] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 15: (Thread::_entry_func(void*)+0xd) [0x7f13c1bbb749] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f13c075f1ca] 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 17: clone() 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f13c0769d20] 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2: gsignal() 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 3: abort() 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f13c1becced] 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f13c1becdd1] 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55f946d6c26b] 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f946d0b428] 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f946d0c8a9] 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f946d0e9c3] 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f946bda85d] 2024-04-25T16:35:14.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f946bdce8e] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f946bdd894] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f946bc805b] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13c1d69ac6] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 15: (DispatchQueue::entry()+0xb27) [0x7f13c1d670b9] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13c1e51ccb] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 17: (Thread::entry_wrapper()+0x43) [0x7f13c1bbb72d] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 18: (Thread::_entry_func(void*)+0xd) [0x7f13c1bbb749] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f13c075f1ca] 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 20: clone() 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.420+0000 7f13b73b8700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: in thread 7f13b73b8700 thread_name:ms_dispatch 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:14.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f13c0769d20] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2: gsignal() 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 3: abort() 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f13c1becced] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f13c1becdd1] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55f946d6c26b] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f946d0b428] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f946d0c8a9] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f946d0e9c3] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f946bda85d] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f946bdce8e] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f946bdd894] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f946bc805b] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13c1d69ac6] 2024-04-25T16:35:14.695 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 15: (DispatchQueue::entry()+0xb27) [0x7f13c1d670b9] 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13c1e51ccb] 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 17: (Thread::entry_wrapper()+0x43) [0x7f13c1bbb72d] 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 18: (Thread::_entry_func(void*)+0xd) [0x7f13c1bbb749] 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f13c075f1ca] 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 20: clone() 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: -1> 2024-04-25T16:35:14.418+0000 7f13b73b8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f13b73b8700 time 2024-04-25T16:35:14.416548+0000 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f13c1becba6] 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f13c1becdd1] 2024-04-25T16:35:14.696 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55f946d6c26b] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f946d0b428] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f946d0c8a9] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f946d0e9c3] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f946bda85d] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f946bdce8e] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f946bdd894] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f946bc805b] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13c1d69ac6] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 12: (DispatchQueue::entry()+0xb27) [0x7f13c1d670b9] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13c1e51ccb] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 14: (Thread::entry_wrapper()+0x43) [0x7f13c1bbb72d] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 15: (Thread::_entry_func(void*)+0xd) [0x7f13c1bbb749] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f13c075f1ca] 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 17: clone() 2024-04-25T16:35:14.697 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 0> 2024-04-25T16:35:14.420+0000 7f13b73b8700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: in thread 7f13b73b8700 thread_name:ms_dispatch 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f13c0769d20] 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2: gsignal() 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 3: abort() 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f13c1becced] 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f13c1becdd1] 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55f946d6c26b] 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f946d0b428] 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f946d0c8a9] 2024-04-25T16:35:14.698 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f946d0e9c3] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f946bda85d] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f946bdce8e] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f946bdd894] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f946bc805b] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13c1d69ac6] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 15: (DispatchQueue::entry()+0xb27) [0x7f13c1d670b9] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13c1e51ccb] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 17: (Thread::entry_wrapper()+0x43) [0x7f13c1bbb72d] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 18: (Thread::_entry_func(void*)+0xd) [0x7f13c1bbb749] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f13c075f1ca] 2024-04-25T16:35:14.699 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 20: clone() 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: -1> 2024-04-25T16:35:14.418+0000 7f13b73b8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f13b73b8700 time 2024-04-25T16:35:14.416548+0000 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f13c1becba6] 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f13c1becdd1] 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55f946d6c26b] 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f946d0b428] 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f946d0c8a9] 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f946d0e9c3] 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f946bda85d] 2024-04-25T16:35:14.700 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f946bdce8e] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f946bdd894] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f946bc805b] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13c1d69ac6] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 12: (DispatchQueue::entry()+0xb27) [0x7f13c1d670b9] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13c1e51ccb] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 14: (Thread::entry_wrapper()+0x43) [0x7f13c1bbb72d] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 15: (Thread::_entry_func(void*)+0xd) [0x7f13c1bbb749] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f13c075f1ca] 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 17: clone() 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 0> 2024-04-25T16:35:14.420+0000 7f13b73b8700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: in thread 7f13b73b8700 thread_name:ms_dispatch 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.701 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f13c0769d20] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2: gsignal() 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 3: abort() 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f13c1becced] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f13c1becdd1] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55f946d6c26b] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55f946d0b428] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55f946d0c8a9] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55f946d0e9c3] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55f946bda85d] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55f946bdce8e] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55f946bdd894] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55f946bc805b] 2024-04-25T16:35:14.702 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f13c1d69ac6] 2024-04-25T16:35:14.703 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 15: (DispatchQueue::entry()+0xb27) [0x7f13c1d670b9] 2024-04-25T16:35:14.703 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f13c1e51ccb] 2024-04-25T16:35:14.703 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 17: (Thread::entry_wrapper()+0x43) [0x7f13c1bbb72d] 2024-04-25T16:35:14.703 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 18: (Thread::_entry_func(void*)+0xd) [0x7f13c1bbb749] 2024-04-25T16:35:14.703 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f13c075f1ca] 2024-04-25T16:35:14.703 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 20: clone() 2024-04-25T16:35:14.704 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:14.704 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[63866]: 2024-04-25T16:35:14.704 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:14 smithi203 podman[66844]: 2024-04-25 16:35:14.487335595 +0000 UTC m=+0.020405744 container died 28934d9ccdf742c818c522ec2bf178a15900131b25d519031c348b78d83b6f8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-25T16:35:14.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:14 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:14.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:14 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:35:14.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:14 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:active 2024-04-25T16:35:14.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:14 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:14.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:14 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:14.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:14 smithi098 ceph-mon[29162]: pgmap v439: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-04-25T16:35:14.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:14 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:14.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:14 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:35:14.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:14 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] up:active 2024-04-25T16:35:14.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:14 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:14.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:14 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:14.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:14 smithi173 ceph-mon[32556]: pgmap v439: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 KiB/s wr, 17 op/s 2024-04-25T16:35:15.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:15 smithi203 podman[66844]: 2024-04-25 16:35:15.303203895 +0000 UTC m=+0.836274089 container remove 28934d9ccdf742c818c522ec2bf178a15900131b25d519031c348b78d83b6f8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T16:35:15.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:15 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:35:15.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:15 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:35:15.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:15 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 7.345s CPU time. 2024-04-25T16:35:16.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:16 smithi203 ceph-mon[33115]: pgmap v440: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 2.7 KiB/s wr, 32 op/s 2024-04-25T16:35:16.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:16 smithi098 ceph-mon[29162]: pgmap v440: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 2.7 KiB/s wr, 32 op/s 2024-04-25T16:35:16.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:16 smithi173 ceph-mon[32556]: pgmap v440: 129 pgs: 129 active+clean; 227 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 2.7 KiB/s wr, 32 op/s 2024-04-25T16:35:18.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:18 smithi203 ceph-mon[33115]: pgmap v441: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 267 KiB/s wr, 106 op/s 2024-04-25T16:35:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:18 smithi098 ceph-mon[29162]: pgmap v441: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 267 KiB/s wr, 106 op/s 2024-04-25T16:35:18.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:18 smithi173 ceph-mon[32556]: pgmap v441: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 267 KiB/s wr, 106 op/s 2024-04-25T16:35:20.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:20 smithi203 ceph-mon[33115]: pgmap v442: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 251 KiB/s wr, 101 op/s 2024-04-25T16:35:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:20 smithi098 ceph-mon[29162]: pgmap v442: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 251 KiB/s wr, 101 op/s 2024-04-25T16:35:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:20 smithi173 ceph-mon[32556]: pgmap v442: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 251 KiB/s wr, 101 op/s 2024-04-25T16:35:22.268 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:35:22.268 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:35:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 -- 172.21.15.98:0/3956782878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdec81050b0 msgr2=0x7fdec81054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 --2- 172.21.15.98:0/3956782878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdec81050b0 0x7fdec81054c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8009900 tx=0x7fdeb802a410 comp rx=0 tx=0).stop 2024-04-25T16:35:22.393 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 -- 172.21.15.98:0/3956782878 shutdown_connections 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 --2- 172.21.15.98:0/3956782878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdec8102260 0x7fdec810bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 --2- 172.21.15.98:0/3956782878 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdec8101760 0x7fdec8101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 --2- 172.21.15.98:0/3956782878 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdec81050b0 0x7fdec81054c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 -- 172.21.15.98:0/3956782878 >> 172.21.15.98:0/3956782878 conn(0x7fdec80fd380 msgr2=0x7fdec80ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 -- 172.21.15.98:0/3956782878 shutdown_connections 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.392+0000 7fdecf2e0640 1 -- 172.21.15.98:0/3956782878 wait complete. 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecf2e0640 1 Processor -- start 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecf2e0640 1 -- start start 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecf2e0640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdec8101760 0x7fdec81041f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecf2e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdec8102260 0x7fdec8102900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecf2e0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdec81050b0 0x7fdec8102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.394 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecf2e0640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fdec810f4b0 con 0x7fdec8101760 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecf2e0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fdec810f330 con 0x7fdec81050b0 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecf2e0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fdec810f630 con 0x7fdec8102260 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecd07c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdec8101760 0x7fdec81041f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecd07c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdec8101760 0x7fdec81041f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50244/0 (socket says 172.21.15.98:50244) 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.393+0000 7fdecd07c640 1 -- 172.21.15.98:0/2834070656 learned_addr learned my addr 172.21.15.98:0/2834070656 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecd87d640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdec81050b0 0x7fdec8102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdec8102260 0x7fdec8102900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecd07c640 1 -- 172.21.15.98:0/2834070656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdec8102260 msgr2=0x7fdec8102900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecd07c640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdec8102260 0x7fdec8102900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecd07c640 1 -- 172.21.15.98:0/2834070656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdec81050b0 msgr2=0x7fdec8102e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecd07c640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdec81050b0 0x7fdec8102e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecd07c640 1 -- 172.21.15.98:0/2834070656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdec8103480 con 0x7fdec8101760 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdec8102260 0x7fdec8102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:35:22.395 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecd07c640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fdec8101760 0x7fdec81041f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8005760 tx=0x7fdeb8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb803f030 con 0x7fdec8101760 2024-04-25T16:35:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeb8004440 con 0x7fdec8101760 2024-04-25T16:35:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecf2e0640 1 -- 172.21.15.98:0/2834070656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdec8103710 con 0x7fdec8101760 2024-04-25T16:35:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8035070 con 0x7fdec8101760 2024-04-25T16:35:22.396 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.394+0000 7fdecf2e0640 1 -- 172.21.15.98:0/2834070656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdec81a45c0 con 0x7fdec8101760 2024-04-25T16:35:22.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fdeb8042030 con 0x7fdec8101760 2024-04-25T16:35:22.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fded6190740 1 Processor -- start 2024-04-25T16:35:22.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fded6190740 1 -- start start 2024-04-25T16:35:22.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fded6190740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326817a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fded6190740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326817fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fded6190740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 0x55e326818c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fded6190740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e32677cd80 con 0x55e32644d4d0 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fded6190740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e3267368b0 con 0x55e326476390 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fded6190740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e326778d40 con 0x55e3264556b0 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fdebe7fc640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdeb40afb70 0x7fdeb40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fdecd07c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326817fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.397+0000 7fdecd87d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326817a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd87d640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdeb40afb70 0x7fdeb40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd07c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326817fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50250/0 (socket says 172.21.15.98:50250) 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd07c640 1 -- 172.21.15.98:0/4242689706 learned_addr learned my addr 172.21.15.98:0/4242689706 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9265+0+0 (secure 0 0 0) 0x7fdeb80ff4c0 con 0x7fdec8101760 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecc87b640 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 0x55e326818c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd07c640 1 -- 172.21.15.98:0/4242689706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 msgr2=0x55e326818c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd07c640 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 0x55e326818c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd07c640 1 -- 172.21.15.98:0/4242689706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 msgr2=0x55e326817a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:22.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd07c640 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326817a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd07c640 1 -- 172.21.15.98:0/4242689706 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e326707a40 con 0x55e32644d4d0 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecc87b640 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 0x55e326818c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd87d640 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326817a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd87d640 1 --2- 172.21.15.98:0/2834070656 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fdeb40afb70 0x7fdeb40b2080 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdec0007fd0 tx=0x7fdec00078b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdecd07c640 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326817fe0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80c7230 tx=0x7fdeb80f5c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3366a76c635aeb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdeb37fe640 1 -- 172.21.15.98:0/4242689706 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8049070 con 0x55e32644d4d0 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdeb37fe640 1 -- 172.21.15.98:0/4242689706 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeb8094e50 con 0x55e32644d4d0 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.398+0000 7fdeb37fe640 1 -- 172.21.15.98:0/4242689706 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80d0390 con 0x55e32644d4d0 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 -- 172.21.15.98:0/4242689706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 msgr2=0x55e326817fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326817fe0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80c7230 tx=0x7fdeb80f5c80 comp rx=0 tx=0).stop 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 -- 172.21.15.98:0/4242689706 shutdown_connections 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 0x55e326818c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326817fe0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 --2- 172.21.15.98:0/4242689706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326817a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.404 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 -- 172.21.15.98:0/4242689706 >> 172.21.15.98:0/4242689706 conn(0x55e32646a4e0 msgr2=0x55e326813270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 -- 172.21.15.98:0/4242689706 shutdown_connections 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.399+0000 7fded6190740 1 -- 172.21.15.98:0/4242689706 wait complete. 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 1 -- start start 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 1 Processor -- start 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326821640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 0x55e326821b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326828ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x55e326819540 con 0x55e32644d4d0 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x55e326703d10 con 0x55e326476390 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x55e3268196c0 con 0x55e3264556b0 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fded6190740 10 client.0 _ll_register_callbacks cb 0x55e326730210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fdecd07c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326821640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fdecd07c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326821640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:50256/0 (socket says 172.21.15.98:50256) 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fdecd07c640 1 -- 172.21.15.98:0/2691354310 learned_addr learned my addr 172.21.15.98:0/2691354310 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:35:22.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fdecd87d640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326828ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.400+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 0x55e326821b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdecd07c640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 msgr2=0x55e326821b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdecd07c640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55e3264556b0 0x55e326821b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdecd07c640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 msgr2=0x55e326828ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdecd07c640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326828ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdecd07c640 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3265587d0 con 0x55e32644d4d0 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdecd87d640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55e326476390 0x55e326828ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdecd07c640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x55e32644d4d0 0x55e326821640 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000c00 tx=0x7fdeb8003f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1caa964b9438b507 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdeb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.98:3300/0 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8049070 con 0x55e32644d4d0 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 2 ==== osd_map(86..86 src has 1..86) ==== 9265+0+0 (secure 0 0 0) 0x7fdeb80c7790 con 0x55e32644d4d0 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fded6190740 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdeb8094e50 con 0x55e32644d4d0 2024-04-25T16:35:22.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fdeb80c7530 con 0x55e32644d4d0 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fded6190740 10 client.16329 fetch_fsmap learned FSMap version 37 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fded6190740 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e32648e3f0 con 0x55e32644d4d0 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdeb8094e50 con 0x55e32644d4d0 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 5 ==== fsmap(e 37) ==== 8362+0+0 (secure 0 0 0) 0x7fdeb80cff90 con 0x55e32644d4d0 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fded6190740 10 client.16329 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.401+0000 7fded6190740 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.f 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.402+0000 7fded6190740 20 client.16329 populate_metadata read hostname 'smithi098' 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.402+0000 7fded6190740 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.402+0000 7fded6190740 4 client.16329 mds_command: new command op to 24445 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.402+0000 7fded6190740 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e32678a990 con 0x55e326434f60 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.402+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.402+0000 7fdeb0ff9640 1 client.16329 ms_handle_refused on v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:22.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.402+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T16:35:22.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:22 smithi203 ceph-mon[33115]: pgmap v443: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 222 KiB/s wr, 86 op/s 2024-04-25T16:35:22.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:22 smithi098 ceph-mon[29162]: pgmap v443: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 222 KiB/s wr, 86 op/s 2024-04-25T16:35:22.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:22 smithi173 ceph-mon[32556]: pgmap v443: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 222 KiB/s wr, 86 op/s 2024-04-25T16:35:22.601 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.603+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:22.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.603+0000 7fdeb0ff9640 1 client.16329 ms_handle_refused on v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:22.602 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:22.603+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T16:35:23.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:23.004+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:23.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:23.004+0000 7fdeb0ff9640 1 client.16329 ms_handle_refused on v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:23.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:23.004+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T16:35:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:23.805+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:23.805+0000 7fdeb0ff9640 1 client.16329 ms_handle_refused on v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:23.804 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:23.805+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T16:35:24.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:24 smithi203 ceph-mon[33115]: pgmap v444: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 222 KiB/s wr, 88 op/s 2024-04-25T16:35:24.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:24 smithi098 ceph-mon[29162]: pgmap v444: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 222 KiB/s wr, 88 op/s 2024-04-25T16:35:24.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:24 smithi173 ceph-mon[32556]: pgmap v444: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 222 KiB/s wr, 88 op/s 2024-04-25T16:35:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:25.406+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:25.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:25.406+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T16:35:25.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:25.406+0000 7fdeb0ff9640 1 client.16329 ms_handle_refused on v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:25.951 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:25 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:35:25.951 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:25 smithi203 systemd[1]: Stopped Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:25.952 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:25 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 7.345s CPU time. 2024-04-25T16:35:25.952 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:25 smithi203 systemd[1]: Starting Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:35:25.952 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:25 smithi203 podman[66942]: 2024-04-25 16:35:25.834333313 +0000 UTC m=+0.018617903 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:35:25.952 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:25 smithi203 podman[66942]: 2024-04-25 16:35:25.953833258 +0000 UTC m=+0.138117831 container create 9afcae795113be5dd0d4e97be47aa3f499767f50b7b660e703a65f8922957bf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:35:26.231 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:26 smithi203 ceph-mon[33115]: pgmap v445: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 221 KiB/s wr, 84 op/s 2024-04-25T16:35:26.231 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:26 smithi203 podman[66942]: 2024-04-25 16:35:26.198472649 +0000 UTC m=+0.382757222 container init 9afcae795113be5dd0d4e97be47aa3f499767f50b7b660e703a65f8922957bf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-04-25T16:35:26.231 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:26 smithi203 podman[66942]: 2024-04-25 16:35:26.201209544 +0000 UTC m=+0.385494121 container start 9afcae795113be5dd0d4e97be47aa3f499767f50b7b660e703a65f8922957bf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:35:26.490 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:26 smithi203 ceph-mds[66961]: -- 172.21.15.203:0/1088438498 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e1b4cf9a40 con 0x55e1b4d41180 2024-04-25T16:35:26.490 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:26 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: starting mds.f at 2024-04-25T16:35:26.490 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:26 smithi203 bash[66942]: 9afcae795113be5dd0d4e97be47aa3f499767f50b7b660e703a65f8922957bf8 2024-04-25T16:35:26.490 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:26 smithi203 systemd[1]: Started Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:26 smithi098 ceph-mon[29162]: pgmap v445: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 221 KiB/s wr, 84 op/s 2024-04-25T16:35:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:26 smithi173 ceph-mon[32556]: pgmap v445: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 221 KiB/s wr, 84 op/s 2024-04-25T16:35:27.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:27 smithi203 ceph-mon[33115]: daemon mds.f restarted 2024-04-25T16:35:27.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:27 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:27.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:27 smithi203 ceph-mon[33115]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T16:35:27.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:27 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:boot 2024-04-25T16:35:27.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:27 smithi203 ceph-mon[33115]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:35:27.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:27 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:27.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:27 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:27.424 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:27 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:27.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:27 smithi098 ceph-mon[29162]: daemon mds.f restarted 2024-04-25T16:35:27.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:27 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:27.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:27 smithi098 ceph-mon[29162]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T16:35:27.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:27 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:boot 2024-04-25T16:35:27.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:27 smithi098 ceph-mon[29162]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:35:27.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:27 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:27.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:27 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:27.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:27 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:27.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:27 smithi173 ceph-mon[32556]: daemon mds.f restarted 2024-04-25T16:35:27.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:27 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:27.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:27 smithi173 ceph-mon[32556]: osdmap e87: 12 total, 12 up, 12 in 2024-04-25T16:35:27.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:27 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:boot 2024-04-25T16:35:27.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:27 smithi173 ceph-mon[32556]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:35:27.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:27 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:27.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:27 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:27.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:27 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:28.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:28 smithi203 ceph-mon[33115]: pgmap v447: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T16:35:28.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:28 smithi098 ceph-mon[29162]: pgmap v447: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T16:35:28.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:28 smithi173 ceph-mon[32556]: pgmap v447: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T16:35:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:28.610+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:28.609 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:28.611+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:35:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:28.611+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:35:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:28.611+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:35:28.610 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:28.611+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-25T16:35:29.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:29 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:resolve 2024-04-25T16:35:29.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:29 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:29.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:29 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:resolve 2024-04-25T16:35:29.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:29 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:29.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:29 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:resolve 2024-04-25T16:35:29.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:29 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:30.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:30 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999979 2024-04-25T16:35:30.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:30 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:reconnect 2024-04-25T16:35:30.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:30 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:30.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:30 smithi098 ceph-mon[29162]: pgmap v448: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 409 B/s wr, 17 op/s 2024-04-25T16:35:30.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:30 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999979 2024-04-25T16:35:30.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:30 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:reconnect 2024-04-25T16:35:30.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:30 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:30.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:30 smithi203 ceph-mon[33115]: pgmap v448: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 409 B/s wr, 17 op/s 2024-04-25T16:35:30.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:30 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999979 2024-04-25T16:35:30.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:30 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:reconnect 2024-04-25T16:35:30.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:30 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:30.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:30 smithi173 ceph-mon[32556]: pgmap v448: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 409 B/s wr, 17 op/s 2024-04-25T16:35:31.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:31 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:rejoin 2024-04-25T16:35:31.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:31 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:31.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:31 smithi203 ceph-mon[33115]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:35:31.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:31 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:rejoin 2024-04-25T16:35:31.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:31 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:31.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:31 smithi098 ceph-mon[29162]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:35:31.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:rejoin 2024-04-25T16:35:31.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:31.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-mon[32556]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:35:31.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3819244700 time 2024-04-25T16:35:31.214239+0000 2024-04-25T16:35:31.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:31.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:31.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3823a78ba6] 2024-04-25T16:35:31.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3823a78dd1] 2024-04-25T16:35:31.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55d6dd00326b] 2024-04-25T16:35:31.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55d6dcfa2428] 2024-04-25T16:35:31.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55d6dcfa38a9] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55d6dcfa59c3] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55d6dce7185d] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55d6dce73e8e] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55d6dce74894] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55d6dce5f05b] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3823bf5ac6] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 12: (DispatchQueue::entry()+0xb27) [0x7f3823bf30b9] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3823cddccb] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 14: (Thread::entry_wrapper()+0x43) [0x7f3823a4772d] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3823a47749] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f38225eb1ca] 2024-04-25T16:35:31.515 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 17: clone() 2024-04-25T16:35:31.517 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.215+0000 7f3819244700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3819244700 time 2024-04-25T16:35:31.214239+0000 2024-04-25T16:35:31.517 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:31.517 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.517 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:31.517 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3823a78ba6] 2024-04-25T16:35:31.517 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3823a78dd1] 2024-04-25T16:35:31.517 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55d6dd00326b] 2024-04-25T16:35:31.517 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55d6dcfa2428] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55d6dcfa38a9] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55d6dcfa59c3] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55d6dce7185d] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55d6dce73e8e] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55d6dce74894] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55d6dce5f05b] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3823bf5ac6] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 12: (DispatchQueue::entry()+0xb27) [0x7f3823bf30b9] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3823cddccb] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 14: (Thread::entry_wrapper()+0x43) [0x7f3823a4772d] 2024-04-25T16:35:31.518 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3823a47749] 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f38225eb1ca] 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 17: clone() 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: *** Caught signal (Aborted) ** 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: in thread 7f3819244700 thread_name:ms_dispatch 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38225f5d20] 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2: gsignal() 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 3: abort() 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3823a78ced] 2024-04-25T16:35:31.519 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3823a78dd1] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55d6dd00326b] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55d6dcfa2428] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55d6dcfa38a9] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55d6dcfa59c3] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55d6dce7185d] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55d6dce73e8e] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55d6dce74894] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55d6dce5f05b] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3823bf5ac6] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 15: (DispatchQueue::entry()+0xb27) [0x7f3823bf30b9] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3823cddccb] 2024-04-25T16:35:31.520 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 17: (Thread::entry_wrapper()+0x43) [0x7f3823a4772d] 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3823a47749] 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f38225eb1ca] 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 20: clone() 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.217+0000 7f3819244700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: in thread 7f3819244700 thread_name:ms_dispatch 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38225f5d20] 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2: gsignal() 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 3: abort() 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3823a78ced] 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3823a78dd1] 2024-04-25T16:35:31.521 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55d6dd00326b] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55d6dcfa2428] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55d6dcfa38a9] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55d6dcfa59c3] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55d6dce7185d] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55d6dce73e8e] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55d6dce74894] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55d6dce5f05b] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3823bf5ac6] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 15: (DispatchQueue::entry()+0xb27) [0x7f3823bf30b9] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3823cddccb] 2024-04-25T16:35:31.522 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 17: (Thread::entry_wrapper()+0x43) [0x7f3823a4772d] 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3823a47749] 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f38225eb1ca] 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 20: clone() 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: -1> 2024-04-25T16:35:31.215+0000 7f3819244700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3819244700 time 2024-04-25T16:35:31.214239+0000 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3823a78ba6] 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3823a78dd1] 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55d6dd00326b] 2024-04-25T16:35:31.523 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55d6dcfa2428] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55d6dcfa38a9] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55d6dcfa59c3] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55d6dce7185d] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55d6dce73e8e] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55d6dce74894] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55d6dce5f05b] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3823bf5ac6] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 12: (DispatchQueue::entry()+0xb27) [0x7f3823bf30b9] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3823cddccb] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 14: (Thread::entry_wrapper()+0x43) [0x7f3823a4772d] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3823a47749] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f38225eb1ca] 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 17: clone() 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.524 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 0> 2024-04-25T16:35:31.217+0000 7f3819244700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: in thread 7f3819244700 thread_name:ms_dispatch 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38225f5d20] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2: gsignal() 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 3: abort() 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3823a78ced] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3823a78dd1] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55d6dd00326b] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55d6dcfa2428] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55d6dcfa38a9] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55d6dcfa59c3] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55d6dce7185d] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55d6dce73e8e] 2024-04-25T16:35:31.525 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55d6dce74894] 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55d6dce5f05b] 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3823bf5ac6] 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 15: (DispatchQueue::entry()+0xb27) [0x7f3823bf30b9] 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3823cddccb] 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 17: (Thread::entry_wrapper()+0x43) [0x7f3823a4772d] 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3823a47749] 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f38225eb1ca] 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 20: clone() 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: -1> 2024-04-25T16:35:31.215+0000 7f3819244700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f3819244700 time 2024-04-25T16:35:31.214239+0000 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.526 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3823a78ba6] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3823a78dd1] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55d6dd00326b] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55d6dcfa2428] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55d6dcfa38a9] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55d6dcfa59c3] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55d6dce7185d] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55d6dce73e8e] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55d6dce74894] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55d6dce5f05b] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3823bf5ac6] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 12: (DispatchQueue::entry()+0xb27) [0x7f3823bf30b9] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3823cddccb] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 14: (Thread::entry_wrapper()+0x43) [0x7f3823a4772d] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3823a47749] 2024-04-25T16:35:31.527 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f38225eb1ca] 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 17: clone() 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 0> 2024-04-25T16:35:31.217+0000 7f3819244700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: in thread 7f3819244700 thread_name:ms_dispatch 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f38225f5d20] 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2: gsignal() 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 3: abort() 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3823a78ced] 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3823a78dd1] 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55d6dd00326b] 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55d6dcfa2428] 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55d6dcfa38a9] 2024-04-25T16:35:31.528 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55d6dcfa59c3] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55d6dce7185d] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55d6dce73e8e] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55d6dce74894] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55d6dce5f05b] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3823bf5ac6] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 15: (DispatchQueue::entry()+0xb27) [0x7f3823bf30b9] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3823cddccb] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 17: (Thread::entry_wrapper()+0x43) [0x7f3823a4772d] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3823a47749] 2024-04-25T16:35:31.529 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f38225eb1ca] 2024-04-25T16:35:31.530 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 20: clone() 2024-04-25T16:35:31.530 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:31.530 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65113]: 2024-04-25T16:35:31.530 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:31 smithi173 podman[65155]: 2024-04-25 16:35:31.276852816 +0000 UTC m=+0.018710368 container died 123225c72bac0aaabf6152c344a11847189663fe099c6026e91e05d758d703e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T16:35:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:32 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:32 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:35:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:32 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:active 2024-04-25T16:35:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:32 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:32 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:32.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:32 smithi173 ceph-mon[32556]: pgmap v449: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 511 B/s wr, 18 op/s 2024-04-25T16:35:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:32 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:32 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:35:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:32 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:active 2024-04-25T16:35:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:32 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:32 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:32 smithi203 ceph-mon[33115]: pgmap v449: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 511 B/s wr, 18 op/s 2024-04-25T16:35:32.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:32 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:32.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:32 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:35:32.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:32 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3538054402,v1:172.21.15.173:6839/3538054402] up:active 2024-04-25T16:35:32.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:32 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:32.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:32 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:32.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:32 smithi098 ceph-mon[29162]: pgmap v449: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 511 B/s wr, 18 op/s 2024-04-25T16:35:32.513 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:32 smithi173 podman[65155]: 2024-04-25 16:35:32.231918696 +0000 UTC m=+0.973776300 container remove 123225c72bac0aaabf6152c344a11847189663fe099c6026e91e05d758d703e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T16:35:32.513 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:32 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:35:32.513 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:32 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-25T16:35:32.513 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:32 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 4.363s CPU time. 2024-04-25T16:35:34.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:34 smithi203 ceph-mon[33115]: pgmap v450: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.5 KiB/s wr, 24 op/s 2024-04-25T16:35:34.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:34 smithi098 ceph-mon[29162]: pgmap v450: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.5 KiB/s wr, 24 op/s 2024-04-25T16:35:34.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:34 smithi173 ceph-mon[32556]: pgmap v450: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 9.5 KiB/s wr, 24 op/s 2024-04-25T16:35:35.015 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:35.016+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:35:35.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:35.017+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:35:35.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:35.017+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:35:35.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:35.017+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:35:35.016 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:35.017+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-25T16:35:36.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:36 smithi203 ceph-mon[33115]: pgmap v451: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.5 KiB/s wr, 25 op/s 2024-04-25T16:35:36.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:36 smithi098 ceph-mon[29162]: pgmap v451: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.5 KiB/s wr, 25 op/s 2024-04-25T16:35:36.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:36 smithi173 ceph-mon[32556]: pgmap v451: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.5 KiB/s wr, 25 op/s 2024-04-25T16:35:38.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:38 smithi203 ceph-mon[33115]: pgmap v452: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:35:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:38 smithi098 ceph-mon[29162]: pgmap v452: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:35:38.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:38 smithi173 ceph-mon[32556]: pgmap v452: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 8.1 KiB/s wr, 23 op/s 2024-04-25T16:35:40.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:40 smithi203 ceph-mon[33115]: pgmap v453: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 KiB/s wr, 22 op/s 2024-04-25T16:35:40.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:40 smithi098 ceph-mon[29162]: pgmap v453: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 KiB/s wr, 22 op/s 2024-04-25T16:35:40.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:40 smithi173 ceph-mon[32556]: pgmap v453: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 KiB/s wr, 22 op/s 2024-04-25T16:35:42.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:42 smithi203 ceph-mon[33115]: pgmap v454: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 KiB/s wr, 16 op/s 2024-04-25T16:35:42.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:42 smithi098 ceph-mon[29162]: pgmap v454: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 KiB/s wr, 16 op/s 2024-04-25T16:35:42.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:42 smithi173 ceph-mon[32556]: pgmap v454: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.0 KiB/s wr, 16 op/s 2024-04-25T16:35:42.797 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:42 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:35:42.797 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:42 smithi173 systemd[1]: Stopped Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:42.797 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:42 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 4.363s CPU time. 2024-04-25T16:35:42.797 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:42 smithi173 systemd[1]: Starting Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:35:42.797 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:42 smithi173 podman[65254]: 2024-04-25 16:35:42.656927382 +0000 UTC m=+0.018570599 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:35:42.797 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:42 smithi173 podman[65254]: 2024-04-25 16:35:42.799005443 +0000 UTC m=+0.160648644 container create 6353aaf641f6a53e57cbed60ef1a4841c5a06e78fb04a9367b42d0c7a5714000 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T16:35:43.076 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:43 smithi173 podman[65254]: 2024-04-25 16:35:43.043747974 +0000 UTC m=+0.405391175 container init 6353aaf641f6a53e57cbed60ef1a4841c5a06e78fb04a9367b42d0c7a5714000 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T16:35:43.077 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:43 smithi173 podman[65254]: 2024-04-25 16:35:43.046760883 +0000 UTC m=+0.408404092 container start 6353aaf641f6a53e57cbed60ef1a4841c5a06e78fb04a9367b42d0c7a5714000 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-25T16:35:43.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:43 smithi203 ceph-mon[33115]: daemon mds.b restarted 2024-04-25T16:35:43.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:43 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:43 smithi098 ceph-mon[29162]: daemon mds.b restarted 2024-04-25T16:35:43.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:43 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:43.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:43 smithi173 ceph-mon[32556]: daemon mds.b restarted 2024-04-25T16:35:43.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:43 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:43.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:43 smithi173 ceph-mds[65273]: -- 172.21.15.173:0/2623395848 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56185e08ba40 con 0x56185e0d3a80 2024-04-25T16:35:43.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:43 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: starting mds.b at 2024-04-25T16:35:43.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:43 smithi173 bash[65254]: 6353aaf641f6a53e57cbed60ef1a4841c5a06e78fb04a9367b42d0c7a5714000 2024-04-25T16:35:43.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:35:43 smithi173 systemd[1]: Started Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:44.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:44 smithi203 ceph-mon[33115]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T16:35:44.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:44 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:boot 2024-04-25T16:35:44.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:44 smithi203 ceph-mon[33115]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-25T16:35:44.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:44 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:44.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:44 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:35:44.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:44 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:44.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:44 smithi203 ceph-mon[33115]: pgmap v456: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 102 B/s wr, 11 op/s 2024-04-25T16:35:44.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:44 smithi098 ceph-mon[29162]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T16:35:44.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:44 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:boot 2024-04-25T16:35:44.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:44 smithi098 ceph-mon[29162]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-25T16:35:44.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:44 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:44.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:44 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:35:44.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:44 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:44.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:44 smithi098 ceph-mon[29162]: pgmap v456: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 102 B/s wr, 11 op/s 2024-04-25T16:35:44.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:44 smithi173 ceph-mon[32556]: osdmap e88: 12 total, 12 up, 12 in 2024-04-25T16:35:44.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:44 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:boot 2024-04-25T16:35:44.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:44 smithi173 ceph-mon[32556]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-25T16:35:44.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:44 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:44.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:44 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:35:44.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:44 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:44.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:44 smithi173 ceph-mon[32556]: pgmap v456: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 102 B/s wr, 11 op/s 2024-04-25T16:35:45.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:45 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:resolve 2024-04-25T16:35:45.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:45 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:45.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:45 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:resolve 2024-04-25T16:35:45.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:45 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:45.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:45 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:resolve 2024-04-25T16:35:45.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:45 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:35:46.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:46 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999976 2024-04-25T16:35:46.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:46 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:reconnect 2024-04-25T16:35:46.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:46 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:46.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:46 smithi203 ceph-mon[33115]: pgmap v457: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 16 op/s 2024-04-25T16:35:46.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:46 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999976 2024-04-25T16:35:46.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:46 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:reconnect 2024-04-25T16:35:46.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:46 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:46.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:46 smithi098 ceph-mon[29162]: pgmap v457: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 16 op/s 2024-04-25T16:35:46.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:46 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999976 2024-04-25T16:35:46.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:46 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:reconnect 2024-04-25T16:35:46.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:46 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:35:46.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:46 smithi173 ceph-mon[32556]: pgmap v457: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 102 B/s wr, 16 op/s 2024-04-25T16:35:47.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:rejoin 2024-04-25T16:35:47.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:47.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-mon[33115]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:35:47.438 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fbdff115700 time 2024-04-25T16:35:47.229512+0000 2024-04-25T16:35:47.438 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:47.438 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbe09949ba6] 2024-04-25T16:35:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe09949dd1] 2024-04-25T16:35:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e1b36a226b] 2024-04-25T16:35:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e1b3641428] 2024-04-25T16:35:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e1b36428a9] 2024-04-25T16:35:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e1b36449c3] 2024-04-25T16:35:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e1b351085d] 2024-04-25T16:35:47.439 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e1b3512e8e] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e1b3513894] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e1b34fe05b] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe09ac6ac6] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 12: (DispatchQueue::entry()+0xb27) [0x7fbe09ac40b9] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe09baeccb] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 14: (Thread::entry_wrapper()+0x43) [0x7fbe0991872d] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 15: (Thread::_entry_func(void*)+0xd) [0x7fbe09918749] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fbe084bc1ca] 2024-04-25T16:35:47.440 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 17: clone() 2024-04-25T16:35:47.441 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: *** Caught signal (Aborted) ** 2024-04-25T16:35:47.441 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: in thread 7fbdff115700 thread_name:ms_dispatch 2024-04-25T16:35:47.442 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.231+0000 7fbdff115700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fbdff115700 time 2024-04-25T16:35:47.229512+0000 2024-04-25T16:35:47.442 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:47.442 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.442 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbe09949ba6] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe09949dd1] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e1b36a226b] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e1b3641428] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e1b36428a9] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e1b36449c3] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e1b351085d] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e1b3512e8e] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e1b3513894] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e1b34fe05b] 2024-04-25T16:35:47.443 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe09ac6ac6] 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 12: (DispatchQueue::entry()+0xb27) [0x7fbe09ac40b9] 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe09baeccb] 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 14: (Thread::entry_wrapper()+0x43) [0x7fbe0991872d] 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 15: (Thread::_entry_func(void*)+0xd) [0x7fbe09918749] 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fbe084bc1ca] 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 17: clone() 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:47.444 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbe084c6d20] 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2: gsignal() 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 3: abort() 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbe09949ced] 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe09949dd1] 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e1b36a226b] 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e1b3641428] 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e1b36428a9] 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e1b36449c3] 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e1b351085d] 2024-04-25T16:35:47.445 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e1b3512e8e] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e1b3513894] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e1b34fe05b] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe09ac6ac6] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 15: (DispatchQueue::entry()+0xb27) [0x7fbe09ac40b9] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe09baeccb] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 17: (Thread::entry_wrapper()+0x43) [0x7fbe0991872d] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 18: (Thread::_entry_func(void*)+0xd) [0x7fbe09918749] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fbe084bc1ca] 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 20: clone() 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.235+0000 7fbdff115700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:47.446 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: in thread 7fbdff115700 thread_name:ms_dispatch 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbe084c6d20] 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2: gsignal() 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 3: abort() 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbe09949ced] 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe09949dd1] 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e1b36a226b] 2024-04-25T16:35:47.447 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e1b3641428] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e1b36428a9] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e1b36449c3] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e1b351085d] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e1b3512e8e] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e1b3513894] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e1b34fe05b] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe09ac6ac6] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 15: (DispatchQueue::entry()+0xb27) [0x7fbe09ac40b9] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe09baeccb] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 17: (Thread::entry_wrapper()+0x43) [0x7fbe0991872d] 2024-04-25T16:35:47.448 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 18: (Thread::_entry_func(void*)+0xd) [0x7fbe09918749] 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fbe084bc1ca] 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 20: clone() 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: -1> 2024-04-25T16:35:47.231+0000 7fbdff115700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fbdff115700 time 2024-04-25T16:35:47.229512+0000 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:47.449 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbe09949ba6] 2024-04-25T16:35:47.450 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe09949dd1] 2024-04-25T16:35:47.450 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e1b36a226b] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e1b3641428] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e1b36428a9] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e1b36449c3] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e1b351085d] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e1b3512e8e] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e1b3513894] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e1b34fe05b] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe09ac6ac6] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 12: (DispatchQueue::entry()+0xb27) [0x7fbe09ac40b9] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe09baeccb] 2024-04-25T16:35:47.451 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 14: (Thread::entry_wrapper()+0x43) [0x7fbe0991872d] 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 15: (Thread::_entry_func(void*)+0xd) [0x7fbe09918749] 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fbe084bc1ca] 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 17: clone() 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 0> 2024-04-25T16:35:47.235+0000 7fbdff115700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: in thread 7fbdff115700 thread_name:ms_dispatch 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbe084c6d20] 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2: gsignal() 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 3: abort() 2024-04-25T16:35:47.452 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbe09949ced] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe09949dd1] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e1b36a226b] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e1b3641428] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e1b36428a9] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e1b36449c3] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e1b351085d] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e1b3512e8e] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e1b3513894] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e1b34fe05b] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe09ac6ac6] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 15: (DispatchQueue::entry()+0xb27) [0x7fbe09ac40b9] 2024-04-25T16:35:47.453 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe09baeccb] 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 17: (Thread::entry_wrapper()+0x43) [0x7fbe0991872d] 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 18: (Thread::_entry_func(void*)+0xd) [0x7fbe09918749] 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fbe084bc1ca] 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 20: clone() 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: -4> 2024-04-25T16:35:47.231+0000 7fbdff115700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fbdff115700 time 2024-04-25T16:35:47.229512+0000 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.454 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fbe09949ba6] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe09949dd1] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55e1b36a226b] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e1b3641428] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e1b36428a9] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e1b36449c3] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e1b351085d] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e1b3512e8e] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e1b3513894] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e1b34fe05b] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe09ac6ac6] 2024-04-25T16:35:47.455 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 12: (DispatchQueue::entry()+0xb27) [0x7fbe09ac40b9] 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe09baeccb] 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 14: (Thread::entry_wrapper()+0x43) [0x7fbe0991872d] 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 15: (Thread::_entry_func(void*)+0xd) [0x7fbe09918749] 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fbe084bc1ca] 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 17: clone() 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: -3> 2024-04-25T16:35:47.235+0000 7fbdff115700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: in thread 7fbdff115700 thread_name:ms_dispatch 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fbe084c6d20] 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2: gsignal() 2024-04-25T16:35:47.456 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 3: abort() 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fbe09949ced] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fbe09949dd1] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55e1b36a226b] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55e1b3641428] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55e1b36428a9] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55e1b36449c3] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55e1b351085d] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55e1b3512e8e] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55e1b3513894] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55e1b34fe05b] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fbe09ac6ac6] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 15: (DispatchQueue::entry()+0xb27) [0x7fbe09ac40b9] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fbe09baeccb] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 17: (Thread::entry_wrapper()+0x43) [0x7fbe0991872d] 2024-04-25T16:35:47.457 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 18: (Thread::_entry_func(void*)+0xd) [0x7fbe09918749] 2024-04-25T16:35:47.458 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fbe084bc1ca] 2024-04-25T16:35:47.458 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 20: clone() 2024-04-25T16:35:47.458 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:35:47.458 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[66957]: 2024-04-25T16:35:47.458 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 podman[67011]: 2024-04-25 16:35:47.298986666 +0000 UTC m=+0.020038072 container died 9afcae795113be5dd0d4e97be47aa3f499767f50b7b660e703a65f8922957bf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T16:35:47.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:47 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:rejoin 2024-04-25T16:35:47.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:47 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:47.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:47 smithi098 ceph-mon[29162]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:35:47.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:47 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:rejoin 2024-04-25T16:35:47.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:47 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:35:47.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:47 smithi173 ceph-mon[32556]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:35:47.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:47.823+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:47.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:47.823+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:35:47.822 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:35:47.823+0000 7fdeb0ff9640 1 client.16329 ms_handle_refused on v2:172.21.15.203:6834/2230353021 2024-04-25T16:35:48.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:48 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:48.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:48 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:35:48.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:48 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:active 2024-04-25T16:35:48.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:48 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:48.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:48 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:48.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:48 smithi203 ceph-mon[33115]: pgmap v458: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.4 KiB/s wr, 21 op/s 2024-04-25T16:35:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 podman[67011]: 2024-04-25 16:35:47.912013997 +0000 UTC m=+0.633065397 container remove 9afcae795113be5dd0d4e97be47aa3f499767f50b7b660e703a65f8922957bf8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T16:35:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:47 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:35:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:48 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:35:48.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:48 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 4.419s CPU time. 2024-04-25T16:35:48.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:48 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:48.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:48 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:35:48.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:48 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:active 2024-04-25T16:35:48.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:48 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:48.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:48 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:48.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:48 smithi098 ceph-mon[29162]: pgmap v458: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.4 KiB/s wr, 21 op/s 2024-04-25T16:35:48.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:48 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:35:48.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:48 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:35:48.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:48 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/890593273,v1:172.21.15.203:6835/890593273] up:active 2024-04-25T16:35:48.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:48 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:35:48.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:48 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:35:48.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:48 smithi173 ceph-mon[32556]: pgmap v458: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 9.4 KiB/s wr, 21 op/s 2024-04-25T16:35:50.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:50 smithi203 ceph-mon[33115]: pgmap v459: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.5 KiB/s wr, 23 op/s 2024-04-25T16:35:50.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:50 smithi098 ceph-mon[29162]: pgmap v459: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.5 KiB/s wr, 23 op/s 2024-04-25T16:35:50.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:50 smithi173 ceph-mon[32556]: pgmap v459: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 9.5 KiB/s wr, 23 op/s 2024-04-25T16:35:51.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:51 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:35:51.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:51 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:35:51.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:51 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:35:52.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:35:52.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:35:52.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:35:52.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:52 smithi203 ceph-mon[33115]: pgmap v460: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.8 KiB/s wr, 25 op/s 2024-04-25T16:35:52.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:35:52.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:35:52.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:35:52.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:52 smithi098 ceph-mon[29162]: pgmap v460: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.8 KiB/s wr, 25 op/s 2024-04-25T16:35:52.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:35:52.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:35:52.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:35:52.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:52 smithi173 ceph-mon[32556]: pgmap v460: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 9.8 KiB/s wr, 25 op/s 2024-04-25T16:35:54.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:54 smithi203 ceph-mon[33115]: pgmap v461: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 KiB/s wr, 24 op/s 2024-04-25T16:35:54.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:54 smithi098 ceph-mon[29162]: pgmap v461: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 KiB/s wr, 24 op/s 2024-04-25T16:35:54.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:54 smithi173 ceph-mon[32556]: pgmap v461: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 9.1 KiB/s wr, 24 op/s 2024-04-25T16:35:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:56 smithi203 ceph-mon[33115]: pgmap v462: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 KiB/s wr, 21 op/s 2024-04-25T16:35:56.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:56 smithi098 ceph-mon[29162]: pgmap v462: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 KiB/s wr, 21 op/s 2024-04-25T16:35:56.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:56 smithi173 ceph-mon[32556]: pgmap v462: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 KiB/s wr, 21 op/s 2024-04-25T16:35:58.426 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:58 smithi203 ceph-mon[33115]: pgmap v463: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-25T16:35:58.426 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:35:58.427 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 systemd[1]: Stopped Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:58.427 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 4.419s CPU time. 2024-04-25T16:35:58.427 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 systemd[1]: Starting Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:35:58.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:58 smithi098 ceph-mon[29162]: pgmap v463: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-25T16:35:58.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:58 smithi173 ceph-mon[32556]: pgmap v463: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 KiB/s wr, 19 op/s 2024-04-25T16:35:58.680 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 podman[67233]: 2024-04-25 16:35:58.331421772 +0000 UTC m=+0.018556514 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:35:58.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 podman[67233]: 2024-04-25 16:35:58.4549305 +0000 UTC m=+0.142065240 container create 3e5f85b27e25c56249efa27cea5d11a03b9e58f4e1dae9204fe574c06cae87e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T16:35:58.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 podman[67233]: 2024-04-25 16:35:58.682793422 +0000 UTC m=+0.369928164 container init 3e5f85b27e25c56249efa27cea5d11a03b9e58f4e1dae9204fe574c06cae87e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True) 2024-04-25T16:35:58.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 podman[67233]: 2024-04-25 16:35:58.68549612 +0000 UTC m=+0.372630871 container start 3e5f85b27e25c56249efa27cea5d11a03b9e58f4e1dae9204fe574c06cae87e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T16:35:58.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 ceph-mds[67251]: -- 172.21.15.203:0/1207135805 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5605d1b79a40 con 0x5605d1bc1600 2024-04-25T16:35:58.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: starting mds.f at 2024-04-25T16:35:58.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 bash[67233]: 3e5f85b27e25c56249efa27cea5d11a03b9e58f4e1dae9204fe574c06cae87e0 2024-04-25T16:35:58.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:35:58 smithi203 systemd[1]: Started Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:35:59.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:59 smithi203 ceph-mon[33115]: daemon mds.f restarted 2024-04-25T16:35:59.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:59 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:59.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:59 smithi203 ceph-mon[33115]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T16:35:59.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:59 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:boot 2024-04-25T16:35:59.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:59 smithi203 ceph-mon[33115]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:35:59.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:59 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:59.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:59.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:35:59 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:59.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:59 smithi098 ceph-mon[29162]: daemon mds.f restarted 2024-04-25T16:35:59.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:59 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:59.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:59 smithi098 ceph-mon[29162]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T16:35:59.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:59 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:boot 2024-04-25T16:35:59.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:59 smithi098 ceph-mon[29162]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:35:59.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:59 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:59.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:59.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:35:59 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:35:59.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:59 smithi173 ceph-mon[32556]: daemon mds.f restarted 2024-04-25T16:35:59.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:59 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:35:59.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:59 smithi173 ceph-mon[32556]: osdmap e89: 12 total, 12 up, 12 in 2024-04-25T16:35:59.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:59 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:boot 2024-04-25T16:35:59.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:59 smithi173 ceph-mon[32556]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:35:59.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:59 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:35:59.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:35:59.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:35:59 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:36:01.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:00 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:resolve 2024-04-25T16:36:01.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:00 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:36:01.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:00 smithi098 ceph-mon[29162]: pgmap v465: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T16:36:01.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:00 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:resolve 2024-04-25T16:36:01.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:00 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:36:01.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:00 smithi173 ceph-mon[32556]: pgmap v465: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T16:36:01.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:00 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:resolve 2024-04-25T16:36:01.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:00 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:36:01.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:00 smithi203 ceph-mon[33115]: pgmap v465: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 511 B/s wr, 12 op/s 2024-04-25T16:36:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:01 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999979 2024-04-25T16:36:02.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:01 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:reconnect 2024-04-25T16:36:02.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:01 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:36:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:01 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999979 2024-04-25T16:36:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:01 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:reconnect 2024-04-25T16:36:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:01 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:36:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:01 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.000999979 2024-04-25T16:36:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:01 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:reconnect 2024-04-25T16:36:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:01 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:36:02.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:02.838+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:02.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:02.838+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:36:02.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:02.838+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:36:02.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:02.838+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:36:02.837 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:02.838+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:36:03.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:02 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:rejoin 2024-04-25T16:36:03.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:02 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:36:03.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:02 smithi098 ceph-mon[29162]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:36:03.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:02 smithi098 ceph-mon[29162]: pgmap v466: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 614 B/s wr, 10 op/s 2024-04-25T16:36:03.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:02 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:rejoin 2024-04-25T16:36:03.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:02 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:36:03.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:02 smithi173 ceph-mon[32556]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:36:03.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:02 smithi173 ceph-mon[32556]: pgmap v466: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 614 B/s wr, 10 op/s 2024-04-25T16:36:03.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:02 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:rejoin 2024-04-25T16:36:03.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:02 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:36:03.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:02 smithi203 ceph-mon[33115]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:36:03.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:02 smithi203 ceph-mon[33115]: pgmap v466: 129 pgs: 129 active+clean; 229 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 614 B/s wr, 10 op/s 2024-04-25T16:36:03.513 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:03.095+0000 7f5391d22700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T16:36:03.513 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:03.095+0000 7f5391d22700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:36:03.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:03.095+0000 7f5391d22700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:36:03.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:03.095+0000 7f5391d22700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T16:36:03.514 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:03.095+0000 7f5391d22700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:36:04.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:04.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:36:04.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:active 2024-04-25T16:36:04.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:36:04.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:03 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:03 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:03 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:36:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:03 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:active 2024-04-25T16:36:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:03 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:36:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:03 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:03 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:03 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:36:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:03 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3664329633,v1:172.21.15.173:6839/3664329633] up:active 2024-04-25T16:36:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:03 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:36:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:03 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:05.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:04 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:05.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:04 smithi203 ceph-mon[33115]: pgmap v467: 129 pgs: 129 active+clean; 230 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 175 KiB/s wr, 46 op/s 2024-04-25T16:36:05.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:04 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:05.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:04 smithi098 ceph-mon[29162]: pgmap v467: 129 pgs: 129 active+clean; 230 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 175 KiB/s wr, 46 op/s 2024-04-25T16:36:05.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:04 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:05.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:04 smithi173 ceph-mon[32556]: pgmap v467: 129 pgs: 129 active+clean; 230 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 175 KiB/s wr, 46 op/s 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5394527700 time 2024-04-25T16:36:05.608516+0000 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f539ed5bba6] 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f539ed5bdd1] 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56185d15826b] 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56185d0f7428] 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56185d0f88a9] 2024-04-25T16:36:05.970 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56185d0fa9c3] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56185cfc685d] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56185cfc8e8e] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56185cfc9894] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56185cfb405b] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f539eed8ac6] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 12: (DispatchQueue::entry()+0xb27) [0x7f539eed60b9] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f539efc0ccb] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 14: (Thread::entry_wrapper()+0x43) [0x7f539ed2a72d] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 15: (Thread::_entry_func(void*)+0xd) [0x7f539ed2a749] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f539d8ce1ca] 2024-04-25T16:36:05.971 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 17: clone() 2024-04-25T16:36:05.972 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.609+0000 7f5394527700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5394527700 time 2024-04-25T16:36:05.608516+0000 2024-04-25T16:36:05.972 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:05.972 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.972 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:05.972 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f539ed5bba6] 2024-04-25T16:36:05.972 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f539ed5bdd1] 2024-04-25T16:36:05.972 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56185d15826b] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56185d0f7428] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56185d0f88a9] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56185d0fa9c3] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56185cfc685d] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56185cfc8e8e] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56185cfc9894] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56185cfb405b] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f539eed8ac6] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 12: (DispatchQueue::entry()+0xb27) [0x7f539eed60b9] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f539efc0ccb] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 14: (Thread::entry_wrapper()+0x43) [0x7f539ed2a72d] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 15: (Thread::_entry_func(void*)+0xd) [0x7f539ed2a749] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f539d8ce1ca] 2024-04-25T16:36:05.973 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 17: clone() 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: *** Caught signal (Aborted) ** 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: in thread 7f5394527700 thread_name:ms_dispatch 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f539d8d8d20] 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2: gsignal() 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 3: abort() 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f539ed5bced] 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f539ed5bdd1] 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56185d15826b] 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56185d0f7428] 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56185d0f88a9] 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56185d0fa9c3] 2024-04-25T16:36:05.974 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56185cfc685d] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56185cfc8e8e] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56185cfc9894] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56185cfb405b] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f539eed8ac6] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 15: (DispatchQueue::entry()+0xb27) [0x7f539eed60b9] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f539efc0ccb] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 17: (Thread::entry_wrapper()+0x43) [0x7f539ed2a72d] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 18: (Thread::_entry_func(void*)+0xd) [0x7f539ed2a749] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f539d8ce1ca] 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 20: clone() 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.612+0000 7f5394527700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: in thread 7f5394527700 thread_name:ms_dispatch 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.975 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f539d8d8d20] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2: gsignal() 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 3: abort() 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f539ed5bced] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f539ed5bdd1] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56185d15826b] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56185d0f7428] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56185d0f88a9] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56185d0fa9c3] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56185cfc685d] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56185cfc8e8e] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56185cfc9894] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56185cfb405b] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f539eed8ac6] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 15: (DispatchQueue::entry()+0xb27) [0x7f539eed60b9] 2024-04-25T16:36:05.976 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f539efc0ccb] 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 17: (Thread::entry_wrapper()+0x43) [0x7f539ed2a72d] 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 18: (Thread::_entry_func(void*)+0xd) [0x7f539ed2a749] 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f539d8ce1ca] 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 20: clone() 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: -1> 2024-04-25T16:36:05.609+0000 7f5394527700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5394527700 time 2024-04-25T16:36:05.608516+0000 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:05.977 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f539ed5bba6] 2024-04-25T16:36:05.978 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f539ed5bdd1] 2024-04-25T16:36:05.978 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56185d15826b] 2024-04-25T16:36:05.978 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56185d0f7428] 2024-04-25T16:36:05.978 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56185d0f88a9] 2024-04-25T16:36:05.978 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56185d0fa9c3] 2024-04-25T16:36:05.978 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56185cfc685d] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56185cfc8e8e] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56185cfc9894] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56185cfb405b] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f539eed8ac6] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 12: (DispatchQueue::entry()+0xb27) [0x7f539eed60b9] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f539efc0ccb] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 14: (Thread::entry_wrapper()+0x43) [0x7f539ed2a72d] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 15: (Thread::_entry_func(void*)+0xd) [0x7f539ed2a749] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f539d8ce1ca] 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 17: clone() 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 0> 2024-04-25T16:36:05.612+0000 7f5394527700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:05.979 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: in thread 7f5394527700 thread_name:ms_dispatch 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f539d8d8d20] 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2: gsignal() 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 3: abort() 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f539ed5bced] 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f539ed5bdd1] 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56185d15826b] 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56185d0f7428] 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56185d0f88a9] 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56185d0fa9c3] 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56185cfc685d] 2024-04-25T16:36:05.980 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56185cfc8e8e] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56185cfc9894] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56185cfb405b] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f539eed8ac6] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 15: (DispatchQueue::entry()+0xb27) [0x7f539eed60b9] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f539efc0ccb] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 17: (Thread::entry_wrapper()+0x43) [0x7f539ed2a72d] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 18: (Thread::_entry_func(void*)+0xd) [0x7f539ed2a749] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f539d8ce1ca] 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 20: clone() 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: -1> 2024-04-25T16:36:05.609+0000 7f5394527700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f5394527700 time 2024-04-25T16:36:05.608516+0000 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.981 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f539ed5bba6] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f539ed5bdd1] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56185d15826b] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56185d0f7428] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56185d0f88a9] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56185d0fa9c3] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56185cfc685d] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56185cfc8e8e] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56185cfc9894] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56185cfb405b] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f539eed8ac6] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 12: (DispatchQueue::entry()+0xb27) [0x7f539eed60b9] 2024-04-25T16:36:05.982 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f539efc0ccb] 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 14: (Thread::entry_wrapper()+0x43) [0x7f539ed2a72d] 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 15: (Thread::_entry_func(void*)+0xd) [0x7f539ed2a749] 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f539d8ce1ca] 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 17: clone() 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 0> 2024-04-25T16:36:05.612+0000 7f5394527700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: in thread 7f5394527700 thread_name:ms_dispatch 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f539d8d8d20] 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2: gsignal() 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 3: abort() 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f539ed5bced] 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f539ed5bdd1] 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56185d15826b] 2024-04-25T16:36:05.983 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56185d0f7428] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56185d0f88a9] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56185d0fa9c3] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56185cfc685d] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56185cfc8e8e] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56185cfc9894] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56185cfb405b] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f539eed8ac6] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 15: (DispatchQueue::entry()+0xb27) [0x7f539eed60b9] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f539efc0ccb] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 17: (Thread::entry_wrapper()+0x43) [0x7f539ed2a72d] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 18: (Thread::_entry_func(void*)+0xd) [0x7f539ed2a749] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f539d8ce1ca] 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 20: clone() 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:05.984 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65269]: 2024-04-25T16:36:05.985 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:05 smithi173 podman[65452]: 2024-04-25 16:36:05.685795997 +0000 UTC m=+0.024114666 container died 6353aaf641f6a53e57cbed60ef1a4841c5a06e78fb04a9367b42d0c7a5714000 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:36:06.705 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:06 smithi173 podman[65452]: 2024-04-25 16:36:06.440952703 +0000 UTC m=+0.779271369 container remove 6353aaf641f6a53e57cbed60ef1a4841c5a06e78fb04a9367b42d0c7a5714000 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T16:36:06.705 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:06 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:36:07.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:06 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:07.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:06 smithi173 ceph-mon[32556]: pgmap v468: 129 pgs: 129 active+clean; 231 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 265 KiB/s wr, 62 op/s 2024-04-25T16:36:07.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:06 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-25T16:36:07.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:06 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 5.388s CPU time. 2024-04-25T16:36:07.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:06 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:07.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:06 smithi203 ceph-mon[33115]: pgmap v468: 129 pgs: 129 active+clean; 231 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 265 KiB/s wr, 62 op/s 2024-04-25T16:36:07.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:06 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:07.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:06 smithi098 ceph-mon[29162]: pgmap v468: 129 pgs: 129 active+clean; 231 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 265 KiB/s wr, 62 op/s 2024-04-25T16:36:08.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:08 smithi203 ceph-mon[33115]: pgmap v469: 129 pgs: 129 active+clean; 240 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-25T16:36:08.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:08 smithi098 ceph-mon[29162]: pgmap v469: 129 pgs: 129 active+clean; 240 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-25T16:36:08.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:08 smithi173 ceph-mon[32556]: pgmap v469: 129 pgs: 129 active+clean; 240 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-04-25T16:36:10.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:10 smithi203 ceph-mon[33115]: pgmap v470: 129 pgs: 129 active+clean; 244 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.4 MiB/s wr, 182 op/s 2024-04-25T16:36:10.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:10 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:10.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:10 smithi098 ceph-mon[29162]: pgmap v470: 129 pgs: 129 active+clean; 244 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.4 MiB/s wr, 182 op/s 2024-04-25T16:36:10.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:10 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:10.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:10 smithi173 ceph-mon[32556]: pgmap v470: 129 pgs: 129 active+clean; 244 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.4 MiB/s wr, 182 op/s 2024-04-25T16:36:10.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:10 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:12.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:12 smithi203 ceph-mon[33115]: pgmap v471: 129 pgs: 129 active+clean; 245 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-04-25T16:36:12.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:12 smithi098 ceph-mon[29162]: pgmap v471: 129 pgs: 129 active+clean; 245 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-04-25T16:36:12.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:12 smithi173 ceph-mon[32556]: pgmap v471: 129 pgs: 129 active+clean; 245 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 1.5 MiB/s wr, 184 op/s 2024-04-25T16:36:14.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:14 smithi203 ceph-mon[33115]: pgmap v472: 129 pgs: 129 active+clean; 249 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-04-25T16:36:14.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:14 smithi098 ceph-mon[29162]: pgmap v472: 129 pgs: 129 active+clean; 249 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-04-25T16:36:14.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:14 smithi173 ceph-mon[32556]: pgmap v472: 129 pgs: 129 active+clean; 249 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-04-25T16:36:16.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:16 smithi203 ceph-mon[33115]: pgmap v473: 129 pgs: 129 active+clean; 249 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-04-25T16:36:16.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:16 smithi098 ceph-mon[29162]: pgmap v473: 129 pgs: 129 active+clean; 249 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-04-25T16:36:16.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:16 smithi173 ceph-mon[32556]: pgmap v473: 129 pgs: 129 active+clean; 249 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 185 op/s 2024-04-25T16:36:17.041 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:16 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:36:17.041 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:16 smithi173 systemd[1]: Stopped Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:17.041 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:16 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 5.388s CPU time. 2024-04-25T16:36:17.041 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:16 smithi173 systemd[1]: Starting Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:36:17.041 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:17 smithi173 podman[65546]: 2024-04-25 16:36:16.903337553 +0000 UTC m=+0.018308129 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:36:17.327 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:17 smithi173 podman[65546]: 2024-04-25 16:36:17.043294993 +0000 UTC m=+0.158265545 container create 504bd8987956ddfe61f3896b33e12a51218a564c8338e2aa0ed1c4547fe1b044 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:36:17.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:17 smithi203 ceph-mon[33115]: daemon mds.b restarted 2024-04-25T16:36:17.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:17 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:17.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:17 smithi203 ceph-mon[33115]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T16:36:17.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:17 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:boot 2024-04-25T16:36:17.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:17 smithi203 ceph-mon[33115]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-25T16:36:17.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:17 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:36:17.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:17 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:36:17.688 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:17 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:36:17.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:17 smithi098 ceph-mon[29162]: daemon mds.b restarted 2024-04-25T16:36:17.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:17 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:17.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:17 smithi098 ceph-mon[29162]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T16:36:17.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:17 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:boot 2024-04-25T16:36:17.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:17 smithi098 ceph-mon[29162]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-25T16:36:17.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:17 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:36:17.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:17 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:36:17.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:17 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:36:17.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mon[32556]: daemon mds.b restarted 2024-04-25T16:36:17.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:36:17.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mon[32556]: osdmap e90: 12 total, 12 up, 12 in 2024-04-25T16:36:17.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:boot 2024-04-25T16:36:17.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mon[32556]: Standby daemon mds.f assigned to filesystem cephfs as rank 0 2024-04-25T16:36:17.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {1=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:36:17.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:36:17.764 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:36:17.764 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:17 smithi173 podman[65546]: 2024-04-25 16:36:17.329763561 +0000 UTC m=+0.444734122 container init 504bd8987956ddfe61f3896b33e12a51218a564c8338e2aa0ed1c4547fe1b044 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:36:17.764 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:17 smithi173 podman[65546]: 2024-04-25 16:36:17.332469299 +0000 UTC m=+0.447439859 container start 504bd8987956ddfe61f3896b33e12a51218a564c8338e2aa0ed1c4547fe1b044 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:36:17.764 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-mds[65566]: -- 172.21.15.173:0/3476186863 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a3912bda40 con 0x55a391305a80 2024-04-25T16:36:17.764 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: starting mds.b at 2024-04-25T16:36:17.764 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:17 smithi173 bash[65546]: 504bd8987956ddfe61f3896b33e12a51218a564c8338e2aa0ed1c4547fe1b044 2024-04-25T16:36:17.764 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:36:17 smithi173 systemd[1]: Started Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:36:17.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:17.844+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:17.844+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:36:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:17.844+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:36:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:17.844+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:36:17.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:17.844+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:36:18.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:18 smithi203 ceph-mon[33115]: pgmap v475: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T16:36:18.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:18 smithi098 ceph-mon[29162]: pgmap v475: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T16:36:18.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:18 smithi173 ceph-mon[32556]: pgmap v475: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-04-25T16:36:19.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:19 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:resolve 2024-04-25T16:36:19.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:19 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:36:19.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:19 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:resolve 2024-04-25T16:36:19.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:19 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:36:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:19 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:resolve 2024-04-25T16:36:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:19 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:36:20.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:20 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00199995 2024-04-25T16:36:20.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:20 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:reconnect 2024-04-25T16:36:20.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:20 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:36:20.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:20 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00199995 2024-04-25T16:36:20.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:20 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:reconnect 2024-04-25T16:36:20.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:20 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:36:20.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:20 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00199995 2024-04-25T16:36:20.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:20 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:reconnect 2024-04-25T16:36:20.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:20 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:36:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:21 smithi098 ceph-mon[29162]: pgmap v476: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 827 KiB/s wr, 66 op/s 2024-04-25T16:36:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:21 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:rejoin 2024-04-25T16:36:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:21 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:36:21.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:21 smithi098 ceph-mon[29162]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:36:21.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:21 smithi173 ceph-mon[32556]: pgmap v476: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 827 KiB/s wr, 66 op/s 2024-04-25T16:36:21.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:21 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:rejoin 2024-04-25T16:36:21.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:21 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:36:21.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:21 smithi173 ceph-mon[32556]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:36:21.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:21 smithi203 ceph-mon[33115]: pgmap v476: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 827 KiB/s wr, 66 op/s 2024-04-25T16:36:21.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:21 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:rejoin 2024-04-25T16:36:21.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:21 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:36:21.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:21 smithi203 ceph-mon[33115]: daemon mds.f is now active in filesystem cephfs as rank 0 2024-04-25T16:36:22.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:22 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:22.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:22 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:36:22.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:22 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:active 2024-04-25T16:36:22.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:22 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:36:22.752 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:22 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:22.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:22 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:22.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:22 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:36:22.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:22 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:active 2024-04-25T16:36:22.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:22 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:36:22.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:22 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:22.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:22 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:36:22.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:22 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:36:22.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:22 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/1151278804,v1:172.21.15.203:6835/1151278804] up:active 2024-04-25T16:36:22.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:22 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:36:22.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:22 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:23.733 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:23 smithi203 ceph-mon[33115]: pgmap v477: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 702 KiB/s wr, 54 op/s 2024-04-25T16:36:23.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:23 smithi098 ceph-mon[29162]: pgmap v477: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 702 KiB/s wr, 54 op/s 2024-04-25T16:36:23.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:23 smithi173 ceph-mon[32556]: pgmap v477: 129 pgs: 129 active+clean; 251 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 702 KiB/s wr, 54 op/s 2024-04-25T16:36:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:36:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:36:23.734+0000 7fe1994dc700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T16:36:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:36:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:36:23.734+0000 7fe1994dc700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:36:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:36:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:36:23.734+0000 7fe1994dc700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:36:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:36:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:36:23.734+0000 7fe1994dc700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T16:36:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:36:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:36:23.734+0000 7fe1994dc700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:36:25.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:25 smithi098 ceph-mon[29162]: pgmap v478: 129 pgs: 129 active+clean; 252 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 373 KiB/s wr, 49 op/s 2024-04-25T16:36:25.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:25 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:25 smithi173 ceph-mon[32556]: pgmap v478: 129 pgs: 129 active+clean; 252 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 373 KiB/s wr, 49 op/s 2024-04-25T16:36:25.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:25 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:25.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:25 smithi203 ceph-mon[33115]: pgmap v478: 129 pgs: 129 active+clean; 252 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 373 KiB/s wr, 49 op/s 2024-04-25T16:36:25.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:25 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:27.737 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:27 smithi203 ceph-mon[33115]: pgmap v479: 129 pgs: 129 active+clean; 254 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 536 KiB/s wr, 61 op/s 2024-04-25T16:36:27.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:27 smithi098 ceph-mon[29162]: pgmap v479: 129 pgs: 129 active+clean; 254 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 536 KiB/s wr, 61 op/s 2024-04-25T16:36:27.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:27 smithi173 ceph-mon[32556]: pgmap v479: 129 pgs: 129 active+clean; 254 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 536 KiB/s wr, 61 op/s 2024-04-25T16:36:29.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:29 smithi098 ceph-mon[29162]: pgmap v480: 129 pgs: 129 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-25T16:36:29.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:29 smithi173 ceph-mon[32556]: pgmap v480: 129 pgs: 129 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-25T16:36:29.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:29 smithi203 ceph-mon[33115]: pgmap v480: 129 pgs: 129 active+clean; 260 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 1.1 MiB/s wr, 135 op/s 2024-04-25T16:36:30.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:30 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:30.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:30 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:30.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:30 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:36:31.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:31 smithi098 ceph-mon[29162]: pgmap v481: 129 pgs: 129 active+clean; 263 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-25T16:36:31.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:31 smithi173 ceph-mon[32556]: pgmap v481: 129 pgs: 129 active+clean; 263 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-25T16:36:31.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:31 smithi203 ceph-mon[33115]: pgmap v481: 129 pgs: 129 active+clean; 263 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.1 MiB/s wr, 175 op/s 2024-04-25T16:36:32.857 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:32.858+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:32.858+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:36:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:32.858+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:36:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:32.858+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:36:32.858 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:32.858+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:36:33.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:33 smithi098 ceph-mon[29162]: pgmap v482: 129 pgs: 129 active+clean; 264 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 188 op/s 2024-04-25T16:36:33.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:33 smithi173 ceph-mon[32556]: pgmap v482: 129 pgs: 129 active+clean; 264 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 188 op/s 2024-04-25T16:36:33.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:33 smithi203 ceph-mon[33115]: pgmap v482: 129 pgs: 129 active+clean; 264 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 1.2 MiB/s wr, 188 op/s 2024-04-25T16:36:35.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:35 smithi098 ceph-mon[29162]: pgmap v483: 129 pgs: 129 active+clean; 272 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-25T16:36:35.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:35 smithi173 ceph-mon[32556]: pgmap v483: 129 pgs: 129 active+clean; 272 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-25T16:36:35.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:35 smithi203 ceph-mon[33115]: pgmap v483: 129 pgs: 129 active+clean; 272 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 1.9 MiB/s wr, 242 op/s 2024-04-25T16:36:37.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:37 smithi098 ceph-mon[29162]: pgmap v484: 129 pgs: 129 active+clean; 274 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 232 op/s 2024-04-25T16:36:37.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:37 smithi173 ceph-mon[32556]: pgmap v484: 129 pgs: 129 active+clean; 274 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 232 op/s 2024-04-25T16:36:37.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:37 smithi203 ceph-mon[33115]: pgmap v484: 129 pgs: 129 active+clean; 274 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.0 MiB/s wr, 232 op/s 2024-04-25T16:36:39.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:39 smithi098 ceph-mon[29162]: pgmap v485: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-25T16:36:39.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:39 smithi173 ceph-mon[32556]: pgmap v485: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-25T16:36:39.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:39 smithi203 ceph-mon[33115]: pgmap v485: 129 pgs: 129 active+clean; 284 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.7 MiB/s wr, 294 op/s 2024-04-25T16:36:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:41 smithi098 ceph-mon[29162]: pgmap v486: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 2.6 MiB/s wr, 269 op/s 2024-04-25T16:36:41.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:41 smithi173 ceph-mon[32556]: pgmap v486: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 2.6 MiB/s wr, 269 op/s 2024-04-25T16:36:41.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:41 smithi203 ceph-mon[33115]: pgmap v486: 129 pgs: 129 active+clean; 288 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 2.6 MiB/s wr, 269 op/s 2024-04-25T16:36:43.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:43 smithi173 ceph-mon[32556]: pgmap v487: 129 pgs: 129 active+clean; 289 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-04-25T16:36:43.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:43 smithi203 ceph-mon[33115]: pgmap v487: 129 pgs: 129 active+clean; 289 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-04-25T16:36:44.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:43 smithi098 ceph-mon[29162]: pgmap v487: 129 pgs: 129 active+clean; 289 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 2.4 MiB/s wr, 230 op/s 2024-04-25T16:36:45.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:45 smithi173 ceph-mon[32556]: pgmap v488: 129 pgs: 129 active+clean; 300 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-04-25T16:36:45.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:45 smithi203 ceph-mon[33115]: pgmap v488: 129 pgs: 129 active+clean; 300 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-04-25T16:36:46.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:45 smithi098 ceph-mon[29162]: pgmap v488: 129 pgs: 129 active+clean; 300 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.2 MiB/s wr, 297 op/s 2024-04-25T16:36:47.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:47 smithi173 ceph-mon[32556]: pgmap v489: 129 pgs: 129 active+clean; 302 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-25T16:36:47.872 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:47.874+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:36:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:47.874+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:36:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:47.874+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:36:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:47.874+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:36:47.873 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:36:47.874+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:36:47.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:47 smithi203 ceph-mon[33115]: pgmap v489: 129 pgs: 129 active+clean; 302 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-25T16:36:48.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:47 smithi098 ceph-mon[29162]: pgmap v489: 129 pgs: 129 active+clean; 302 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.7 MiB/s wr, 262 op/s 2024-04-25T16:36:49.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:49 smithi173 ceph-mon[32556]: pgmap v490: 129 pgs: 129 active+clean; 307 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-25T16:36:49.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:49 smithi203 ceph-mon[33115]: pgmap v490: 129 pgs: 129 active+clean; 307 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-25T16:36:50.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:49 smithi098 ceph-mon[29162]: pgmap v490: 129 pgs: 129 active+clean; 307 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 3.0 MiB/s wr, 316 op/s 2024-04-25T16:36:51.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:51 smithi173 ceph-mon[32556]: pgmap v491: 129 pgs: 129 active+clean; 312 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.6 MiB/s wr, 294 op/s 2024-04-25T16:36:51.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:51 smithi203 ceph-mon[33115]: pgmap v491: 129 pgs: 129 active+clean; 312 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.6 MiB/s wr, 294 op/s 2024-04-25T16:36:52.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:51 smithi098 ceph-mon[29162]: pgmap v491: 129 pgs: 129 active+clean; 312 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 2.6 MiB/s wr, 294 op/s 2024-04-25T16:36:52.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:52 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:36:53.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:52 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:36:53.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:52 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:36:53.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:53 smithi203 ceph-mon[33115]: pgmap v492: 129 pgs: 129 active+clean; 313 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.3 MiB/s wr, 265 op/s 2024-04-25T16:36:53.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:53 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:36:53.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:53 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:36:53.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:53 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:36:54.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:53 smithi098 ceph-mon[29162]: pgmap v492: 129 pgs: 129 active+clean; 313 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.3 MiB/s wr, 265 op/s 2024-04-25T16:36:54.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:53 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:36:54.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:53 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:36:54.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:53 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:36:54.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:53 smithi173 ceph-mon[32556]: pgmap v492: 129 pgs: 129 active+clean; 313 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 2.3 MiB/s wr, 265 op/s 2024-04-25T16:36:54.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:53 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:36:54.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:53 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:36:54.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:53 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:36:55.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-mon[33115]: pgmap v493: 129 pgs: 129 active+clean; 324 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-04-25T16:36:55.937 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f236a102700 time 2024-04-25T16:36:55.614718+0000 2024-04-25T16:36:55.937 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:55.937 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2374936ba6] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2374936dd1] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56157403c26b] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561573fdb428] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561573fdc8a9] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561573fde9c3] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561573eaa85d] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561573eace8e] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561573ead894] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561573e9805b] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f2374ab3ac6] 2024-04-25T16:36:55.938 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 12: (DispatchQueue::entry()+0xb27) [0x7f2374ab10b9] 2024-04-25T16:36:55.939 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2374b9bccb] 2024-04-25T16:36:55.939 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 14: (Thread::entry_wrapper()+0x43) [0x7f237490572d] 2024-04-25T16:36:55.939 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 15: (Thread::_entry_func(void*)+0xd) [0x7f2374905749] 2024-04-25T16:36:55.939 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f23734a91ca] 2024-04-25T16:36:55.939 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 17: clone() 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.616+0000 7f236a102700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f236a102700 time 2024-04-25T16:36:55.614718+0000 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2374936ba6] 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2374936dd1] 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56157403c26b] 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561573fdb428] 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561573fdc8a9] 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561573fde9c3] 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561573eaa85d] 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561573eace8e] 2024-04-25T16:36:55.940 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561573ead894] 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561573e9805b] 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f2374ab3ac6] 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 12: (DispatchQueue::entry()+0xb27) [0x7f2374ab10b9] 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2374b9bccb] 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 14: (Thread::entry_wrapper()+0x43) [0x7f237490572d] 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 15: (Thread::_entry_func(void*)+0xd) [0x7f2374905749] 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f23734a91ca] 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 17: clone() 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: *** Caught signal (Aborted) ** 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: in thread 7f236a102700 thread_name:ms_dispatch 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:55.941 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f23734b3d20] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2: gsignal() 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 3: abort() 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2374936ced] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2374936dd1] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56157403c26b] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561573fdb428] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561573fdc8a9] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561573fde9c3] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561573eaa85d] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561573eace8e] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561573ead894] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561573e9805b] 2024-04-25T16:36:55.942 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f2374ab3ac6] 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 15: (DispatchQueue::entry()+0xb27) [0x7f2374ab10b9] 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2374b9bccb] 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 17: (Thread::entry_wrapper()+0x43) [0x7f237490572d] 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 18: (Thread::_entry_func(void*)+0xd) [0x7f2374905749] 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f23734a91ca] 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 20: clone() 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.619+0000 7f236a102700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: in thread 7f236a102700 thread_name:ms_dispatch 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f23734b3d20] 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2: gsignal() 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 3: abort() 2024-04-25T16:36:55.943 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2374936ced] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2374936dd1] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56157403c26b] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561573fdb428] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561573fdc8a9] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561573fde9c3] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561573eaa85d] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561573eace8e] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561573ead894] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561573e9805b] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f2374ab3ac6] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 15: (DispatchQueue::entry()+0xb27) [0x7f2374ab10b9] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2374b9bccb] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 17: (Thread::entry_wrapper()+0x43) [0x7f237490572d] 2024-04-25T16:36:55.944 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 18: (Thread::_entry_func(void*)+0xd) [0x7f2374905749] 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f23734a91ca] 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 20: clone() 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: -1> 2024-04-25T16:36:55.616+0000 7f236a102700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f236a102700 time 2024-04-25T16:36:55.614718+0000 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:55.945 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2374936ba6] 2024-04-25T16:36:55.946 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2374936dd1] 2024-04-25T16:36:55.946 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56157403c26b] 2024-04-25T16:36:55.946 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561573fdb428] 2024-04-25T16:36:55.946 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561573fdc8a9] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561573fde9c3] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561573eaa85d] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561573eace8e] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561573ead894] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561573e9805b] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f2374ab3ac6] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 12: (DispatchQueue::entry()+0xb27) [0x7f2374ab10b9] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2374b9bccb] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 14: (Thread::entry_wrapper()+0x43) [0x7f237490572d] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 15: (Thread::_entry_func(void*)+0xd) [0x7f2374905749] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f23734a91ca] 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 17: clone() 2024-04-25T16:36:55.947 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 0> 2024-04-25T16:36:55.619+0000 7f236a102700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: in thread 7f236a102700 thread_name:ms_dispatch 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f23734b3d20] 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2: gsignal() 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 3: abort() 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2374936ced] 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2374936dd1] 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56157403c26b] 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561573fdb428] 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561573fdc8a9] 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561573fde9c3] 2024-04-25T16:36:55.948 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561573eaa85d] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561573eace8e] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561573ead894] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561573e9805b] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f2374ab3ac6] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 15: (DispatchQueue::entry()+0xb27) [0x7f2374ab10b9] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2374b9bccb] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 17: (Thread::entry_wrapper()+0x43) [0x7f237490572d] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 18: (Thread::_entry_func(void*)+0xd) [0x7f2374905749] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f23734a91ca] 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 20: clone() 2024-04-25T16:36:55.949 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: -1> 2024-04-25T16:36:55.616+0000 7f236a102700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f236a102700 time 2024-04-25T16:36:55.614718+0000 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f2374936ba6] 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2374936dd1] 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56157403c26b] 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561573fdb428] 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561573fdc8a9] 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561573fde9c3] 2024-04-25T16:36:55.950 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561573eaa85d] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561573eace8e] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561573ead894] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561573e9805b] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f2374ab3ac6] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 12: (DispatchQueue::entry()+0xb27) [0x7f2374ab10b9] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2374b9bccb] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 14: (Thread::entry_wrapper()+0x43) [0x7f237490572d] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 15: (Thread::_entry_func(void*)+0xd) [0x7f2374905749] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f23734a91ca] 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 17: clone() 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 0> 2024-04-25T16:36:55.619+0000 7f236a102700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: in thread 7f236a102700 thread_name:ms_dispatch 2024-04-25T16:36:55.951 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f23734b3d20] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2: gsignal() 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 3: abort() 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f2374936ced] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2374936dd1] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56157403c26b] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561573fdb428] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561573fdc8a9] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561573fde9c3] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561573eaa85d] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561573eace8e] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561573ead894] 2024-04-25T16:36:55.952 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561573e9805b] 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f2374ab3ac6] 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 15: (DispatchQueue::entry()+0xb27) [0x7f2374ab10b9] 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2374b9bccb] 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 17: (Thread::entry_wrapper()+0x43) [0x7f237490572d] 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 18: (Thread::_entry_func(void*)+0xd) [0x7f2374905749] 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f23734a91ca] 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 20: clone() 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[64179]: 2024-04-25T16:36:55.953 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:55 smithi203 podman[67442]: 2024-04-25 16:36:55.694681754 +0000 UTC m=+0.025027977 container died 684bc2cb2319ff9f23ef8c4c53c3ac3f474ca3a37e8eba2a8b2387a10e99c7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-04-25T16:36:56.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:55 smithi098 ceph-mon[29162]: pgmap v493: 129 pgs: 129 active+clean; 324 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-04-25T16:36:56.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:55 smithi173 ceph-mon[32556]: pgmap v493: 129 pgs: 129 active+clean; 324 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.1 MiB/s wr, 321 op/s 2024-04-25T16:36:56.652 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:56 smithi203 podman[67442]: 2024-04-25 16:36:56.354453131 +0000 UTC m=+0.684799340 container remove 684bc2cb2319ff9f23ef8c4c53c3ac3f474ca3a37e8eba2a8b2387a10e99c7e9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:36:56.652 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:56 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:36:56.937 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:56 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-25T16:36:56.937 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:36:56 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Consumed 3.406s CPU time. 2024-04-25T16:36:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:57 smithi173 ceph-mon[32556]: pgmap v494: 129 pgs: 129 active+clean; 325 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.3 MiB/s wr, 262 op/s 2024-04-25T16:36:57.837 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:57 smithi203 ceph-mon[33115]: pgmap v494: 129 pgs: 129 active+clean; 325 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.3 MiB/s wr, 262 op/s 2024-04-25T16:36:58.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:57 smithi098 ceph-mon[29162]: pgmap v494: 129 pgs: 129 active+clean; 325 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 2.3 MiB/s wr, 262 op/s 2024-04-25T16:36:59.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:36:59 smithi203 ceph-mon[33115]: pgmap v495: 129 pgs: 129 active+clean; 329 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 2.5 MiB/s wr, 304 op/s 2024-04-25T16:37:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:36:59 smithi098 ceph-mon[29162]: pgmap v495: 129 pgs: 129 active+clean; 329 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 2.5 MiB/s wr, 304 op/s 2024-04-25T16:37:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:36:59 smithi173 ceph-mon[32556]: pgmap v495: 129 pgs: 129 active+clean; 329 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 2.5 MiB/s wr, 304 op/s 2024-04-25T16:37:01.802 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:01 smithi098 ceph-mon[29162]: pgmap v496: 129 pgs: 129 active+clean; 337 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 286 op/s 2024-04-25T16:37:01.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:01 smithi203 ceph-mon[33115]: pgmap v496: 129 pgs: 129 active+clean; 337 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 286 op/s 2024-04-25T16:37:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:01 smithi173 ceph-mon[32556]: pgmap v496: 129 pgs: 129 active+clean; 337 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 286 op/s 2024-04-25T16:37:02.878 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:02.879+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:02.880+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:37:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:02.880+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:37:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:02.880+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:37:02.879 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:02.880+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:37:03.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:03 smithi203 ceph-mon[33115]: pgmap v497: 129 pgs: 129 active+clean; 339 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.4 MiB/s wr, 255 op/s 2024-04-25T16:37:04.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:03 smithi098 ceph-mon[29162]: pgmap v497: 129 pgs: 129 active+clean; 339 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.4 MiB/s wr, 255 op/s 2024-04-25T16:37:04.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:03 smithi173 ceph-mon[32556]: pgmap v497: 129 pgs: 129 active+clean; 339 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 2.4 MiB/s wr, 255 op/s 2024-04-25T16:37:05.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:05 smithi203 ceph-mon[33115]: pgmap v498: 129 pgs: 129 active+clean; 347 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 301 op/s 2024-04-25T16:37:06.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:05 smithi098 ceph-mon[29162]: pgmap v498: 129 pgs: 129 active+clean; 347 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 301 op/s 2024-04-25T16:37:06.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:05 smithi173 ceph-mon[32556]: pgmap v498: 129 pgs: 129 active+clean; 347 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 301 op/s 2024-04-25T16:37:07.187 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:06 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:37:07.187 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:06 smithi203 systemd[1]: Stopped Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:37:07.187 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:06 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Consumed 3.406s CPU time. 2024-04-25T16:37:07.187 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:06 smithi203 systemd[1]: Starting Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:37:07.187 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:06 smithi203 podman[67533]: 2024-04-25 16:37:06.822196001 +0000 UTC m=+0.018053227 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:37:07.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:07 smithi203 ceph-mon[33115]: pgmap v499: 129 pgs: 129 active+clean; 348 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 2.2 MiB/s wr, 239 op/s 2024-04-25T16:37:08.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:07 smithi098 ceph-mon[29162]: pgmap v499: 129 pgs: 129 active+clean; 348 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 2.2 MiB/s wr, 239 op/s 2024-04-25T16:37:08.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:07 smithi173 ceph-mon[32556]: pgmap v499: 129 pgs: 129 active+clean; 348 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 2.2 MiB/s wr, 239 op/s 2024-04-25T16:37:08.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:07 smithi203 podman[67533]: 2024-04-25 16:37:07.980727605 +0000 UTC m=+1.176584802 container create 4c1bebd6b08f46c71d5af873dea04bf682c08f71b85b779674c31b328a0cb994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T16:37:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:09 smithi203 ceph-mon[33115]: pgmap v500: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-04-25T16:37:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:09 smithi203 ceph-mon[33115]: Replacing daemon mds.c as rank 4 with standby daemon mds.g 2024-04-25T16:37:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:09 smithi203 ceph-mon[33115]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-25T16:37:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:09 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:09 smithi203 ceph-mon[33115]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T16:37:09.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:09 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:37:10.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:09 smithi098 ceph-mon[29162]: pgmap v500: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-04-25T16:37:10.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:09 smithi098 ceph-mon[29162]: Replacing daemon mds.c as rank 4 with standby daemon mds.g 2024-04-25T16:37:10.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:09 smithi098 ceph-mon[29162]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-25T16:37:10.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:09 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:10.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:09 smithi098 ceph-mon[29162]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T16:37:10.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:09 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:37:10.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:09 smithi173 ceph-mon[32556]: pgmap v500: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 2.3 MiB/s wr, 220 op/s 2024-04-25T16:37:10.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:09 smithi173 ceph-mon[32556]: Replacing daemon mds.c as rank 4 with standby daemon mds.g 2024-04-25T16:37:10.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:09 smithi173 ceph-mon[32556]: MDS daemon mds.c is removed because it is dead or otherwise unavailable. 2024-04-25T16:37:10.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:09 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:37:10.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:09 smithi173 ceph-mon[32556]: osdmap e91: 12 total, 12 up, 12 in 2024-04-25T16:37:10.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:09 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:37:10.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:10 smithi203 podman[67533]: 2024-04-25 16:37:10.101400873 +0000 UTC m=+3.297258082 container init 4c1bebd6b08f46c71d5af873dea04bf682c08f71b85b779674c31b328a0cb994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T16:37:10.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:10 smithi203 podman[67533]: 2024-04-25 16:37:10.105143786 +0000 UTC m=+3.301000984 container start 4c1bebd6b08f46c71d5af873dea04bf682c08f71b85b779674c31b328a0cb994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True) 2024-04-25T16:37:10.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:10 smithi203 ceph-mds[67552]: -- 172.21.15.203:0/3868735088 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56499d761a40 con 0x56499d7a9600 2024-04-25T16:37:10.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:10 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: starting mds.c at 2024-04-25T16:37:11.687 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:11 smithi203 bash[67533]: 4c1bebd6b08f46c71d5af873dea04bf682c08f71b85b779674c31b328a0cb994 2024-04-25T16:37:11.687 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:37:11 smithi203 systemd[1]: Started Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:37:11.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:11 smithi203 ceph-mon[33115]: pgmap v502: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-25T16:37:11.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:11 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:resolve 2024-04-25T16:37:11.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:11 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:boot 2024-04-25T16:37:11.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:11 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:37:11.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:37:12.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:11 smithi098 ceph-mon[29162]: pgmap v502: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-25T16:37:12.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:11 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:resolve 2024-04-25T16:37:12.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:11 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:boot 2024-04-25T16:37:12.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:11 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:37:12.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:37:12.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:11 smithi173 ceph-mon[32556]: pgmap v502: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-04-25T16:37:12.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:11 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:resolve 2024-04-25T16:37:12.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:11 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:boot 2024-04-25T16:37:12.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:11 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:37:12.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:37:12.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:12 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:reconnect 2024-04-25T16:37:12.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:12 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:37:12.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:12 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00999986 2024-04-25T16:37:13.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:12 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:reconnect 2024-04-25T16:37:13.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:12 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:37:13.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:12 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00999986 2024-04-25T16:37:13.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:12 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:reconnect 2024-04-25T16:37:13.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:12 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:37:13.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:12 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00999986 2024-04-25T16:37:13.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:13 smithi203 ceph-mon[33115]: pgmap v503: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-04-25T16:37:13.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:13 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:rejoin 2024-04-25T16:37:13.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:13 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:37:13.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:13 smithi203 ceph-mon[33115]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:37:13.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:37:13 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:37:13.737+0000 7fe1994dc700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:37:14.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:13 smithi098 ceph-mon[29162]: pgmap v503: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-04-25T16:37:14.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:13 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:rejoin 2024-04-25T16:37:14.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:13 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:37:14.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:13 smithi098 ceph-mon[29162]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:37:14.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:13 smithi173 ceph-mon[32556]: pgmap v503: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 1.3 MiB/s wr, 107 op/s 2024-04-25T16:37:14.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:13 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:rejoin 2024-04-25T16:37:14.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:13 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:37:14.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:13 smithi173 ceph-mon[32556]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:37:14.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:14 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:14.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:14 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:37:14.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:14 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:active 2024-04-25T16:37:14.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:14 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:37:14.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:14 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:15.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:14 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:15.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:14 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:37:15.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:14 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:active 2024-04-25T16:37:15.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:14 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:37:15.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:14 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:15.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:14 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:37:15.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:14 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:37:15.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:14 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/3308601909,v1:172.21.15.98:6839/3308601909] up:active 2024-04-25T16:37:15.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:14 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:37:15.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:14 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:15.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:15 smithi203 ceph-mon[33115]: pgmap v504: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 485 KiB/s wr, 37 op/s 2024-04-25T16:37:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:15 smithi098 ceph-mon[29162]: pgmap v504: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 485 KiB/s wr, 37 op/s 2024-04-25T16:37:16.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:15 smithi173 ceph-mon[32556]: pgmap v504: 129 pgs: 129 active+clean; 351 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 485 KiB/s wr, 37 op/s 2024-04-25T16:37:17.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:17.895+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:17.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:17.896+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:37:17.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:17.896+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:37:17.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:17.896+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:37:17.894 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:17.896+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:37:17.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:17 smithi203 ceph-mon[33115]: pgmap v505: 129 pgs: 129 active+clean; 352 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 344 KiB/s wr, 27 op/s 2024-04-25T16:37:18.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:17 smithi098 ceph-mon[29162]: pgmap v505: 129 pgs: 129 active+clean; 352 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 344 KiB/s wr, 27 op/s 2024-04-25T16:37:18.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:17 smithi173 ceph-mon[32556]: pgmap v505: 129 pgs: 129 active+clean; 352 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 344 KiB/s wr, 27 op/s 2024-04-25T16:37:19.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:19 smithi203 ceph-mon[33115]: pgmap v506: 129 pgs: 129 active+clean; 354 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 370 KiB/s wr, 81 op/s 2024-04-25T16:37:20.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:19 smithi098 ceph-mon[29162]: pgmap v506: 129 pgs: 129 active+clean; 354 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 370 KiB/s wr, 81 op/s 2024-04-25T16:37:20.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:19 smithi173 ceph-mon[32556]: pgmap v506: 129 pgs: 129 active+clean; 354 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 370 KiB/s wr, 81 op/s 2024-04-25T16:37:21.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:21 smithi203 ceph-mon[33115]: pgmap v507: 129 pgs: 129 active+clean; 355 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 490 KiB/s wr, 103 op/s 2024-04-25T16:37:22.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:21 smithi098 ceph-mon[29162]: pgmap v507: 129 pgs: 129 active+clean; 355 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 490 KiB/s wr, 103 op/s 2024-04-25T16:37:22.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:21 smithi173 ceph-mon[32556]: pgmap v507: 129 pgs: 129 active+clean; 355 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 490 KiB/s wr, 103 op/s 2024-04-25T16:37:22.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:22 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:23.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:22 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:23.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:22 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:23.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:23 smithi203 ceph-mon[33115]: pgmap v508: 129 pgs: 129 active+clean; 359 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 762 KiB/s wr, 106 op/s 2024-04-25T16:37:24.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:23 smithi098 ceph-mon[29162]: pgmap v508: 129 pgs: 129 active+clean; 359 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 762 KiB/s wr, 106 op/s 2024-04-25T16:37:24.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:23 smithi173 ceph-mon[32556]: pgmap v508: 129 pgs: 129 active+clean; 359 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 762 KiB/s wr, 106 op/s 2024-04-25T16:37:25.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:25 smithi203 ceph-mon[33115]: pgmap v509: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-04-25T16:37:26.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:25 smithi098 ceph-mon[29162]: pgmap v509: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-04-25T16:37:26.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:25 smithi173 ceph-mon[32556]: pgmap v509: 129 pgs: 129 active+clean; 363 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 1.2 MiB/s wr, 181 op/s 2024-04-25T16:37:27.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:27 smithi203 ceph-mon[33115]: pgmap v510: 129 pgs: 129 active+clean; 364 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-04-25T16:37:27.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:27 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:27 smithi098 ceph-mon[29162]: pgmap v510: 129 pgs: 129 active+clean; 364 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-04-25T16:37:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:27 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:28.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:27 smithi173 ceph-mon[32556]: pgmap v510: 129 pgs: 129 active+clean; 364 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 1.2 MiB/s wr, 197 op/s 2024-04-25T16:37:28.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:27 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:29.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:29 smithi203 ceph-mon[33115]: pgmap v511: 129 pgs: 129 active+clean; 374 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 276 op/s 2024-04-25T16:37:30.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:29 smithi098 ceph-mon[29162]: pgmap v511: 129 pgs: 129 active+clean; 374 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 276 op/s 2024-04-25T16:37:30.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:29 smithi173 ceph-mon[32556]: pgmap v511: 129 pgs: 129 active+clean; 374 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 276 op/s 2024-04-25T16:37:31.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:31 smithi203 ceph-mon[33115]: pgmap v512: 129 pgs: 129 active+clean; 379 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.3 MiB/s wr, 296 op/s 2024-04-25T16:37:32.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:31 smithi098 ceph-mon[29162]: pgmap v512: 129 pgs: 129 active+clean; 379 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.3 MiB/s wr, 296 op/s 2024-04-25T16:37:32.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:31 smithi173 ceph-mon[32556]: pgmap v512: 129 pgs: 129 active+clean; 379 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.3 MiB/s wr, 296 op/s 2024-04-25T16:37:32.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:32.911+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:32.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:32.912+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:37:32.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:32.912+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:37:32.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:32.912+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:37:32.910 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:32.912+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:37:33.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:33 smithi203 ceph-mon[33115]: pgmap v513: 129 pgs: 129 active+clean; 380 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.2 MiB/s wr, 281 op/s 2024-04-25T16:37:34.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:33 smithi098 ceph-mon[29162]: pgmap v513: 129 pgs: 129 active+clean; 380 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.2 MiB/s wr, 281 op/s 2024-04-25T16:37:34.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:33 smithi173 ceph-mon[32556]: pgmap v513: 129 pgs: 129 active+clean; 380 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.2 MiB/s wr, 281 op/s 2024-04-25T16:37:35.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:35 smithi203 ceph-mon[33115]: pgmap v514: 129 pgs: 129 active+clean; 387 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 294 op/s 2024-04-25T16:37:36.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:35 smithi098 ceph-mon[29162]: pgmap v514: 129 pgs: 129 active+clean; 387 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 294 op/s 2024-04-25T16:37:36.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:35 smithi173 ceph-mon[32556]: pgmap v514: 129 pgs: 129 active+clean; 387 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 2.5 MiB/s wr, 294 op/s 2024-04-25T16:37:37.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:37 smithi203 ceph-mon[33115]: pgmap v515: 129 pgs: 129 active+clean; 387 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 2.1 MiB/s wr, 220 op/s 2024-04-25T16:37:38.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:37 smithi098 ceph-mon[29162]: pgmap v515: 129 pgs: 129 active+clean; 387 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 2.1 MiB/s wr, 220 op/s 2024-04-25T16:37:38.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:37 smithi173 ceph-mon[32556]: pgmap v515: 129 pgs: 129 active+clean; 387 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 2.1 MiB/s wr, 220 op/s 2024-04-25T16:37:39.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:39 smithi203 ceph-mon[33115]: pgmap v516: 129 pgs: 129 active+clean; 396 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 258 op/s 2024-04-25T16:37:40.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:39 smithi098 ceph-mon[29162]: pgmap v516: 129 pgs: 129 active+clean; 396 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 258 op/s 2024-04-25T16:37:40.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:39 smithi173 ceph-mon[32556]: pgmap v516: 129 pgs: 129 active+clean; 396 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.8 MiB/s wr, 258 op/s 2024-04-25T16:37:41.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:41 smithi203 ceph-mon[33115]: pgmap v517: 129 pgs: 129 active+clean; 408 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 3.0 MiB/s wr, 243 op/s 2024-04-25T16:37:42.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:41 smithi098 ceph-mon[29162]: pgmap v517: 129 pgs: 129 active+clean; 408 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 3.0 MiB/s wr, 243 op/s 2024-04-25T16:37:42.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:41 smithi173 ceph-mon[32556]: pgmap v517: 129 pgs: 129 active+clean; 408 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 3.0 MiB/s wr, 243 op/s 2024-04-25T16:37:43.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:43 smithi203 ceph-mon[33115]: pgmap v518: 129 pgs: 129 active+clean; 413 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 3.0 MiB/s wr, 198 op/s 2024-04-25T16:37:44.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:43 smithi098 ceph-mon[29162]: pgmap v518: 129 pgs: 129 active+clean; 413 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 3.0 MiB/s wr, 198 op/s 2024-04-25T16:37:44.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:43 smithi173 ceph-mon[32556]: pgmap v518: 129 pgs: 129 active+clean; 413 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 3.0 MiB/s wr, 198 op/s 2024-04-25T16:37:45.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:45 smithi203 ceph-mon[33115]: pgmap v519: 129 pgs: 129 active+clean; 421 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 252 op/s 2024-04-25T16:37:46.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:45 smithi098 ceph-mon[29162]: pgmap v519: 129 pgs: 129 active+clean; 421 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 252 op/s 2024-04-25T16:37:46.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:45 smithi173 ceph-mon[32556]: pgmap v519: 129 pgs: 129 active+clean; 421 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.6 MiB/s wr, 252 op/s 2024-04-25T16:37:47.911 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:47.913+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:37:47.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:47.913+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:37:47.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:47.913+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:37:47.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:47.913+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:37:47.912 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:37:47.913+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:37:47.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:47 smithi203 ceph-mon[33115]: pgmap v520: 129 pgs: 129 active+clean; 424 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 3.3 MiB/s wr, 236 op/s 2024-04-25T16:37:48.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:47 smithi098 ceph-mon[29162]: pgmap v520: 129 pgs: 129 active+clean; 424 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 3.3 MiB/s wr, 236 op/s 2024-04-25T16:37:48.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:47 smithi173 ceph-mon[32556]: pgmap v520: 129 pgs: 129 active+clean; 424 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 3.3 MiB/s wr, 236 op/s 2024-04-25T16:37:49.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:49 smithi203 ceph-mon[33115]: pgmap v521: 129 pgs: 129 active+clean; 437 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 325 op/s 2024-04-25T16:37:50.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:49 smithi098 ceph-mon[29162]: pgmap v521: 129 pgs: 129 active+clean; 437 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 325 op/s 2024-04-25T16:37:50.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:49 smithi173 ceph-mon[32556]: pgmap v521: 129 pgs: 129 active+clean; 437 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 325 op/s 2024-04-25T16:37:51.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:51 smithi203 ceph-mon[33115]: pgmap v522: 129 pgs: 129 active+clean; 443 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-04-25T16:37:52.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:51 smithi098 ceph-mon[29162]: pgmap v522: 129 pgs: 129 active+clean; 443 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-04-25T16:37:52.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:51 smithi173 ceph-mon[32556]: pgmap v522: 129 pgs: 129 active+clean; 443 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.1 MiB/s wr, 311 op/s 2024-04-25T16:37:53.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:53 smithi203 ceph-mon[33115]: pgmap v523: 129 pgs: 129 active+clean; 447 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 264 op/s 2024-04-25T16:37:53.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:53 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:37:54.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:53 smithi098 ceph-mon[29162]: pgmap v523: 129 pgs: 129 active+clean; 447 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 264 op/s 2024-04-25T16:37:54.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:53 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:37:54.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:53 smithi173 ceph-mon[32556]: pgmap v523: 129 pgs: 129 active+clean; 447 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.4 MiB/s wr, 264 op/s 2024-04-25T16:37:54.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:53 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:37:54.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:37:54.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:37:54.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:37:55.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:37:55.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:37:55.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:37:55.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:37:55.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:37:55.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:37:56.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:55 smithi098 ceph-mon[29162]: pgmap v524: 129 pgs: 129 active+clean; 463 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 328 op/s 2024-04-25T16:37:56.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:55 smithi173 ceph-mon[32556]: pgmap v524: 129 pgs: 129 active+clean; 463 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 328 op/s 2024-04-25T16:37:56.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:55 smithi203 ceph-mon[33115]: pgmap v524: 129 pgs: 129 active+clean; 463 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.3 MiB/s wr, 328 op/s 2024-04-25T16:37:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:56 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:56 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:57.087 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:56 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:37:58.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:57 smithi098 ceph-mon[29162]: pgmap v525: 129 pgs: 129 active+clean; 466 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 3.9 MiB/s wr, 306 op/s 2024-04-25T16:37:58.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:57 smithi173 ceph-mon[32556]: pgmap v525: 129 pgs: 129 active+clean; 466 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 3.9 MiB/s wr, 306 op/s 2024-04-25T16:37:58.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:57 smithi203 ceph-mon[33115]: pgmap v525: 129 pgs: 129 active+clean; 466 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 3.9 MiB/s wr, 306 op/s 2024-04-25T16:38:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:37:59 smithi098 ceph-mon[29162]: pgmap v526: 129 pgs: 129 active+clean; 478 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 361 op/s 2024-04-25T16:38:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:37:59 smithi173 ceph-mon[32556]: pgmap v526: 129 pgs: 129 active+clean; 478 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 361 op/s 2024-04-25T16:38:00.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:37:59 smithi203 ceph-mon[33115]: pgmap v526: 129 pgs: 129 active+clean; 478 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.7 MiB/s wr, 361 op/s 2024-04-25T16:38:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:01 smithi098 ceph-mon[29162]: pgmap v527: 129 pgs: 129 active+clean; 487 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 342 op/s 2024-04-25T16:38:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:01 smithi173 ceph-mon[32556]: pgmap v527: 129 pgs: 129 active+clean; 487 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 342 op/s 2024-04-25T16:38:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:01 smithi203 ceph-mon[33115]: pgmap v527: 129 pgs: 129 active+clean; 487 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 342 op/s 2024-04-25T16:38:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:02.928+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:02.928+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:38:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:02.928+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:38:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:02.928+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:38:02.927 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:02.928+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:38:04.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:03 smithi098 ceph-mon[29162]: pgmap v528: 129 pgs: 129 active+clean; 490 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 327 op/s 2024-04-25T16:38:04.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:03 smithi173 ceph-mon[32556]: pgmap v528: 129 pgs: 129 active+clean; 490 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 327 op/s 2024-04-25T16:38:04.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:03 smithi203 ceph-mon[33115]: pgmap v528: 129 pgs: 129 active+clean; 490 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 327 op/s 2024-04-25T16:38:06.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:05 smithi098 ceph-mon[29162]: pgmap v529: 129 pgs: 129 active+clean; 508 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.3 MiB/s wr, 375 op/s 2024-04-25T16:38:06.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:05 smithi173 ceph-mon[32556]: pgmap v529: 129 pgs: 129 active+clean; 508 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.3 MiB/s wr, 375 op/s 2024-04-25T16:38:06.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:05 smithi203 ceph-mon[33115]: pgmap v529: 129 pgs: 129 active+clean; 508 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.3 MiB/s wr, 375 op/s 2024-04-25T16:38:08.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:07 smithi098 ceph-mon[29162]: pgmap v530: 129 pgs: 129 active+clean; 511 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 304 op/s 2024-04-25T16:38:08.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:07 smithi173 ceph-mon[32556]: pgmap v530: 129 pgs: 129 active+clean; 511 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 304 op/s 2024-04-25T16:38:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:07 smithi203 ceph-mon[33115]: pgmap v530: 129 pgs: 129 active+clean; 511 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 304 op/s 2024-04-25T16:38:10.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:09 smithi098 ceph-mon[29162]: pgmap v531: 129 pgs: 129 active+clean; 555 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.6 MiB/s wr, 317 op/s 2024-04-25T16:38:10.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:09 smithi173 ceph-mon[32556]: pgmap v531: 129 pgs: 129 active+clean; 555 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.6 MiB/s wr, 317 op/s 2024-04-25T16:38:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:09 smithi203 ceph-mon[33115]: pgmap v531: 129 pgs: 129 active+clean; 555 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.6 MiB/s wr, 317 op/s 2024-04-25T16:38:12.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:11 smithi098 ceph-mon[29162]: pgmap v532: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 11 MiB/s wr, 269 op/s 2024-04-25T16:38:12.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:11 smithi173 ceph-mon[32556]: pgmap v532: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 11 MiB/s wr, 269 op/s 2024-04-25T16:38:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:11 smithi203 ceph-mon[33115]: pgmap v532: 129 pgs: 129 active+clean; 606 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 11 MiB/s wr, 269 op/s 2024-04-25T16:38:14.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:13 smithi098 ceph-mon[29162]: pgmap v533: 129 pgs: 129 active+clean; 655 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 14 MiB/s wr, 217 op/s 2024-04-25T16:38:14.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:13 smithi173 ceph-mon[32556]: pgmap v533: 129 pgs: 129 active+clean; 655 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 14 MiB/s wr, 217 op/s 2024-04-25T16:38:14.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:13 smithi203 ceph-mon[33115]: pgmap v533: 129 pgs: 129 active+clean; 655 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 14 MiB/s wr, 217 op/s 2024-04-25T16:38:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:15 smithi098 ceph-mon[29162]: pgmap v534: 129 pgs: 129 active+clean; 752 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 22 MiB/s wr, 211 op/s 2024-04-25T16:38:16.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:15 smithi173 ceph-mon[32556]: pgmap v534: 129 pgs: 129 active+clean; 752 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 22 MiB/s wr, 211 op/s 2024-04-25T16:38:16.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:15 smithi203 ceph-mon[33115]: pgmap v534: 129 pgs: 129 active+clean; 752 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 22 MiB/s wr, 211 op/s 2024-04-25T16:38:17.941 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:17.943+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:17.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:17.943+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:38:17.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:17.943+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:38:17.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:17.943+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:38:17.942 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:17.943+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:38:18.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:17 smithi173 ceph-mon[32556]: pgmap v535: 129 pgs: 129 active+clean; 800 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 24 MiB/s wr, 147 op/s 2024-04-25T16:38:18.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:17 smithi203 ceph-mon[33115]: pgmap v535: 129 pgs: 129 active+clean; 800 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 24 MiB/s wr, 147 op/s 2024-04-25T16:38:18.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:17 smithi098 ceph-mon[29162]: pgmap v535: 129 pgs: 129 active+clean; 800 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 24 MiB/s wr, 147 op/s 2024-04-25T16:38:20.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:19 smithi173 ceph-mon[32556]: pgmap v536: 129 pgs: 129 active+clean; 876 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 30 MiB/s wr, 180 op/s 2024-04-25T16:38:20.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:19 smithi203 ceph-mon[33115]: pgmap v536: 129 pgs: 129 active+clean; 876 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 30 MiB/s wr, 180 op/s 2024-04-25T16:38:20.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:19 smithi098 ceph-mon[29162]: pgmap v536: 129 pgs: 129 active+clean; 876 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 30 MiB/s wr, 180 op/s 2024-04-25T16:38:22.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:21 smithi173 ceph-mon[32556]: pgmap v537: 129 pgs: 129 active+clean; 919 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 30 MiB/s wr, 154 op/s 2024-04-25T16:38:22.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:21 smithi203 ceph-mon[33115]: pgmap v537: 129 pgs: 129 active+clean; 919 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 30 MiB/s wr, 154 op/s 2024-04-25T16:38:22.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:21 smithi098 ceph-mon[29162]: pgmap v537: 129 pgs: 129 active+clean; 919 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 30 MiB/s wr, 154 op/s 2024-04-25T16:38:22.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:22.412+0000 7fdebd7fa640 1 -- 172.21.15.98:0/2834070656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fdea00034c0 con 0x7fdec8101760 2024-04-25T16:38:22.410 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:22.412+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x55e32648e3f0 con 0x7fdec8101760 2024-04-25T16:38:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:22.417+0000 7fdea77fe640 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fde980034c0 con 0x55e32644d4d0 2024-04-25T16:38:22.415 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:22.417+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fde980034c0 con 0x55e32644d4d0 2024-04-25T16:38:24.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:23 smithi173 ceph-mon[32556]: pgmap v538: 129 pgs: 129 active+clean; 924 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 27 MiB/s wr, 150 op/s 2024-04-25T16:38:24.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:23 smithi203 ceph-mon[33115]: pgmap v538: 129 pgs: 129 active+clean; 924 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 27 MiB/s wr, 150 op/s 2024-04-25T16:38:24.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:23 smithi098 ceph-mon[29162]: pgmap v538: 129 pgs: 129 active+clean; 924 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 27 MiB/s wr, 150 op/s 2024-04-25T16:38:26.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:25 smithi173 ceph-mon[32556]: pgmap v539: 129 pgs: 129 active+clean; 931 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 23 MiB/s wr, 180 op/s 2024-04-25T16:38:26.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:25 smithi203 ceph-mon[33115]: pgmap v539: 129 pgs: 129 active+clean; 931 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 23 MiB/s wr, 180 op/s 2024-04-25T16:38:26.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:25 smithi098 ceph-mon[29162]: pgmap v539: 129 pgs: 129 active+clean; 931 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 23 MiB/s wr, 180 op/s 2024-04-25T16:38:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:27 smithi203 ceph-mon[33115]: pgmap v540: 129 pgs: 129 active+clean; 932 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 15 MiB/s wr, 167 op/s 2024-04-25T16:38:28.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:27 smithi098 ceph-mon[29162]: pgmap v540: 129 pgs: 129 active+clean; 932 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 15 MiB/s wr, 167 op/s 2024-04-25T16:38:28.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:27 smithi173 ceph-mon[32556]: pgmap v540: 129 pgs: 129 active+clean; 932 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 15 MiB/s wr, 167 op/s 2024-04-25T16:38:30.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:29 smithi203 ceph-mon[33115]: pgmap v541: 129 pgs: 129 active+clean; 944 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 12 MiB/s wr, 245 op/s 2024-04-25T16:38:30.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:29 smithi098 ceph-mon[29162]: pgmap v541: 129 pgs: 129 active+clean; 944 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 12 MiB/s wr, 245 op/s 2024-04-25T16:38:30.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:29 smithi173 ceph-mon[32556]: pgmap v541: 129 pgs: 129 active+clean; 944 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 12 MiB/s wr, 245 op/s 2024-04-25T16:38:32.052 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:31 smithi098 ceph-mon[29162]: pgmap v542: 129 pgs: 129 active+clean; 950 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 6.3 MiB/s wr, 250 op/s 2024-04-25T16:38:32.052 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:31 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:38:32.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:31 smithi203 ceph-mon[33115]: pgmap v542: 129 pgs: 129 active+clean; 950 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 6.3 MiB/s wr, 250 op/s 2024-04-25T16:38:32.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:31 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:38:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:31 smithi173 ceph-mon[32556]: pgmap v542: 129 pgs: 129 active+clean; 950 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 6.3 MiB/s wr, 250 op/s 2024-04-25T16:38:32.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:31 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:38:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:32.958+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:32.959+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:38:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:32.959+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:38:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:32.959+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:38:32.957 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:32.959+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:38:34.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:33 smithi203 ceph-mon[33115]: pgmap v543: 129 pgs: 129 active+clean; 951 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 2.8 MiB/s wr, 241 op/s 2024-04-25T16:38:34.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:33 smithi098 ceph-mon[29162]: pgmap v543: 129 pgs: 129 active+clean; 951 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 2.8 MiB/s wr, 241 op/s 2024-04-25T16:38:34.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:33 smithi173 ceph-mon[32556]: pgmap v543: 129 pgs: 129 active+clean; 951 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 2.8 MiB/s wr, 241 op/s 2024-04-25T16:38:36.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:35 smithi203 ceph-mon[33115]: pgmap v544: 129 pgs: 129 active+clean; 962 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.4 MiB/s wr, 318 op/s 2024-04-25T16:38:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:35 smithi098 ceph-mon[29162]: pgmap v544: 129 pgs: 129 active+clean; 962 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.4 MiB/s wr, 318 op/s 2024-04-25T16:38:36.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:35 smithi173 ceph-mon[32556]: pgmap v544: 129 pgs: 129 active+clean; 962 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 3.4 MiB/s wr, 318 op/s 2024-04-25T16:38:38.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:37 smithi203 ceph-mon[33115]: pgmap v545: 129 pgs: 129 active+clean; 965 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 3.0 MiB/s wr, 285 op/s 2024-04-25T16:38:38.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:37 smithi098 ceph-mon[29162]: pgmap v545: 129 pgs: 129 active+clean; 965 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 3.0 MiB/s wr, 285 op/s 2024-04-25T16:38:38.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:37 smithi173 ceph-mon[32556]: pgmap v545: 129 pgs: 129 active+clean; 965 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 3.0 MiB/s wr, 285 op/s 2024-04-25T16:38:40.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:39 smithi203 ceph-mon[33115]: pgmap v546: 129 pgs: 129 active+clean; 975 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 341 op/s 2024-04-25T16:38:40.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:39 smithi098 ceph-mon[29162]: pgmap v546: 129 pgs: 129 active+clean; 975 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 341 op/s 2024-04-25T16:38:40.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:39 smithi173 ceph-mon[32556]: pgmap v546: 129 pgs: 129 active+clean; 975 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.8 MiB/s wr, 341 op/s 2024-04-25T16:38:42.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:41 smithi203 ceph-mon[33115]: pgmap v547: 129 pgs: 129 active+clean; 984 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 3.5 MiB/s wr, 304 op/s 2024-04-25T16:38:42.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:41 smithi098 ceph-mon[29162]: pgmap v547: 129 pgs: 129 active+clean; 984 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 3.5 MiB/s wr, 304 op/s 2024-04-25T16:38:42.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:41 smithi173 ceph-mon[32556]: pgmap v547: 129 pgs: 129 active+clean; 984 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 3.5 MiB/s wr, 304 op/s 2024-04-25T16:38:44.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:43 smithi203 ceph-mon[33115]: pgmap v548: 129 pgs: 129 active+clean; 990 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 272 op/s 2024-04-25T16:38:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:43 smithi098 ceph-mon[29162]: pgmap v548: 129 pgs: 129 active+clean; 990 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 272 op/s 2024-04-25T16:38:44.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:43 smithi173 ceph-mon[32556]: pgmap v548: 129 pgs: 129 active+clean; 990 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.5 MiB/s wr, 272 op/s 2024-04-25T16:38:46.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:45 smithi203 ceph-mon[33115]: pgmap v549: 129 pgs: 129 active+clean; 1003 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 363 op/s 2024-04-25T16:38:46.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:45 smithi098 ceph-mon[29162]: pgmap v549: 129 pgs: 129 active+clean; 1003 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 363 op/s 2024-04-25T16:38:46.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:45 smithi173 ceph-mon[32556]: pgmap v549: 129 pgs: 129 active+clean; 1003 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 363 op/s 2024-04-25T16:38:47.972 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:47.974+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:38:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:47.974+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:38:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:47.974+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:38:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:47.974+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:38:47.973 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:38:47.974+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:38:48.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:47 smithi203 ceph-mon[33115]: pgmap v550: 129 pgs: 129 active+clean; 1006 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 292 op/s 2024-04-25T16:38:48.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:47 smithi098 ceph-mon[29162]: pgmap v550: 129 pgs: 129 active+clean; 1006 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 292 op/s 2024-04-25T16:38:48.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:47 smithi173 ceph-mon[32556]: pgmap v550: 129 pgs: 129 active+clean; 1006 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.9 MiB/s wr, 292 op/s 2024-04-25T16:38:50.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:49 smithi203 ceph-mon[33115]: pgmap v551: 129 pgs: 129 active+clean; 1023 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 383 op/s 2024-04-25T16:38:50.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:49 smithi098 ceph-mon[29162]: pgmap v551: 129 pgs: 129 active+clean; 1023 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 383 op/s 2024-04-25T16:38:50.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:49 smithi173 ceph-mon[32556]: pgmap v551: 129 pgs: 129 active+clean; 1023 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.1 MiB/s wr, 383 op/s 2024-04-25T16:38:52.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:51 smithi203 ceph-mon[33115]: pgmap v552: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 369 op/s 2024-04-25T16:38:52.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:51 smithi098 ceph-mon[29162]: pgmap v552: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 369 op/s 2024-04-25T16:38:52.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:51 smithi173 ceph-mon[32556]: pgmap v552: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.9 MiB/s wr, 369 op/s 2024-04-25T16:38:54.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:53 smithi203 ceph-mon[33115]: pgmap v553: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 343 op/s 2024-04-25T16:38:54.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:53 smithi098 ceph-mon[29162]: pgmap v553: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 343 op/s 2024-04-25T16:38:54.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:53 smithi173 ceph-mon[32556]: pgmap v553: 129 pgs: 129 active+clean; 1.0 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.4 MiB/s wr, 343 op/s 2024-04-25T16:38:55.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:38:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:38:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:38:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:38:55.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:38:55.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:38:55.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:38:55.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:38:55.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:38:55.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:38:55.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:38:55.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:38:55.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:38:56.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:55 smithi203 ceph-mon[33115]: pgmap v554: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.3 MiB/s wr, 420 op/s 2024-04-25T16:38:56.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:55 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T16:38:56.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:55 smithi203 ceph-mon[33115]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T16:38:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:55 smithi098 ceph-mon[29162]: pgmap v554: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.3 MiB/s wr, 420 op/s 2024-04-25T16:38:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:55 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T16:38:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:55 smithi098 ceph-mon[29162]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T16:38:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:55 smithi173 ceph-mon[32556]: pgmap v554: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.3 MiB/s wr, 420 op/s 2024-04-25T16:38:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:55 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T16:38:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:55 smithi173 ceph-mon[32556]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T16:38:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:57 smithi173 ceph-mon[32556]: pgmap v555: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 331 op/s 2024-04-25T16:38:58.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:57 smithi203 ceph-mon[33115]: pgmap v555: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 331 op/s 2024-04-25T16:38:58.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:57 smithi098 ceph-mon[29162]: pgmap v555: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 331 op/s 2024-04-25T16:39:00.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:38:59 smithi203 ceph-mon[33115]: pgmap v556: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 384 op/s 2024-04-25T16:39:00.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:38:59 smithi098 ceph-mon[29162]: pgmap v556: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 384 op/s 2024-04-25T16:39:00.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:38:59 smithi173 ceph-mon[32556]: pgmap v556: 129 pgs: 129 active+clean; 1.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 384 op/s 2024-04-25T16:39:02.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:01 smithi203 ceph-mon[33115]: pgmap v557: 129 pgs: 129 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 346 op/s 2024-04-25T16:39:02.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:01 smithi098 ceph-mon[29162]: pgmap v557: 129 pgs: 129 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 346 op/s 2024-04-25T16:39:02.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:01 smithi173 ceph-mon[32556]: pgmap v557: 129 pgs: 129 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.6 MiB/s wr, 346 op/s 2024-04-25T16:39:02.987 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:02.989+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:02.989+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:39:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:02.989+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:39:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:02.989+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:39:02.988 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:02.989+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:39:04.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:03 smithi203 ceph-mon[33115]: pgmap v558: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 311 op/s 2024-04-25T16:39:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:03 smithi098 ceph-mon[29162]: pgmap v558: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 311 op/s 2024-04-25T16:39:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:03 smithi173 ceph-mon[32556]: pgmap v558: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 311 op/s 2024-04-25T16:39:06.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:05 smithi203 ceph-mon[33115]: pgmap v559: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 364 op/s 2024-04-25T16:39:06.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:05 smithi098 ceph-mon[29162]: pgmap v559: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 364 op/s 2024-04-25T16:39:06.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:05 smithi173 ceph-mon[32556]: pgmap v559: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.0 MiB/s wr, 364 op/s 2024-04-25T16:39:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:07 smithi203 ceph-mon[33115]: pgmap v560: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 314 op/s 2024-04-25T16:39:08.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:07 smithi098 ceph-mon[29162]: pgmap v560: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 314 op/s 2024-04-25T16:39:08.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:07 smithi173 ceph-mon[32556]: pgmap v560: 129 pgs: 129 active+clean; 1.1 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.1 MiB/s wr, 314 op/s 2024-04-25T16:39:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:09 smithi203 ceph-mon[33115]: pgmap v561: 129 pgs: 129 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.8 MiB/s wr, 363 op/s 2024-04-25T16:39:10.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:09 smithi098 ceph-mon[29162]: pgmap v561: 129 pgs: 129 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.8 MiB/s wr, 363 op/s 2024-04-25T16:39:10.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:09 smithi173 ceph-mon[32556]: pgmap v561: 129 pgs: 129 active+clean; 1.1 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.8 MiB/s wr, 363 op/s 2024-04-25T16:39:12.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:11 smithi203 ceph-mon[33115]: pgmap v562: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 378 op/s 2024-04-25T16:39:12.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:11 smithi098 ceph-mon[29162]: pgmap v562: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 378 op/s 2024-04-25T16:39:12.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:11 smithi173 ceph-mon[32556]: pgmap v562: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 378 op/s 2024-04-25T16:39:13.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:12 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:13.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:12 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:13.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:12 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:14.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:13 smithi203 ceph-mon[33115]: pgmap v563: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 324 op/s 2024-04-25T16:39:14.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:13 smithi098 ceph-mon[29162]: pgmap v563: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 324 op/s 2024-04-25T16:39:14.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:13 smithi173 ceph-mon[32556]: pgmap v563: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.5 MiB/s wr, 324 op/s 2024-04-25T16:39:16.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:15 smithi203 ceph-mon[33115]: pgmap v564: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 417 op/s 2024-04-25T16:39:16.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:15 smithi098 ceph-mon[29162]: pgmap v564: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 417 op/s 2024-04-25T16:39:16.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:15 smithi173 ceph-mon[32556]: pgmap v564: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.3 MiB/s wr, 417 op/s 2024-04-25T16:39:16.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f04df7b4700 time 2024-04-25T16:39:16.406695+0000 2024-04-25T16:39:16.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f04e9fe8ba6] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04e9fe8dd1] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x564f9d98526b] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564f9d924428] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564f9d9258a9] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564f9d9279c3] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564f9d7f385d] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564f9d7f5e8e] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564f9d7f6894] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564f9d7e105b] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f04ea165ac6] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 12: (DispatchQueue::entry()+0xb27) [0x7f04ea1630b9] 2024-04-25T16:39:16.688 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f04ea24dccb] 2024-04-25T16:39:16.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 14: (Thread::entry_wrapper()+0x43) [0x7f04e9fb772d] 2024-04-25T16:39:16.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 15: (Thread::_entry_func(void*)+0xd) [0x7f04e9fb7749] 2024-04-25T16:39:16.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f04e8b5b1ca] 2024-04-25T16:39:16.689 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 17: clone() 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.408+0000 7f04df7b4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f04df7b4700 time 2024-04-25T16:39:16.406695+0000 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f04e9fe8ba6] 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04e9fe8dd1] 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x564f9d98526b] 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564f9d924428] 2024-04-25T16:39:16.690 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564f9d9258a9] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564f9d9279c3] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564f9d7f385d] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564f9d7f5e8e] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564f9d7f6894] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564f9d7e105b] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f04ea165ac6] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 12: (DispatchQueue::entry()+0xb27) [0x7f04ea1630b9] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f04ea24dccb] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 14: (Thread::entry_wrapper()+0x43) [0x7f04e9fb772d] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 15: (Thread::_entry_func(void*)+0xd) [0x7f04e9fb7749] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f04e8b5b1ca] 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 17: clone() 2024-04-25T16:39:16.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: *** Caught signal (Aborted) ** 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: in thread 7f04df7b4700 thread_name:ms_dispatch 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04e8b65d20] 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2: gsignal() 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 3: abort() 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f04e9fe8ced] 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04e9fe8dd1] 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x564f9d98526b] 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564f9d924428] 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564f9d9258a9] 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564f9d9279c3] 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564f9d7f385d] 2024-04-25T16:39:16.692 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564f9d7f5e8e] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564f9d7f6894] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564f9d7e105b] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f04ea165ac6] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 15: (DispatchQueue::entry()+0xb27) [0x7f04ea1630b9] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f04ea24dccb] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 17: (Thread::entry_wrapper()+0x43) [0x7f04e9fb772d] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 18: (Thread::_entry_func(void*)+0xd) [0x7f04e9fb7749] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f04e8b5b1ca] 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 20: clone() 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.411+0000 7f04df7b4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: in thread 7f04df7b4700 thread_name:ms_dispatch 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:16.693 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04e8b65d20] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2: gsignal() 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 3: abort() 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f04e9fe8ced] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04e9fe8dd1] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x564f9d98526b] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564f9d924428] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564f9d9258a9] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564f9d9279c3] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564f9d7f385d] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564f9d7f5e8e] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564f9d7f6894] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564f9d7e105b] 2024-04-25T16:39:16.694 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f04ea165ac6] 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 15: (DispatchQueue::entry()+0xb27) [0x7f04ea1630b9] 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f04ea24dccb] 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 17: (Thread::entry_wrapper()+0x43) [0x7f04e9fb772d] 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 18: (Thread::_entry_func(void*)+0xd) [0x7f04e9fb7749] 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f04e8b5b1ca] 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 20: clone() 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: -1> 2024-04-25T16:39:16.408+0000 7f04df7b4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f04df7b4700 time 2024-04-25T16:39:16.406695+0000 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f04e9fe8ba6] 2024-04-25T16:39:16.695 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04e9fe8dd1] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x564f9d98526b] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564f9d924428] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564f9d9258a9] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564f9d9279c3] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564f9d7f385d] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564f9d7f5e8e] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564f9d7f6894] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564f9d7e105b] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f04ea165ac6] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 12: (DispatchQueue::entry()+0xb27) [0x7f04ea1630b9] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f04ea24dccb] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 14: (Thread::entry_wrapper()+0x43) [0x7f04e9fb772d] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 15: (Thread::_entry_func(void*)+0xd) [0x7f04e9fb7749] 2024-04-25T16:39:16.696 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f04e8b5b1ca] 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 17: clone() 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 0> 2024-04-25T16:39:16.411+0000 7f04df7b4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: in thread 7f04df7b4700 thread_name:ms_dispatch 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04e8b65d20] 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2: gsignal() 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 3: abort() 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f04e9fe8ced] 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04e9fe8dd1] 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x564f9d98526b] 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564f9d924428] 2024-04-25T16:39:16.697 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564f9d9258a9] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564f9d9279c3] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564f9d7f385d] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564f9d7f5e8e] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564f9d7f6894] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564f9d7e105b] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f04ea165ac6] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 15: (DispatchQueue::entry()+0xb27) [0x7f04ea1630b9] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f04ea24dccb] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 17: (Thread::entry_wrapper()+0x43) [0x7f04e9fb772d] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 18: (Thread::_entry_func(void*)+0xd) [0x7f04e9fb7749] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f04e8b5b1ca] 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 20: clone() 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:16.698 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.699 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: -4> 2024-04-25T16:39:16.408+0000 7f04df7b4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f04df7b4700 time 2024-04-25T16:39:16.406695+0000 2024-04-25T16:39:16.699 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:39:16.699 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.699 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:16.699 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f04e9fe8ba6] 2024-04-25T16:39:16.699 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04e9fe8dd1] 2024-04-25T16:39:16.699 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x564f9d98526b] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564f9d924428] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564f9d9258a9] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564f9d9279c3] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564f9d7f385d] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564f9d7f5e8e] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564f9d7f6894] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564f9d7e105b] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f04ea165ac6] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 12: (DispatchQueue::entry()+0xb27) [0x7f04ea1630b9] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f04ea24dccb] 2024-04-25T16:39:16.700 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 14: (Thread::entry_wrapper()+0x43) [0x7f04e9fb772d] 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 15: (Thread::_entry_func(void*)+0xd) [0x7f04e9fb7749] 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f04e8b5b1ca] 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 17: clone() 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: -3> 2024-04-25T16:39:16.411+0000 7f04df7b4700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: in thread 7f04df7b4700 thread_name:ms_dispatch 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f04e8b65d20] 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2: gsignal() 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 3: abort() 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f04e9fe8ced] 2024-04-25T16:39:16.701 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f04e9fe8dd1] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x564f9d98526b] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564f9d924428] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564f9d9258a9] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564f9d9279c3] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564f9d7f385d] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564f9d7f5e8e] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564f9d7f6894] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564f9d7e105b] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f04ea165ac6] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 15: (DispatchQueue::entry()+0xb27) [0x7f04ea1630b9] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f04ea24dccb] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 17: (Thread::entry_wrapper()+0x43) [0x7f04e9fb772d] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 18: (Thread::_entry_func(void*)+0xd) [0x7f04e9fb7749] 2024-04-25T16:39:16.702 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f04e8b5b1ca] 2024-04-25T16:39:16.703 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 20: clone() 2024-04-25T16:39:16.703 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:39:16.703 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[64499]: 2024-04-25T16:39:16.703 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:16 smithi203 podman[67945]: 2024-04-25 16:39:16.494693767 +0000 UTC m=+0.027153581 container died 33242d8aa2635e47e909082b48a72890de0b43f7489d00a0e0472a8d82eeabd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:39:17.118 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:16 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:17.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:16 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:17.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:16 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:17.379 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:17 smithi203 podman[67945]: 2024-04-25 16:39:17.120709487 +0000 UTC m=+0.653169280 container remove 33242d8aa2635e47e909082b48a72890de0b43f7489d00a0e0472a8d82eeabd7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T16:39:17.379 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:17 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:39:17.686 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:17 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-25T16:39:17.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:17 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Consumed 5.530s CPU time. 2024-04-25T16:39:18.002 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:18.004+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:18.004+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:39:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:18.004+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:39:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:18.004+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:39:18.003 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:18.004+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:39:18.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:17 smithi203 ceph-mon[33115]: pgmap v565: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 365 op/s 2024-04-25T16:39:18.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:17 smithi098 ceph-mon[29162]: pgmap v565: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 365 op/s 2024-04-25T16:39:18.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:17 smithi173 ceph-mon[32556]: pgmap v565: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.6 MiB/s wr, 365 op/s 2024-04-25T16:39:20.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:19 smithi203 ceph-mon[33115]: pgmap v566: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 400 op/s 2024-04-25T16:39:20.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:19 smithi098 ceph-mon[29162]: pgmap v566: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 400 op/s 2024-04-25T16:39:20.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:19 smithi173 ceph-mon[32556]: pgmap v566: 129 pgs: 129 active+clean; 1.1 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 6.7 MiB/s wr, 400 op/s 2024-04-25T16:39:21.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:20 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:21.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:20 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:21.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:20 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:22.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:21 smithi203 ceph-mon[33115]: pgmap v567: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.4 MiB/s wr, 384 op/s 2024-04-25T16:39:22.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:21 smithi098 ceph-mon[29162]: pgmap v567: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.4 MiB/s wr, 384 op/s 2024-04-25T16:39:22.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:21 smithi173 ceph-mon[32556]: pgmap v567: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.4 MiB/s wr, 384 op/s 2024-04-25T16:39:24.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:23 smithi203 ceph-mon[33115]: pgmap v568: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T16:39:24.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:23 smithi098 ceph-mon[29162]: pgmap v568: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T16:39:24.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:23 smithi173 ceph-mon[32556]: pgmap v568: 129 pgs: 129 active+clean; 1.2 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.6 MiB/s wr, 302 op/s 2024-04-25T16:39:26.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:25 smithi203 ceph-mon[33115]: pgmap v569: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 514 op/s 2024-04-25T16:39:26.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:25 smithi098 ceph-mon[29162]: pgmap v569: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 514 op/s 2024-04-25T16:39:26.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:25 smithi173 ceph-mon[32556]: pgmap v569: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.3 MiB/s wr, 514 op/s 2024-04-25T16:39:27.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:27 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:39:27.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:27 smithi203 systemd[1]: Stopped Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:27.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:27 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Consumed 5.530s CPU time. 2024-04-25T16:39:27.691 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:27 smithi203 systemd[1]: Starting Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:39:28.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:27 smithi203 ceph-mon[33115]: pgmap v570: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 524 op/s 2024-04-25T16:39:28.187 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:27 smithi203 podman[68044]: 2024-04-25 16:39:27.596838332 +0000 UTC m=+0.020793039 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:39:28.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:27 smithi098 ceph-mon[29162]: pgmap v570: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 524 op/s 2024-04-25T16:39:28.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:27 smithi173 ceph-mon[32556]: pgmap v570: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 524 op/s 2024-04-25T16:39:29.186 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:28 smithi203 podman[68044]: 2024-04-25 16:39:28.872330625 +0000 UTC m=+1.296285305 container create 91f363d74dcebfe9ad694abf3d63494b945b268ee9f297ae0dd5fb4dd752d963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:39:30.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:29 smithi203 ceph-mon[33115]: pgmap v571: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 725 op/s 2024-04-25T16:39:30.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:29 smithi098 ceph-mon[29162]: pgmap v571: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 725 op/s 2024-04-25T16:39:30.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:29 smithi173 ceph-mon[32556]: pgmap v571: 129 pgs: 129 active+clean; 1.2 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 6.2 MiB/s wr, 725 op/s 2024-04-25T16:39:30.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:30 smithi203 podman[68044]: 2024-04-25 16:39:30.384514856 +0000 UTC m=+2.808469538 container init 91f363d74dcebfe9ad694abf3d63494b945b268ee9f297ae0dd5fb4dd752d963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:39:30.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:30 smithi203 podman[68044]: 2024-04-25 16:39:30.388824952 +0000 UTC m=+2.812779639 container start 91f363d74dcebfe9ad694abf3d63494b945b268ee9f297ae0dd5fb4dd752d963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T16:39:30.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mds[68065]: -- 172.21.15.203:0/1761048288 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561e4a0f5a40 con 0x561e4a13da80 2024-04-25T16:39:30.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: starting mds.i at 2024-04-25T16:39:30.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:30 smithi203 bash[68044]: 91f363d74dcebfe9ad694abf3d63494b945b268ee9f297ae0dd5fb4dd752d963 2024-04-25T16:39:30.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:39:30 smithi203 systemd[1]: Started Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: daemon mds.i restarted 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:boot 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=f=up:active,1=h=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:39:31.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:30 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: daemon mds.i restarted 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:boot 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=f=up:active,1=h=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:39:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:30 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: daemon mds.i restarted 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: osdmap e92: 12 total, 12 up, 12 in 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:boot 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=f=up:active,1=h=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:39:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:30 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:39:32.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:31 smithi098 ceph-mon[29162]: pgmap v572: 129 pgs: 129 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 770 op/s 2024-04-25T16:39:32.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:31 smithi203 ceph-mon[33115]: pgmap v572: 129 pgs: 129 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 770 op/s 2024-04-25T16:39:32.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:31 smithi173 ceph-mon[32556]: pgmap v572: 129 pgs: 129 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.6 MiB/s wr, 770 op/s 2024-04-25T16:39:33.017 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:33.019+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:33.019+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:39:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:33.019+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:39:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:33.019+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:39:33.018 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:33.019+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:39:33.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:32 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:resolve 2024-04-25T16:39:33.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:32 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:39:33.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:32 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:resolve 2024-04-25T16:39:33.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:32 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:39:33.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:32 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:resolve 2024-04-25T16:39:33.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:32 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:39:34.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:33 smithi203 ceph-mon[33115]: pgmap v574: 129 pgs: 129 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 858 op/s 2024-04-25T16:39:34.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:33 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:reconnect 2024-04-25T16:39:34.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:33 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:39:34.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:33 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0149996 2024-04-25T16:39:34.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:33 smithi098 ceph-mon[29162]: pgmap v574: 129 pgs: 129 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 858 op/s 2024-04-25T16:39:34.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:33 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:reconnect 2024-04-25T16:39:34.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:33 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:39:34.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:33 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0149996 2024-04-25T16:39:34.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:33 smithi173 ceph-mon[32556]: pgmap v574: 129 pgs: 129 active+clean; 1.2 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.6 MiB/s wr, 858 op/s 2024-04-25T16:39:34.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:33 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:reconnect 2024-04-25T16:39:34.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:33 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:39:34.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:33 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0149996 2024-04-25T16:39:35.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:34 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:rejoin 2024-04-25T16:39:35.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:34 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:39:35.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:34 smithi203 ceph-mon[33115]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:39:35.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:34 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:35.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:34 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:39:35.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:34 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:active 2024-04-25T16:39:35.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:34 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:35.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:34 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:rejoin 2024-04-25T16:39:35.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:34 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:39:35.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:34 smithi098 ceph-mon[29162]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:39:35.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:34 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:35.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:34 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:39:35.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:34 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:active 2024-04-25T16:39:35.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:34 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:35.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:34 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:rejoin 2024-04-25T16:39:35.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:34 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:39:35.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:34 smithi173 ceph-mon[32556]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:39:35.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:34 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:39:35.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:34 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:39:35.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:34 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/1945729206,v1:172.21.15.203:6833/1945729206] up:active 2024-04-25T16:39:35.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:34 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:36.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:35 smithi203 ceph-mon[33115]: pgmap v575: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 686 op/s 2024-04-25T16:39:36.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:35 smithi098 ceph-mon[29162]: pgmap v575: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 686 op/s 2024-04-25T16:39:36.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:35 smithi173 ceph-mon[32556]: pgmap v575: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.9 MiB/s wr, 686 op/s 2024-04-25T16:39:38.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:37 smithi203 ceph-mon[33115]: pgmap v576: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 560 op/s 2024-04-25T16:39:38.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:37 smithi098 ceph-mon[29162]: pgmap v576: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 560 op/s 2024-04-25T16:39:38.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:37 smithi173 ceph-mon[32556]: pgmap v576: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 5.1 MiB/s wr, 560 op/s 2024-04-25T16:39:40.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:39 smithi203 ceph-mon[33115]: pgmap v577: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-04-25T16:39:40.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:39 smithi098 ceph-mon[29162]: pgmap v577: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-04-25T16:39:40.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:39 smithi173 ceph-mon[32556]: pgmap v577: 129 pgs: 129 active+clean; 1.2 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.8 MiB/s wr, 482 op/s 2024-04-25T16:39:42.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:41 smithi203 ceph-mon[33115]: pgmap v578: 129 pgs: 129 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 525 op/s 2024-04-25T16:39:42.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:41 smithi098 ceph-mon[29162]: pgmap v578: 129 pgs: 129 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 525 op/s 2024-04-25T16:39:42.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:41 smithi173 ceph-mon[32556]: pgmap v578: 129 pgs: 129 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.7 MiB/s wr, 525 op/s 2024-04-25T16:39:44.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:43 smithi203 ceph-mon[33115]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 492 op/s 2024-04-25T16:39:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:43 smithi098 ceph-mon[29162]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 492 op/s 2024-04-25T16:39:44.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:43 smithi173 ceph-mon[32556]: pgmap v579: 129 pgs: 129 active+clean; 1.3 GiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 4.3 MiB/s wr, 492 op/s 2024-04-25T16:39:46.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:45 smithi203 ceph-mon[33115]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 661 op/s 2024-04-25T16:39:46.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:45 smithi098 ceph-mon[29162]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 661 op/s 2024-04-25T16:39:46.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:45 smithi173 ceph-mon[32556]: pgmap v580: 129 pgs: 129 active+clean; 1.3 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 661 op/s 2024-04-25T16:39:48.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:48.034+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:39:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:48.034+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:39:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:48.034+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:39:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:48.034+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:39:48.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:39:48.034+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:39:48.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:47 smithi203 ceph-mon[33115]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 619 op/s 2024-04-25T16:39:48.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:47 smithi098 ceph-mon[29162]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 619 op/s 2024-04-25T16:39:48.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:47 smithi173 ceph-mon[32556]: pgmap v581: 129 pgs: 129 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.4 MiB/s wr, 619 op/s 2024-04-25T16:39:50.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:49 smithi098 ceph-mon[29162]: pgmap v582: 129 pgs: 129 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 814 op/s 2024-04-25T16:39:50.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:49 smithi173 ceph-mon[32556]: pgmap v582: 129 pgs: 129 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 814 op/s 2024-04-25T16:39:50.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:49 smithi203 ceph-mon[33115]: pgmap v582: 129 pgs: 129 active+clean; 1.3 GiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 5.4 MiB/s wr, 814 op/s 2024-04-25T16:39:52.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:51 smithi098 ceph-mon[29162]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.4 MiB/s wr, 879 op/s 2024-04-25T16:39:52.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:51 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:52.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:51 smithi173 ceph-mon[32556]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.4 MiB/s wr, 879 op/s 2024-04-25T16:39:52.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:51 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:52.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:51 smithi203 ceph-mon[33115]: pgmap v583: 129 pgs: 129 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 5.4 MiB/s wr, 879 op/s 2024-04-25T16:39:52.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:51 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:39:54.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:53 smithi098 ceph-mon[29162]: pgmap v584: 129 pgs: 129 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 762 op/s 2024-04-25T16:39:54.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:53 smithi173 ceph-mon[32556]: pgmap v584: 129 pgs: 129 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 762 op/s 2024-04-25T16:39:54.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:53 smithi203 ceph-mon[33115]: pgmap v584: 129 pgs: 129 active+clean; 1.3 GiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.6 MiB/s wr, 762 op/s 2024-04-25T16:39:55.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:54 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:39:55.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:54 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:39:55.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:54 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:39:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: pgmap v585: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.3 MiB/s wr, 1.06k op/s 2024-04-25T16:39:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:39:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:39:56.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:55 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: pgmap v585: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.3 MiB/s wr, 1.06k op/s 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:39:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:39:56.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:55 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: pgmap v585: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 6.3 MiB/s wr, 1.06k op/s 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:39:56.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:39:56.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:55 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:39:57.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:56 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T16:39:57.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:56 smithi098 ceph-mon[29162]: Unable to set osd_memory_target on smithi173 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T16:39:57.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:56 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-25T16:39:57.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:56 smithi098 ceph-mon[29162]: Unable to set osd_memory_target on smithi203 to 184758784: error parsing value: Value '184758784' is below minimum 939524096 2024-04-25T16:39:57.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:56 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T16:39:57.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:56 smithi173 ceph-mon[32556]: Unable to set osd_memory_target on smithi173 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T16:39:57.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:56 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-25T16:39:57.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:56 smithi173 ceph-mon[32556]: Unable to set osd_memory_target on smithi203 to 184758784: error parsing value: Value '184758784' is below minimum 939524096 2024-04-25T16:39:57.337 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:56 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T16:39:57.337 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:56 smithi203 ceph-mon[33115]: Unable to set osd_memory_target on smithi173 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T16:39:57.337 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:56 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-25T16:39:57.337 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:56 smithi203 ceph-mon[33115]: Unable to set osd_memory_target on smithi203 to 184758784: error parsing value: Value '184758784' is below minimum 939524096 2024-04-25T16:39:58.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:57 smithi098 ceph-mon[29162]: pgmap v586: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 934 op/s 2024-04-25T16:39:58.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:57 smithi173 ceph-mon[32556]: pgmap v586: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 934 op/s 2024-04-25T16:39:58.358 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:57 smithi203 ceph-mon[33115]: pgmap v586: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 5.0 MiB/s wr, 934 op/s 2024-04-25T16:40:00.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:39:59 smithi203 ceph-mon[33115]: pgmap v587: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2024-04-25T16:40:00.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:39:59 smithi098 ceph-mon[29162]: pgmap v587: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2024-04-25T16:40:00.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:39:59 smithi173 ceph-mon[32556]: pgmap v587: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2024-04-25T16:40:01.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:00 smithi098 ceph-mon[29162]: overall HEALTH_OK 2024-04-25T16:40:01.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:00 smithi173 ceph-mon[32556]: overall HEALTH_OK 2024-04-25T16:40:01.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:00 smithi203 ceph-mon[33115]: overall HEALTH_OK 2024-04-25T16:40:02.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:01 smithi098 ceph-mon[29162]: pgmap v588: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 1.03k op/s 2024-04-25T16:40:02.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:01 smithi173 ceph-mon[32556]: pgmap v588: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 1.03k op/s 2024-04-25T16:40:02.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:01 smithi203 ceph-mon[33115]: pgmap v588: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 1.03k op/s 2024-04-25T16:40:03.047 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:03.049+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:03.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:03.049+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:40:03.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:03.049+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:40:03.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:03.049+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:40:03.048 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:03.049+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:40:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:03 smithi098 ceph-mon[29162]: pgmap v589: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 3.4 MiB/s wr, 856 op/s 2024-04-25T16:40:04.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:03 smithi173 ceph-mon[32556]: pgmap v589: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 3.4 MiB/s wr, 856 op/s 2024-04-25T16:40:04.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:03 smithi203 ceph-mon[33115]: pgmap v589: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 3.4 MiB/s wr, 856 op/s 2024-04-25T16:40:06.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:05 smithi098 ceph-mon[29162]: pgmap v590: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 1.02k op/s 2024-04-25T16:40:06.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:05 smithi173 ceph-mon[32556]: pgmap v590: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 1.02k op/s 2024-04-25T16:40:06.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:05 smithi203 ceph-mon[33115]: pgmap v590: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.7 MiB/s wr, 1.02k op/s 2024-04-25T16:40:07.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:06 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:07.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:06 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:07.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:06 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:08.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:08 smithi098 ceph-mon[29162]: pgmap v591: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 3.1 MiB/s wr, 741 op/s 2024-04-25T16:40:08.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:08 smithi173 ceph-mon[32556]: pgmap v591: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 3.1 MiB/s wr, 741 op/s 2024-04-25T16:40:08.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:08 smithi203 ceph-mon[33115]: pgmap v591: 129 pgs: 129 active+clean; 1.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 3.1 MiB/s wr, 741 op/s 2024-04-25T16:40:10.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:10 smithi098 ceph-mon[29162]: pgmap v592: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 881 op/s 2024-04-25T16:40:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:10 smithi173 ceph-mon[32556]: pgmap v592: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 881 op/s 2024-04-25T16:40:10.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:10 smithi203 ceph-mon[33115]: pgmap v592: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.1 MiB/s wr, 881 op/s 2024-04-25T16:40:12.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:12 smithi173 ceph-mon[32556]: pgmap v593: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 827 op/s 2024-04-25T16:40:12.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:12 smithi203 ceph-mon[33115]: pgmap v593: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 827 op/s 2024-04-25T16:40:12.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:12 smithi098 ceph-mon[29162]: pgmap v593: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 827 op/s 2024-04-25T16:40:13.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa70ed23700 time 2024-04-25T16:40:13.506500+0000 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa719557ba6] 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa719557dd1] 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56101ae5e26b] 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56101adfd428] 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56101adfe8a9] 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56101ae009c3] 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56101accc85d] 2024-04-25T16:40:13.764 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56101accee8e] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56101accf894] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56101acba05b] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa7196d4ac6] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 12: (DispatchQueue::entry()+0xb27) [0x7fa7196d20b9] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa7197bcccb] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 14: (Thread::entry_wrapper()+0x43) [0x7fa71952672d] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa719526749] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa7180ca1ca] 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 17: clone() 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.508+0000 7fa70ed23700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa70ed23700 time 2024-04-25T16:40:13.506500+0000 2024-04-25T16:40:13.765 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa719557ba6] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa719557dd1] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56101ae5e26b] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56101adfd428] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56101adfe8a9] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56101ae009c3] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56101accc85d] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56101accee8e] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56101accf894] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56101acba05b] 2024-04-25T16:40:13.766 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa7196d4ac6] 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 12: (DispatchQueue::entry()+0xb27) [0x7fa7196d20b9] 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa7197bcccb] 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 14: (Thread::entry_wrapper()+0x43) [0x7fa71952672d] 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa719526749] 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa7180ca1ca] 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 17: clone() 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: *** Caught signal (Aborted) ** 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: in thread 7fa70ed23700 thread_name:ms_dispatch 2024-04-25T16:40:13.767 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:13.768 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa7180d4d20] 2024-04-25T16:40:13.768 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2: gsignal() 2024-04-25T16:40:13.768 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 3: abort() 2024-04-25T16:40:13.768 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa719557ced] 2024-04-25T16:40:13.768 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa719557dd1] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56101ae5e26b] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56101adfd428] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56101adfe8a9] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56101ae009c3] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56101accc85d] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56101accee8e] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56101accf894] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56101acba05b] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa7196d4ac6] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 15: (DispatchQueue::entry()+0xb27) [0x7fa7196d20b9] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa7197bcccb] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 17: (Thread::entry_wrapper()+0x43) [0x7fa71952672d] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa719526749] 2024-04-25T16:40:13.769 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa7180ca1ca] 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 20: clone() 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.510+0000 7fa70ed23700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: in thread 7fa70ed23700 thread_name:ms_dispatch 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa7180d4d20] 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2: gsignal() 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 3: abort() 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa719557ced] 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa719557dd1] 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56101ae5e26b] 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56101adfd428] 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56101adfe8a9] 2024-04-25T16:40:13.770 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56101ae009c3] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56101accc85d] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56101accee8e] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56101accf894] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56101acba05b] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa7196d4ac6] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 15: (DispatchQueue::entry()+0xb27) [0x7fa7196d20b9] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa7197bcccb] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 17: (Thread::entry_wrapper()+0x43) [0x7fa71952672d] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa719526749] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa7180ca1ca] 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 20: clone() 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.771 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: -1> 2024-04-25T16:40:13.508+0000 7fa70ed23700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa70ed23700 time 2024-04-25T16:40:13.506500+0000 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa719557ba6] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa719557dd1] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56101ae5e26b] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56101adfd428] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56101adfe8a9] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56101ae009c3] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56101accc85d] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56101accee8e] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56101accf894] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56101acba05b] 2024-04-25T16:40:13.772 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa7196d4ac6] 2024-04-25T16:40:13.773 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 12: (DispatchQueue::entry()+0xb27) [0x7fa7196d20b9] 2024-04-25T16:40:13.773 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa7197bcccb] 2024-04-25T16:40:13.773 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 14: (Thread::entry_wrapper()+0x43) [0x7fa71952672d] 2024-04-25T16:40:13.773 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa719526749] 2024-04-25T16:40:13.773 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa7180ca1ca] 2024-04-25T16:40:13.773 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 17: clone() 2024-04-25T16:40:13.773 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.773 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 0> 2024-04-25T16:40:13.510+0000 7fa70ed23700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: in thread 7fa70ed23700 thread_name:ms_dispatch 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa7180d4d20] 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2: gsignal() 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 3: abort() 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa719557ced] 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa719557dd1] 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56101ae5e26b] 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56101adfd428] 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56101adfe8a9] 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56101ae009c3] 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56101accc85d] 2024-04-25T16:40:13.774 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56101accee8e] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56101accf894] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56101acba05b] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa7196d4ac6] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 15: (DispatchQueue::entry()+0xb27) [0x7fa7196d20b9] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa7197bcccb] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 17: (Thread::entry_wrapper()+0x43) [0x7fa71952672d] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa719526749] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa7180ca1ca] 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 20: clone() 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: -1> 2024-04-25T16:40:13.508+0000 7fa70ed23700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fa70ed23700 time 2024-04-25T16:40:13.506500+0000 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:13.775 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fa719557ba6] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa719557dd1] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56101ae5e26b] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56101adfd428] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56101adfe8a9] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56101ae009c3] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56101accc85d] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56101accee8e] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56101accf894] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56101acba05b] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa7196d4ac6] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 12: (DispatchQueue::entry()+0xb27) [0x7fa7196d20b9] 2024-04-25T16:40:13.776 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa7197bcccb] 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 14: (Thread::entry_wrapper()+0x43) [0x7fa71952672d] 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 15: (Thread::_entry_func(void*)+0xd) [0x7fa719526749] 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fa7180ca1ca] 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 17: clone() 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 0> 2024-04-25T16:40:13.510+0000 7fa70ed23700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: in thread 7fa70ed23700 thread_name:ms_dispatch 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa7180d4d20] 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2: gsignal() 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 3: abort() 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fa719557ced] 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fa719557dd1] 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56101ae5e26b] 2024-04-25T16:40:13.777 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56101adfd428] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56101adfe8a9] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56101ae009c3] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56101accc85d] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56101accee8e] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56101accf894] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56101acba05b] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fa7196d4ac6] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 15: (DispatchQueue::entry()+0xb27) [0x7fa7196d20b9] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa7197bcccb] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 17: (Thread::entry_wrapper()+0x43) [0x7fa71952672d] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 18: (Thread::_entry_func(void*)+0xd) [0x7fa719526749] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fa7180ca1ca] 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 20: clone() 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:13.778 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[62497]: 2024-04-25T16:40:13.779 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:13 smithi173 podman[66383]: 2024-04-25 16:40:13.579404894 +0000 UTC m=+0.022101689 container died e35367aa004fd27ad58265fb85aea9df2269a3c33d89c8b7d4745b017273b04d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True) 2024-04-25T16:40:14.155 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:14 smithi173 ceph-mon[32556]: pgmap v594: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 725 op/s 2024-04-25T16:40:14.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:14 smithi203 ceph-mon[33115]: pgmap v594: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 725 op/s 2024-04-25T16:40:14.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:14 smithi098 ceph-mon[29162]: pgmap v594: 129 pgs: 129 active+clean; 1.4 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 725 op/s 2024-04-25T16:40:14.930 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:14 smithi173 podman[66383]: 2024-04-25 16:40:14.668641468 +0000 UTC m=+1.111338244 container remove e35367aa004fd27ad58265fb85aea9df2269a3c33d89c8b7d4745b017273b04d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-04-25T16:40:14.930 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:14 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:40:15.263 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:14 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-25T16:40:15.263 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:14 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Consumed 22.282s CPU time. 2024-04-25T16:40:16.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:16 smithi173 ceph-mon[32556]: pgmap v595: 129 pgs: 129 active+clean; 1.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 952 op/s 2024-04-25T16:40:16.313 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-mon[33115]: pgmap v595: 129 pgs: 129 active+clean; 1.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 952 op/s 2024-04-25T16:40:16.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:16 smithi098 ceph-mon[29162]: pgmap v595: 129 pgs: 129 active+clean; 1.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.5 MiB/s wr, 952 op/s 2024-04-25T16:40:16.677 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe19bce1700 time 2024-04-25T16:40:16.315633+0000 2024-04-25T16:40:16.678 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:16.678 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:16.678 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe1a6515ba6] 2024-04-25T16:40:16.678 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe1a6515dd1] 2024-04-25T16:40:16.678 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5605cf72c26b] 2024-04-25T16:40:16.678 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5605cf6cb428] 2024-04-25T16:40:16.678 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5605cf6cc8a9] 2024-04-25T16:40:16.678 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5605cf6ce9c3] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5605cf59a85d] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5605cf59ce8e] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5605cf59d894] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5605cf58805b] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe1a6692ac6] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 12: (DispatchQueue::entry()+0xb27) [0x7fe1a66900b9] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe1a677accb] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 14: (Thread::entry_wrapper()+0x43) [0x7fe1a64e472d] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe1a64e4749] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe1a50881ca] 2024-04-25T16:40:16.679 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 17: clone() 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.317+0000 7fe19bce1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe19bce1700 time 2024-04-25T16:40:16.315633+0000 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe1a6515ba6] 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe1a6515dd1] 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5605cf72c26b] 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5605cf6cb428] 2024-04-25T16:40:16.681 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5605cf6cc8a9] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5605cf6ce9c3] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5605cf59a85d] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5605cf59ce8e] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5605cf59d894] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5605cf58805b] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe1a6692ac6] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 12: (DispatchQueue::entry()+0xb27) [0x7fe1a66900b9] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe1a677accb] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 14: (Thread::entry_wrapper()+0x43) [0x7fe1a64e472d] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe1a64e4749] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe1a50881ca] 2024-04-25T16:40:16.682 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 17: clone() 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: *** Caught signal (Aborted) ** 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: in thread 7fe19bce1700 thread_name:ms_dispatch 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe1a5092d20] 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2: gsignal() 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 3: abort() 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe1a6515ced] 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe1a6515dd1] 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5605cf72c26b] 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5605cf6cb428] 2024-04-25T16:40:16.683 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5605cf6cc8a9] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5605cf6ce9c3] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5605cf59a85d] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5605cf59ce8e] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5605cf59d894] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5605cf58805b] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe1a6692ac6] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 15: (DispatchQueue::entry()+0xb27) [0x7fe1a66900b9] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe1a677accb] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 17: (Thread::entry_wrapper()+0x43) [0x7fe1a64e472d] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe1a64e4749] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe1a50881ca] 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 20: clone() 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.320+0000 7fe19bce1700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:16.684 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: in thread 7fe19bce1700 thread_name:ms_dispatch 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe1a5092d20] 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2: gsignal() 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 3: abort() 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe1a6515ced] 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe1a6515dd1] 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5605cf72c26b] 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5605cf6cb428] 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5605cf6cc8a9] 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5605cf6ce9c3] 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5605cf59a85d] 2024-04-25T16:40:16.685 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5605cf59ce8e] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5605cf59d894] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5605cf58805b] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe1a6692ac6] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 15: (DispatchQueue::entry()+0xb27) [0x7fe1a66900b9] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe1a677accb] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 17: (Thread::entry_wrapper()+0x43) [0x7fe1a64e472d] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe1a64e4749] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe1a50881ca] 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 20: clone() 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: -1> 2024-04-25T16:40:16.317+0000 7fe19bce1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe19bce1700 time 2024-04-25T16:40:16.315633+0000 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe1a6515ba6] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe1a6515dd1] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5605cf72c26b] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5605cf6cb428] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5605cf6cc8a9] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5605cf6ce9c3] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5605cf59a85d] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5605cf59ce8e] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5605cf59d894] 2024-04-25T16:40:16.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5605cf58805b] 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe1a6692ac6] 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 12: (DispatchQueue::entry()+0xb27) [0x7fe1a66900b9] 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe1a677accb] 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 14: (Thread::entry_wrapper()+0x43) [0x7fe1a64e472d] 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe1a64e4749] 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe1a50881ca] 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 17: clone() 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 0> 2024-04-25T16:40:16.320+0000 7fe19bce1700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: in thread 7fe19bce1700 thread_name:ms_dispatch 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe1a5092d20] 2024-04-25T16:40:16.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2: gsignal() 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 3: abort() 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe1a6515ced] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe1a6515dd1] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5605cf72c26b] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5605cf6cb428] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5605cf6cc8a9] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5605cf6ce9c3] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5605cf59a85d] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5605cf59ce8e] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5605cf59d894] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5605cf58805b] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe1a6692ac6] 2024-04-25T16:40:16.689 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 15: (DispatchQueue::entry()+0xb27) [0x7fe1a66900b9] 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe1a677accb] 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 17: (Thread::entry_wrapper()+0x43) [0x7fe1a64e472d] 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe1a64e4749] 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe1a50881ca] 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 20: clone() 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: -1> 2024-04-25T16:40:16.317+0000 7fe19bce1700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fe19bce1700 time 2024-04-25T16:40:16.315633+0000 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fe1a6515ba6] 2024-04-25T16:40:16.690 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe1a6515dd1] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5605cf72c26b] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5605cf6cb428] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5605cf6cc8a9] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5605cf6ce9c3] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5605cf59a85d] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5605cf59ce8e] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5605cf59d894] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5605cf58805b] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe1a6692ac6] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 12: (DispatchQueue::entry()+0xb27) [0x7fe1a66900b9] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe1a677accb] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 14: (Thread::entry_wrapper()+0x43) [0x7fe1a64e472d] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 15: (Thread::_entry_func(void*)+0xd) [0x7fe1a64e4749] 2024-04-25T16:40:16.691 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fe1a50881ca] 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 17: clone() 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 0> 2024-04-25T16:40:16.320+0000 7fe19bce1700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: in thread 7fe19bce1700 thread_name:ms_dispatch 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe1a5092d20] 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2: gsignal() 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 3: abort() 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fe1a6515ced] 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe1a6515dd1] 2024-04-25T16:40:16.692 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5605cf72c26b] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5605cf6cb428] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5605cf6cc8a9] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5605cf6ce9c3] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5605cf59a85d] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5605cf59ce8e] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5605cf59d894] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5605cf58805b] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fe1a6692ac6] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 15: (DispatchQueue::entry()+0xb27) [0x7fe1a66900b9] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe1a677accb] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 17: (Thread::entry_wrapper()+0x43) [0x7fe1a64e472d] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 18: (Thread::_entry_func(void*)+0xd) [0x7fe1a64e4749] 2024-04-25T16:40:16.693 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fe1a50881ca] 2024-04-25T16:40:16.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 20: clone() 2024-04-25T16:40:16.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:16.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[67247]: 2024-04-25T16:40:16.694 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:16 smithi203 podman[68273]: 2024-04-25 16:40:16.391619481 +0000 UTC m=+0.019824713 container died 3e5f85b27e25c56249efa27cea5d11a03b9e58f4e1dae9204fe574c06cae87e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T16:40:17.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:17 smithi203 podman[68273]: 2024-04-25 16:40:17.590770318 +0000 UTC m=+1.218975602 container remove 3e5f85b27e25c56249efa27cea5d11a03b9e58f4e1dae9204fe574c06cae87e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T16:40:17.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:17 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:40:17.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:17 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:40:17.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:17 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 3min 44.666s CPU time. 2024-04-25T16:40:18.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:18.059+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:6834/2230353021 2024-04-25T16:40:18.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:18.059+0000 7fdeb0ff9640 1 client.16329 ms_handle_refused on v2:172.21.15.203:6834/2230353021 2024-04-25T16:40:18.057 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:18.059+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:40:18.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:18 smithi173 ceph-mon[32556]: pgmap v596: 129 pgs: 129 active+clean; 1.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 818 op/s 2024-04-25T16:40:18.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:18 smithi203 ceph-mon[33115]: pgmap v596: 129 pgs: 129 active+clean; 1.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 818 op/s 2024-04-25T16:40:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:18 smithi098 ceph-mon[29162]: pgmap v596: 129 pgs: 129 active+clean; 1.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.3 MiB/s wr, 818 op/s 2024-04-25T16:40:20.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:20 smithi203 ceph-mon[33115]: pgmap v597: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 853 op/s 2024-04-25T16:40:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:20 smithi098 ceph-mon[29162]: pgmap v597: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 853 op/s 2024-04-25T16:40:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:20 smithi173 ceph-mon[32556]: pgmap v597: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 4.9 MiB/s wr, 853 op/s 2024-04-25T16:40:22.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:22 smithi203 ceph-mon[33115]: pgmap v598: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 717 op/s 2024-04-25T16:40:22.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:22 smithi098 ceph-mon[29162]: pgmap v598: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 717 op/s 2024-04-25T16:40:22.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:22 smithi173 ceph-mon[32556]: pgmap v598: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.2 MiB/s wr, 717 op/s 2024-04-25T16:40:24.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:24 smithi203 ceph-mon[33115]: pgmap v599: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 573 op/s 2024-04-25T16:40:24.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:24 smithi098 ceph-mon[29162]: pgmap v599: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 573 op/s 2024-04-25T16:40:24.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:24 smithi173 ceph-mon[32556]: pgmap v599: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.3 MiB/s wr, 573 op/s 2024-04-25T16:40:25.285 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:40:25.285 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 systemd[1]: Stopped Ceph mds.h for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:25.285 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Consumed 22.282s CPU time. 2024-04-25T16:40:25.285 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 systemd[1]: Starting Ceph mds.h for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:40:25.285 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 podman[66481]: 2024-04-25 16:40:25.155901614 +0000 UTC m=+0.018529204 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:40:25.555 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 podman[66481]: 2024-04-25 16:40:25.287520055 +0000 UTC m=+0.150147661 container create 33e77d5898bc8a7ed94d6d6ac8b9d06abe63211843bceef294f25337dd286cfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T16:40:26.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 podman[66481]: 2024-04-25 16:40:25.557733043 +0000 UTC m=+0.420360611 container init 33e77d5898bc8a7ed94d6d6ac8b9d06abe63211843bceef294f25337dd286cfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T16:40:26.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 podman[66481]: 2024-04-25 16:40:25.5604065 +0000 UTC m=+0.423034073 container start 33e77d5898bc8a7ed94d6d6ac8b9d06abe63211843bceef294f25337dd286cfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-25T16:40:26.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 ceph-mds[66500]: -- 172.21.15.173:0/556125357 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e8a485a40 con 0x563e8a4cd600 2024-04-25T16:40:26.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: starting mds.h at 2024-04-25T16:40:26.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 bash[66481]: 33e77d5898bc8a7ed94d6d6ac8b9d06abe63211843bceef294f25337dd286cfe 2024-04-25T16:40:26.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:40:25 smithi173 systemd[1]: Started Ceph mds.h for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:26.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: pgmap v600: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 2.9 MiB/s wr, 531 op/s 2024-04-25T16:40:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: daemon mds.h restarted 2024-04-25T16:40:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T16:40:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:boot 2024-04-25T16:40:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-25T16:40:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=f=up:active,2=l=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:40:26.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:26 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:40:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: pgmap v600: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 2.9 MiB/s wr, 531 op/s 2024-04-25T16:40:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: daemon mds.h restarted 2024-04-25T16:40:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T16:40:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:boot 2024-04-25T16:40:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-25T16:40:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=f=up:active,2=l=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:26.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:40:26.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:26 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: pgmap v600: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 2.9 MiB/s wr, 531 op/s 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: daemon mds.h restarted 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: osdmap e93: 12 total, 12 up, 12 in 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:boot 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=f=up:active,2=l=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:40:26.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:26 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:40:27.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:27 smithi203 ceph-mon[33115]: pgmap v602: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-04-25T16:40:27.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:27 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:resolve 2024-04-25T16:40:27.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:27 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:40:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:27 smithi098 ceph-mon[29162]: pgmap v602: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-04-25T16:40:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:27 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:resolve 2024-04-25T16:40:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:27 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:40:28.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:27 smithi173 ceph-mon[32556]: pgmap v602: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 1.6 MiB/s wr, 209 op/s 2024-04-25T16:40:28.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:27 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:resolve 2024-04-25T16:40:28.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:27 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:40:28.214 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:27 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-04-25T16:40:28.214 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:27 smithi203 systemd[1]: Stopped Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:28.214 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:27 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 3min 44.666s CPU time. 2024-04-25T16:40:28.214 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:27 smithi203 systemd[1]: Starting Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:40:28.214 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:28 smithi203 podman[68374]: 2024-04-25 16:40:28.082533554 +0000 UTC m=+0.018613990 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:40:28.214 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:28 smithi203 podman[68374]: 2024-04-25 16:40:28.216465341 +0000 UTC m=+0.152545808 container create c8fe897e36cbced92dcf5365bf55e734492246b1669633e5637b5c89e162888a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-25T16:40:28.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-mon[33115]: daemon mds.f restarted 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-mon[33115]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:boot 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-mon[33115]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {1=k=up:resolve,2=l=up:active,3=e=up:active,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:28 smithi203 podman[68374]: 2024-04-25 16:40:28.485897544 +0000 UTC m=+0.421977969 container init c8fe897e36cbced92dcf5365bf55e734492246b1669633e5637b5c89e162888a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:28 smithi203 podman[68374]: 2024-04-25 16:40:28.488850065 +0000 UTC m=+0.424930501 container start c8fe897e36cbced92dcf5365bf55e734492246b1669633e5637b5c89e162888a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T16:40:28.937 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-mds[68392]: -- 172.21.15.203:0/3496868465 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7d198fa40 con 0x55b7d19d7180 2024-04-25T16:40:28.938 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: starting mds.f at 2024-04-25T16:40:28.938 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:28 smithi203 bash[68374]: c8fe897e36cbced92dcf5365bf55e734492246b1669633e5637b5c89e162888a 2024-04-25T16:40:28.938 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:40:28 smithi203 systemd[1]: Started Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:29.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:28 smithi098 ceph-mon[29162]: daemon mds.f restarted 2024-04-25T16:40:29.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:28 smithi098 ceph-mon[29162]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T16:40:29.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:28 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:boot 2024-04-25T16:40:29.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:28 smithi098 ceph-mon[29162]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:40:29.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:28 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {1=k=up:resolve,2=l=up:active,3=e=up:active,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:40:29.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:28 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:40:29.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:28 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:40:29.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:28 smithi173 ceph-mon[32556]: daemon mds.f restarted 2024-04-25T16:40:29.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:28 smithi173 ceph-mon[32556]: osdmap e94: 12 total, 12 up, 12 in 2024-04-25T16:40:29.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:28 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:boot 2024-04-25T16:40:29.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:28 smithi173 ceph-mon[32556]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:40:29.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:28 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {1=k=up:resolve,2=l=up:active,3=e=up:active,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:40:29.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:28 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:40:29.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:28 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:40:29.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:29 smithi203 ceph-mon[33115]: pgmap v603: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 615 KiB/s wr, 99 op/s 2024-04-25T16:40:30.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:29 smithi098 ceph-mon[29162]: pgmap v603: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 615 KiB/s wr, 99 op/s 2024-04-25T16:40:30.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:29 smithi173 ceph-mon[32556]: pgmap v603: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 615 KiB/s wr, 99 op/s 2024-04-25T16:40:30.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:30 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:resolve 2024-04-25T16:40:30.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:30 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:40:31.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:30 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:resolve 2024-04-25T16:40:31.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:30 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:40:31.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:30 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:resolve 2024-04-25T16:40:31.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:30 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:40:31.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:31 smithi203 ceph-mon[33115]: pgmap v605: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 4.6 KiB/s wr, 57 op/s 2024-04-25T16:40:31.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:31 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:reconnect 2024-04-25T16:40:31.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:31 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:reconnect 2024-04-25T16:40:31.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:31 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:40:31.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:31 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0329993 2024-04-25T16:40:31.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:31 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0349993 2024-04-25T16:40:32.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:31 smithi098 ceph-mon[29162]: pgmap v605: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 4.6 KiB/s wr, 57 op/s 2024-04-25T16:40:32.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:31 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:reconnect 2024-04-25T16:40:32.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:31 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:reconnect 2024-04-25T16:40:32.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:31 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:40:32.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:31 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0329993 2024-04-25T16:40:32.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:31 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0349993 2024-04-25T16:40:32.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:31 smithi173 ceph-mon[32556]: pgmap v605: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 4.6 KiB/s wr, 57 op/s 2024-04-25T16:40:32.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:31 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:reconnect 2024-04-25T16:40:32.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:31 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:reconnect 2024-04-25T16:40:32.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:31 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:40:32.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:31 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0329993 2024-04-25T16:40:32.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:31 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0349993 2024-04-25T16:40:32.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:32 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:rejoin 2024-04-25T16:40:32.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:32 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:rejoin 2024-04-25T16:40:32.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:32 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:40:32.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:32 smithi203 ceph-mon[33115]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-25T16:40:33.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:32 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:rejoin 2024-04-25T16:40:33.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:32 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:rejoin 2024-04-25T16:40:33.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:32 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:40:33.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:32 smithi098 ceph-mon[29162]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-25T16:40:33.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:32 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:rejoin 2024-04-25T16:40:33.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:32 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:rejoin 2024-04-25T16:40:33.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:32 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:40:33.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:32 smithi173 ceph-mon[32556]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-25T16:40:33.072 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:33.074+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:33.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:33.074+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:40:33.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:33.074+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:40:33.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:33.074+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:40:33.073 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:33.074+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:40:33.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:33 smithi203 ceph-mon[33115]: pgmap v606: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 2.1 KiB/s wr, 55 op/s 2024-04-25T16:40:33.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:33 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:active 2024-04-25T16:40:33.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:33 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:clientreplay 2024-04-25T16:40:33.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:33 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:40:34.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:33 smithi098 ceph-mon[29162]: pgmap v606: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 2.1 KiB/s wr, 55 op/s 2024-04-25T16:40:34.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:33 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:active 2024-04-25T16:40:34.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:33 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:clientreplay 2024-04-25T16:40:34.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:33 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:40:34.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:33 smithi173 ceph-mon[32556]: pgmap v606: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 2.1 KiB/s wr, 55 op/s 2024-04-25T16:40:34.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:33 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/3037734321,v1:172.21.15.173:6833/3037734321] up:active 2024-04-25T16:40:34.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:33 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:clientreplay 2024-04-25T16:40:34.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:33 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:40:34.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:34 smithi203 ceph-mon[33115]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:40:35.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:34 smithi098 ceph-mon[29162]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:40:35.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:34 smithi173 ceph-mon[32556]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:40:35.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:35 smithi203 ceph-mon[33115]: pgmap v607: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 11 KiB/s wr, 64 op/s 2024-04-25T16:40:35.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:35 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:35.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:35 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:40:35.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:35 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:active 2024-04-25T16:40:35.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:35 smithi203 ceph-mon[33115]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:40:35.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:35 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:36.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:35 smithi098 ceph-mon[29162]: pgmap v607: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 11 KiB/s wr, 64 op/s 2024-04-25T16:40:36.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:35 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:36.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:35 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:40:36.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:35 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:active 2024-04-25T16:40:36.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:35 smithi098 ceph-mon[29162]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:40:36.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:35 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:36.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:35 smithi173 ceph-mon[32556]: pgmap v607: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 11 KiB/s wr, 64 op/s 2024-04-25T16:40:36.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:35 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:40:36.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:35 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:40:36.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:35 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/2873375163,v1:172.21.15.173:6839/2873375163] up:active 2024-04-25T16:40:36.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:35 smithi173 ceph-mon[32556]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:40:36.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:35 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:37.671 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:37.398+0000 7fedca961700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T16:40:37.672 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:37.398+0000 7fedca961700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:40:37.672 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:37.398+0000 7fedca961700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:40:37.672 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:37.398+0000 7fedca961700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T16:40:37.672 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:37.398+0000 7fedca961700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:40:37.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:37 smithi203 ceph-mon[33115]: pgmap v608: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.5 KiB/s wr, 56 op/s 2024-04-25T16:40:38.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:37 smithi098 ceph-mon[29162]: pgmap v608: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.5 KiB/s wr, 56 op/s 2024-04-25T16:40:38.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:37 smithi173 ceph-mon[32556]: pgmap v608: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 9.5 KiB/s wr, 56 op/s 2024-04-25T16:40:39.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:39 smithi203 ceph-mon[33115]: pgmap v609: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 578 KiB/s wr, 778 op/s 2024-04-25T16:40:39.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:39 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:40.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:39 smithi098 ceph-mon[29162]: pgmap v609: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 578 KiB/s wr, 778 op/s 2024-04-25T16:40:40.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:39 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:40.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:39 smithi173 ceph-mon[32556]: pgmap v609: 129 pgs: 129 active+clean; 1.4 GiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 578 KiB/s wr, 778 op/s 2024-04-25T16:40:40.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:39 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:40:41.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fedcd166700 time 2024-04-25T16:40:40.679607+0000 2024-04-25T16:40:41.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fedd799aba6] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fedd799add1] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a38fcc326b] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a38fc62428] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a38fc638a9] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a38fc659c3] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a38fb3185d] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a38fb33e8e] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a38fb34894] 2024-04-25T16:40:41.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a38fb1f05b] 2024-04-25T16:40:41.015 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fedd7b17ac6] 2024-04-25T16:40:41.015 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 12: (DispatchQueue::entry()+0xb27) [0x7fedd7b150b9] 2024-04-25T16:40:41.015 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fedd7bffccb] 2024-04-25T16:40:41.015 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 14: (Thread::entry_wrapper()+0x43) [0x7fedd796972d] 2024-04-25T16:40:41.015 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 15: (Thread::_entry_func(void*)+0xd) [0x7fedd7969749] 2024-04-25T16:40:41.015 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fedd650d1ca] 2024-04-25T16:40:41.015 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 17: clone() 2024-04-25T16:40:41.016 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:40.682+0000 7fedcd166700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fedcd166700 time 2024-04-25T16:40:40.679607+0000 2024-04-25T16:40:41.016 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fedd799aba6] 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fedd799add1] 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a38fcc326b] 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a38fc62428] 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a38fc638a9] 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a38fc659c3] 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a38fb3185d] 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a38fb33e8e] 2024-04-25T16:40:41.017 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a38fb34894] 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a38fb1f05b] 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fedd7b17ac6] 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 12: (DispatchQueue::entry()+0xb27) [0x7fedd7b150b9] 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fedd7bffccb] 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 14: (Thread::entry_wrapper()+0x43) [0x7fedd796972d] 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 15: (Thread::_entry_func(void*)+0xd) [0x7fedd7969749] 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fedd650d1ca] 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 17: clone() 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: *** Caught signal (Aborted) ** 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: in thread 7fedcd166700 thread_name:ms_dispatch 2024-04-25T16:40:41.018 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fedd6517d20] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2: gsignal() 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 3: abort() 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fedd799aced] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fedd799add1] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a38fcc326b] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a38fc62428] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a38fc638a9] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a38fc659c3] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a38fb3185d] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a38fb33e8e] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a38fb34894] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a38fb1f05b] 2024-04-25T16:40:41.019 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fedd7b17ac6] 2024-04-25T16:40:41.020 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 15: (DispatchQueue::entry()+0xb27) [0x7fedd7b150b9] 2024-04-25T16:40:41.020 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fedd7bffccb] 2024-04-25T16:40:41.020 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 17: (Thread::entry_wrapper()+0x43) [0x7fedd796972d] 2024-04-25T16:40:41.020 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 18: (Thread::_entry_func(void*)+0xd) [0x7fedd7969749] 2024-04-25T16:40:41.020 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fedd650d1ca] 2024-04-25T16:40:41.020 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 20: clone() 2024-04-25T16:40:41.021 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:40.684+0000 7fedcd166700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:41.021 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: in thread 7fedcd166700 thread_name:ms_dispatch 2024-04-25T16:40:41.021 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.021 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:41.021 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fedd6517d20] 2024-04-25T16:40:41.022 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2: gsignal() 2024-04-25T16:40:41.022 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 3: abort() 2024-04-25T16:40:41.023 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fedd799aced] 2024-04-25T16:40:41.023 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fedd799add1] 2024-04-25T16:40:41.023 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a38fcc326b] 2024-04-25T16:40:41.024 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a38fc62428] 2024-04-25T16:40:41.024 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a38fc638a9] 2024-04-25T16:40:41.024 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a38fc659c3] 2024-04-25T16:40:41.024 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a38fb3185d] 2024-04-25T16:40:41.024 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a38fb33e8e] 2024-04-25T16:40:41.024 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a38fb34894] 2024-04-25T16:40:41.025 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a38fb1f05b] 2024-04-25T16:40:41.025 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fedd7b17ac6] 2024-04-25T16:40:41.025 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 15: (DispatchQueue::entry()+0xb27) [0x7fedd7b150b9] 2024-04-25T16:40:41.025 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fedd7bffccb] 2024-04-25T16:40:41.025 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 17: (Thread::entry_wrapper()+0x43) [0x7fedd796972d] 2024-04-25T16:40:41.025 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 18: (Thread::_entry_func(void*)+0xd) [0x7fedd7969749] 2024-04-25T16:40:41.026 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fedd650d1ca] 2024-04-25T16:40:41.026 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 20: clone() 2024-04-25T16:40:41.026 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:41.026 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.026 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: -1> 2024-04-25T16:40:40.682+0000 7fedcd166700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fedcd166700 time 2024-04-25T16:40:40.679607+0000 2024-04-25T16:40:41.027 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:41.027 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.027 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:41.027 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fedd799aba6] 2024-04-25T16:40:41.027 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fedd799add1] 2024-04-25T16:40:41.027 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a38fcc326b] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a38fc62428] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a38fc638a9] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a38fc659c3] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a38fb3185d] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a38fb33e8e] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a38fb34894] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a38fb1f05b] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fedd7b17ac6] 2024-04-25T16:40:41.028 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 12: (DispatchQueue::entry()+0xb27) [0x7fedd7b150b9] 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fedd7bffccb] 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 14: (Thread::entry_wrapper()+0x43) [0x7fedd796972d] 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 15: (Thread::_entry_func(void*)+0xd) [0x7fedd7969749] 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fedd650d1ca] 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 17: clone() 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 0> 2024-04-25T16:40:40.684+0000 7fedcd166700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: in thread 7fedcd166700 thread_name:ms_dispatch 2024-04-25T16:40:41.029 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.030 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:41.030 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fedd6517d20] 2024-04-25T16:40:41.030 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2: gsignal() 2024-04-25T16:40:41.030 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 3: abort() 2024-04-25T16:40:41.030 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fedd799aced] 2024-04-25T16:40:41.030 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fedd799add1] 2024-04-25T16:40:41.030 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a38fcc326b] 2024-04-25T16:40:41.030 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a38fc62428] 2024-04-25T16:40:41.031 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a38fc638a9] 2024-04-25T16:40:41.031 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a38fc659c3] 2024-04-25T16:40:41.031 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a38fb3185d] 2024-04-25T16:40:41.031 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a38fb33e8e] 2024-04-25T16:40:41.031 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a38fb34894] 2024-04-25T16:40:41.031 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a38fb1f05b] 2024-04-25T16:40:41.031 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fedd7b17ac6] 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 15: (DispatchQueue::entry()+0xb27) [0x7fedd7b150b9] 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fedd7bffccb] 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 17: (Thread::entry_wrapper()+0x43) [0x7fedd796972d] 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 18: (Thread::_entry_func(void*)+0xd) [0x7fedd7969749] 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fedd650d1ca] 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 20: clone() 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.032 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: -1> 2024-04-25T16:40:40.682+0000 7fedcd166700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fedcd166700 time 2024-04-25T16:40:40.679607+0000 2024-04-25T16:40:41.033 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:40:41.033 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.033 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:41.033 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fedd799aba6] 2024-04-25T16:40:41.033 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fedd799add1] 2024-04-25T16:40:41.033 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55a38fcc326b] 2024-04-25T16:40:41.033 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a38fc62428] 2024-04-25T16:40:41.033 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a38fc638a9] 2024-04-25T16:40:41.034 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a38fc659c3] 2024-04-25T16:40:41.034 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a38fb3185d] 2024-04-25T16:40:41.034 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a38fb33e8e] 2024-04-25T16:40:41.034 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a38fb34894] 2024-04-25T16:40:41.034 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a38fb1f05b] 2024-04-25T16:40:41.034 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fedd7b17ac6] 2024-04-25T16:40:41.034 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 12: (DispatchQueue::entry()+0xb27) [0x7fedd7b150b9] 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fedd7bffccb] 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 14: (Thread::entry_wrapper()+0x43) [0x7fedd796972d] 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 15: (Thread::_entry_func(void*)+0xd) [0x7fedd7969749] 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fedd650d1ca] 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 17: clone() 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 0> 2024-04-25T16:40:40.684+0000 7fedcd166700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: in thread 7fedcd166700 thread_name:ms_dispatch 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:40:41.035 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fedd6517d20] 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2: gsignal() 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 3: abort() 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fedd799aced] 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fedd799add1] 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55a38fcc326b] 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55a38fc62428] 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55a38fc638a9] 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55a38fc659c3] 2024-04-25T16:40:41.036 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55a38fb3185d] 2024-04-25T16:40:41.037 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55a38fb33e8e] 2024-04-25T16:40:41.037 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55a38fb34894] 2024-04-25T16:40:41.037 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55a38fb1f05b] 2024-04-25T16:40:41.037 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fedd7b17ac6] 2024-04-25T16:40:41.037 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 15: (DispatchQueue::entry()+0xb27) [0x7fedd7b150b9] 2024-04-25T16:40:41.037 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fedd7bffccb] 2024-04-25T16:40:41.037 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 17: (Thread::entry_wrapper()+0x43) [0x7fedd796972d] 2024-04-25T16:40:41.038 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 18: (Thread::_entry_func(void*)+0xd) [0x7fedd7969749] 2024-04-25T16:40:41.038 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fedd650d1ca] 2024-04-25T16:40:41.038 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 20: clone() 2024-04-25T16:40:41.038 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:40:41.038 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[65562]: 2024-04-25T16:40:41.038 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:40 smithi173 podman[66538]: 2024-04-25 16:40:40.757330583 +0000 UTC m=+0.025804188 container died 504bd8987956ddfe61f3896b33e12a51218a564c8338e2aa0ed1c4547fe1b044 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T16:40:41.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:41 smithi203 ceph-mon[33115]: pgmap v610: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.2 MiB/s wr, 1.29k op/s 2024-04-25T16:40:42.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:41 smithi098 ceph-mon[29162]: pgmap v610: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.2 MiB/s wr, 1.29k op/s 2024-04-25T16:40:42.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:41 smithi173 ceph-mon[32556]: pgmap v610: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 1.2 MiB/s wr, 1.29k op/s 2024-04-25T16:40:43.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:43 smithi203 ceph-mon[33115]: pgmap v611: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.5 MiB/s wr, 1.33k op/s 2024-04-25T16:40:44.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:43 smithi098 ceph-mon[29162]: pgmap v611: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.5 MiB/s wr, 1.33k op/s 2024-04-25T16:40:44.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:43 smithi173 ceph-mon[32556]: pgmap v611: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 1.5 MiB/s wr, 1.33k op/s 2024-04-25T16:40:44.762 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:44 smithi173 podman[66538]: 2024-04-25 16:40:44.511913177 +0000 UTC m=+3.780386768 container remove 504bd8987956ddfe61f3896b33e12a51218a564c8338e2aa0ed1c4547fe1b044 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T16:40:44.762 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:44 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:40:45.263 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:44 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Failed with result 'exit-code'. 2024-04-25T16:40:45.263 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:44 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 32.991s CPU time. 2024-04-25T16:40:46.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:45 smithi098 ceph-mon[29162]: pgmap v612: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 1.50k op/s 2024-04-25T16:40:46.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:45 smithi173 ceph-mon[32556]: pgmap v612: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 1.50k op/s 2024-04-25T16:40:46.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:45 smithi203 ceph-mon[33115]: pgmap v612: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 1.50k op/s 2024-04-25T16:40:48.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:47 smithi098 ceph-mon[29162]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 1.51k op/s 2024-04-25T16:40:48.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:47 smithi173 ceph-mon[32556]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 1.51k op/s 2024-04-25T16:40:48.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:48.090+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:40:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:48.090+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:40:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:48.090+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:40:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:48.090+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:40:48.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:40:48.090+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:40:48.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:47 smithi203 ceph-mon[33115]: pgmap v613: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 1.51k op/s 2024-04-25T16:40:50.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:49 smithi098 ceph-mon[29162]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 1.51k op/s 2024-04-25T16:40:50.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:49 smithi173 ceph-mon[32556]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 1.51k op/s 2024-04-25T16:40:50.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:49 smithi203 ceph-mon[33115]: pgmap v614: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 2.4 MiB/s wr, 1.51k op/s 2024-04-25T16:40:52.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:51 smithi098 ceph-mon[29162]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 902 op/s 2024-04-25T16:40:52.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:51 smithi173 ceph-mon[32556]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 902 op/s 2024-04-25T16:40:52.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:51 smithi203 ceph-mon[33115]: pgmap v615: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 902 op/s 2024-04-25T16:40:54.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:53 smithi098 ceph-mon[29162]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 309 op/s 2024-04-25T16:40:54.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:53 smithi173 ceph-mon[32556]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 309 op/s 2024-04-25T16:40:54.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:53 smithi203 ceph-mon[33115]: pgmap v616: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 309 op/s 2024-04-25T16:40:55.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:54 smithi098 ceph-mon[29162]: Replacing daemon mds.b as rank 0 with standby daemon mds.h 2024-04-25T16:40:55.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:54 smithi098 ceph-mon[29162]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:55.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:54 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:55.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:54 smithi098 ceph-mon[29162]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T16:40:55.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:54 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:40:55.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:54 smithi173 ceph-mon[32556]: Replacing daemon mds.b as rank 0 with standby daemon mds.h 2024-04-25T16:40:55.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:54 smithi173 ceph-mon[32556]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:55.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:54 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:55.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:54 smithi173 ceph-mon[32556]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T16:40:55.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:54 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:40:55.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:54 smithi203 ceph-mon[33115]: Replacing daemon mds.b as rank 0 with standby daemon mds.h 2024-04-25T16:40:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:54 smithi203 ceph-mon[33115]: MDS daemon mds.b is removed because it is dead or otherwise unavailable. 2024-04-25T16:40:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:54 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:40:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:54 smithi203 ceph-mon[33115]: osdmap e95: 12 total, 12 up, 12 in 2024-04-25T16:40:55.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:54 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-04-25T16:40:55.285 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:40:55.285 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 systemd[1]: Stopped Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:55.285 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 32.991s CPU time. 2024-04-25T16:40:55.285 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 systemd[1]: Starting Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:40:55.285 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 podman[66632]: 2024-04-25 16:40:55.150684249 +0000 UTC m=+0.018881242 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:40:55.286 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 podman[66632]: 2024-04-25 16:40:55.28804457 +0000 UTC m=+0.156241544 container create f14df9f6defc8915e614d874aad5bea811f7baf20669ad19ab6ac1b756033328 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:40:56.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:55 smithi098 ceph-mon[29162]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 986 KiB/s wr, 191 op/s 2024-04-25T16:40:56.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:55 smithi173 ceph-mon[32556]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 986 KiB/s wr, 191 op/s 2024-04-25T16:40:56.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 podman[66632]: 2024-04-25 16:40:55.54931356 +0000 UTC m=+0.417510533 container init f14df9f6defc8915e614d874aad5bea811f7baf20669ad19ab6ac1b756033328 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T16:40:56.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 podman[66632]: 2024-04-25 16:40:55.552057375 +0000 UTC m=+0.420254353 container start f14df9f6defc8915e614d874aad5bea811f7baf20669ad19ab6ac1b756033328 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-25T16:40:56.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 ceph-mds[66650]: -- 172.21.15.173:0/2623760065 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eec0525a40 con 0x55eec056d180 2024-04-25T16:40:56.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: starting mds.b at 2024-04-25T16:40:56.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 bash[66632]: f14df9f6defc8915e614d874aad5bea811f7baf20669ad19ab6ac1b756033328 2024-04-25T16:40:56.014 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:40:55 smithi173 systemd[1]: Started Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:40:56.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:55 smithi203 ceph-mon[33115]: pgmap v617: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 986 KiB/s wr, 191 op/s 2024-04-25T16:40:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:56 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:resolve 2024-04-25T16:40:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:56 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:boot 2024-04-25T16:40:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:56 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:40:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:40:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:40:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:40:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:40:57.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:56 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:40:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:56 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:resolve 2024-04-25T16:40:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:56 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:boot 2024-04-25T16:40:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:56 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:40:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:40:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:40:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:40:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:40:57.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:56 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:40:57.087 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:56 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:resolve 2024-04-25T16:40:57.087 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:56 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:boot 2024-04-25T16:40:57.088 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:56 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:40:57.088 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:40:57.088 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:40:57.088 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:40:57.088 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:40:57.088 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:56 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:40:58.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:57 smithi098 ceph-mon[29162]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 716 B/s wr, 10 op/s 2024-04-25T16:40:58.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:57 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:reconnect 2024-04-25T16:40:58.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:57 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:40:58.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:57 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0409992 2024-04-25T16:40:58.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:57 smithi173 ceph-mon[32556]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 716 B/s wr, 10 op/s 2024-04-25T16:40:58.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:57 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:reconnect 2024-04-25T16:40:58.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:57 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:40:58.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:57 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0409992 2024-04-25T16:40:58.098 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:57 smithi203 ceph-mon[33115]: pgmap v619: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 716 B/s wr, 10 op/s 2024-04-25T16:40:58.098 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:57 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:reconnect 2024-04-25T16:40:58.098 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:57 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:40:58.098 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:57 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0409992 2024-04-25T16:40:59.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:58 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:rejoin 2024-04-25T16:40:59.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:58 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:40:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:58 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:rejoin 2024-04-25T16:40:59.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:58 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:40:59.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:58 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:rejoin 2024-04-25T16:40:59.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:58 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:41:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:59 smithi098 ceph-mon[29162]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 614 B/s wr, 11 op/s 2024-04-25T16:41:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:59 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:clientreplay 2024-04-25T16:41:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:40:59 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:41:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:59 smithi173 ceph-mon[32556]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 614 B/s wr, 11 op/s 2024-04-25T16:41:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:59 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:clientreplay 2024-04-25T16:41:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:40:59 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:41:00.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:59 smithi203 ceph-mon[33115]: pgmap v620: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 614 B/s wr, 11 op/s 2024-04-25T16:41:00.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:59 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:clientreplay 2024-04-25T16:41:00.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:40:59 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:41:01.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:00 smithi098 ceph-mon[29162]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-25T16:41:01.086 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:00 smithi173 ceph-mon[32556]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-25T16:41:01.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:00 smithi203 ceph-mon[33115]: daemon mds.h is now active in filesystem cephfs as rank 0 2024-04-25T16:41:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:01 smithi098 ceph-mon[29162]: pgmap v621: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 614 B/s wr, 17 op/s 2024-04-25T16:41:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:01 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:01 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:41:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:01 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:active 2024-04-25T16:41:02.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:01 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:41:02.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:01 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:01 smithi173 ceph-mon[32556]: pgmap v621: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 614 B/s wr, 17 op/s 2024-04-25T16:41:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:01 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:01 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:41:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:01 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:active 2024-04-25T16:41:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:01 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:41:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:01 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:01 smithi203 ceph-mon[33115]: pgmap v621: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 614 B/s wr, 17 op/s 2024-04-25T16:41:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:01 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:01 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:41:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:01 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/3031685940,v1:172.21.15.173:6837/3031685940] up:active 2024-04-25T16:41:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:01 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:41:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:01 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:03.103 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:03.105+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:03.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:03.105+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:41:03.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:03.105+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:41:03.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:03.105+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:41:03.104 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:03.105+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:41:04.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:03 smithi098 ceph-mon[29162]: pgmap v622: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 450 KiB/s wr, 151 op/s 2024-04-25T16:41:04.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:03 smithi173 ceph-mon[32556]: pgmap v622: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 450 KiB/s wr, 151 op/s 2024-04-25T16:41:04.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:03 smithi203 ceph-mon[33115]: pgmap v622: 129 pgs: 129 active+clean; 1.4 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 450 KiB/s wr, 151 op/s 2024-04-25T16:41:06.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:05 smithi098 ceph-mon[29162]: pgmap v623: 129 pgs: 129 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 931 KiB/s wr, 471 op/s 2024-04-25T16:41:06.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:05 smithi173 ceph-mon[32556]: pgmap v623: 129 pgs: 129 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 931 KiB/s wr, 471 op/s 2024-04-25T16:41:06.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:41:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:41:05.613+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T16:41:06.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:41:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:41:05.613+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:41:06.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:41:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:41:05.613+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:41:06.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:41:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:41:05.613+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T16:41:06.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:41:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:41:05.613+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:41:06.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:05 smithi203 ceph-mon[33115]: pgmap v623: 129 pgs: 129 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 931 KiB/s wr, 471 op/s 2024-04-25T16:41:08.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:07 smithi098 ceph-mon[29162]: pgmap v624: 129 pgs: 129 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 992 KiB/s wr, 497 op/s 2024-04-25T16:41:08.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:07 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:08.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:07 smithi173 ceph-mon[32556]: pgmap v624: 129 pgs: 129 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 992 KiB/s wr, 497 op/s 2024-04-25T16:41:08.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:07 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:07 smithi203 ceph-mon[33115]: pgmap v624: 129 pgs: 129 active+clean; 1.4 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 992 KiB/s wr, 497 op/s 2024-04-25T16:41:08.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:07 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:10.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:09 smithi098 ceph-mon[29162]: pgmap v625: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 709 op/s 2024-04-25T16:41:10.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:09 smithi173 ceph-mon[32556]: pgmap v625: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 709 op/s 2024-04-25T16:41:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:09 smithi203 ceph-mon[33115]: pgmap v625: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 1.7 MiB/s wr, 709 op/s 2024-04-25T16:41:11.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:11 smithi173 ceph-mon[32556]: pgmap v626: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.1 MiB/s wr, 856 op/s 2024-04-25T16:41:11.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:11 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:11 smithi203 ceph-mon[33115]: pgmap v626: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.1 MiB/s wr, 856 op/s 2024-04-25T16:41:12.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:11 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:12.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:11 smithi098 ceph-mon[29162]: pgmap v626: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 2.1 MiB/s wr, 856 op/s 2024-04-25T16:41:12.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:11 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:14.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:13 smithi173 ceph-mon[32556]: pgmap v627: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 889 op/s 2024-04-25T16:41:14.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:13 smithi203 ceph-mon[33115]: pgmap v627: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 889 op/s 2024-04-25T16:41:14.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:13 smithi098 ceph-mon[29162]: pgmap v627: 129 pgs: 129 active+clean; 1.4 GiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 2.2 MiB/s wr, 889 op/s 2024-04-25T16:41:15.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-mon[29162]: pgmap v628: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 1.10k op/s 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd03ab2b700 time 2024-04-25T16:41:15.626522+0000 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd04535fba6] 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd04535fdd1] 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55909aae826b] 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55909aa87428] 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55909aa888a9] 2024-04-25T16:41:15.997 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55909aa8a9c3] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55909a95685d] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55909a958e8e] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55909a959894] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55909a94405b] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd0454dcac6] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 12: (DispatchQueue::entry()+0xb27) [0x7fd0454da0b9] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd0455c4ccb] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 14: (Thread::entry_wrapper()+0x43) [0x7fd04532e72d] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd04532e749] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd043ed21ca] 2024-04-25T16:41:15.998 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 17: clone() 2024-04-25T16:41:15.999 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:15.628+0000 7fd03ab2b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd03ab2b700 time 2024-04-25T16:41:15.626522+0000 2024-04-25T16:41:15.999 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd04535fba6] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd04535fdd1] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55909aae826b] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55909aa87428] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55909aa888a9] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55909aa8a9c3] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55909a95685d] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55909a958e8e] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55909a959894] 2024-04-25T16:41:16.000 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55909a94405b] 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd0454dcac6] 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 12: (DispatchQueue::entry()+0xb27) [0x7fd0454da0b9] 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd0455c4ccb] 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 14: (Thread::entry_wrapper()+0x43) [0x7fd04532e72d] 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd04532e749] 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd043ed21ca] 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 17: clone() 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: *** Caught signal (Aborted) ** 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: in thread 7fd03ab2b700 thread_name:ms_dispatch 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd043edcd20] 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2: gsignal() 2024-04-25T16:41:16.001 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 3: abort() 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd04535fced] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd04535fdd1] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55909aae826b] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55909aa87428] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55909aa888a9] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55909aa8a9c3] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55909a95685d] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55909a958e8e] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55909a959894] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55909a94405b] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd0454dcac6] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 15: (DispatchQueue::entry()+0xb27) [0x7fd0454da0b9] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd0455c4ccb] 2024-04-25T16:41:16.002 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 17: (Thread::entry_wrapper()+0x43) [0x7fd04532e72d] 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd04532e749] 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd043ed21ca] 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 20: clone() 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:15.630+0000 7fd03ab2b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: in thread 7fd03ab2b700 thread_name:ms_dispatch 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd043edcd20] 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2: gsignal() 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 3: abort() 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd04535fced] 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd04535fdd1] 2024-04-25T16:41:16.003 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55909aae826b] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55909aa87428] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55909aa888a9] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55909aa8a9c3] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55909a95685d] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55909a958e8e] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55909a959894] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55909a94405b] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd0454dcac6] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 15: (DispatchQueue::entry()+0xb27) [0x7fd0454da0b9] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd0455c4ccb] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 17: (Thread::entry_wrapper()+0x43) [0x7fd04532e72d] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd04532e749] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd043ed21ca] 2024-04-25T16:41:16.004 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 20: clone() 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: -1> 2024-04-25T16:41:15.628+0000 7fd03ab2b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd03ab2b700 time 2024-04-25T16:41:15.626522+0000 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd04535fba6] 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd04535fdd1] 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55909aae826b] 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55909aa87428] 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55909aa888a9] 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55909aa8a9c3] 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55909a95685d] 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55909a958e8e] 2024-04-25T16:41:16.005 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55909a959894] 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55909a94405b] 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd0454dcac6] 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 12: (DispatchQueue::entry()+0xb27) [0x7fd0454da0b9] 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd0455c4ccb] 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 14: (Thread::entry_wrapper()+0x43) [0x7fd04532e72d] 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd04532e749] 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd043ed21ca] 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 17: clone() 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 0> 2024-04-25T16:41:15.630+0000 7fd03ab2b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: in thread 7fd03ab2b700 thread_name:ms_dispatch 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.006 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd043edcd20] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2: gsignal() 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 3: abort() 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd04535fced] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd04535fdd1] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55909aae826b] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55909aa87428] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55909aa888a9] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55909aa8a9c3] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55909a95685d] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55909a958e8e] 2024-04-25T16:41:16.007 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55909a959894] 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55909a94405b] 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd0454dcac6] 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 15: (DispatchQueue::entry()+0xb27) [0x7fd0454da0b9] 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd0455c4ccb] 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 17: (Thread::entry_wrapper()+0x43) [0x7fd04532e72d] 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd04532e749] 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd043ed21ca] 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 20: clone() 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: -1> 2024-04-25T16:41:15.628+0000 7fd03ab2b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd03ab2b700 time 2024-04-25T16:41:15.626522+0000 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:41:16.008 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.009 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:16.009 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd04535fba6] 2024-04-25T16:41:16.009 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd04535fdd1] 2024-04-25T16:41:16.009 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55909aae826b] 2024-04-25T16:41:16.009 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55909aa87428] 2024-04-25T16:41:16.009 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55909aa888a9] 2024-04-25T16:41:16.009 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55909aa8a9c3] 2024-04-25T16:41:16.009 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55909a95685d] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55909a958e8e] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55909a959894] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55909a94405b] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd0454dcac6] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 12: (DispatchQueue::entry()+0xb27) [0x7fd0454da0b9] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd0455c4ccb] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 14: (Thread::entry_wrapper()+0x43) [0x7fd04532e72d] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd04532e749] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd043ed21ca] 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 17: clone() 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 0> 2024-04-25T16:41:15.630+0000 7fd03ab2b700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:41:16.010 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: in thread 7fd03ab2b700 thread_name:ms_dispatch 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd043edcd20] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2: gsignal() 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 3: abort() 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd04535fced] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd04535fdd1] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55909aae826b] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55909aa87428] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55909aa888a9] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55909aa8a9c3] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55909a95685d] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55909a958e8e] 2024-04-25T16:41:16.011 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55909a959894] 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55909a94405b] 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd0454dcac6] 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 15: (DispatchQueue::entry()+0xb27) [0x7fd0454da0b9] 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd0455c4ccb] 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 17: (Thread::entry_wrapper()+0x43) [0x7fd04532e72d] 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd04532e749] 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd043ed21ca] 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 20: clone() 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[70401]: 2024-04-25T16:41:16.012 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:15 smithi098 podman[88640]: 2024-04-25 16:41:15.698202953 +0000 UTC m=+0.026245750 container died ca2f757698d5afa7f459498c081e0f081289481535cc76e24e372b0c2ea36f1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T16:41:16.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:15 smithi173 ceph-mon[32556]: pgmap v628: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 1.10k op/s 2024-04-25T16:41:16.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:15 smithi203 ceph-mon[33115]: pgmap v628: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 1.10k op/s 2024-04-25T16:41:16.751 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:16 smithi098 podman[88640]: 2024-04-25 16:41:16.477503359 +0000 UTC m=+0.805546201 container remove ca2f757698d5afa7f459498c081e0f081289481535cc76e24e372b0c2ea36f1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:41:16.752 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:16 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:41:16.752 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:16 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:41:16.752 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:16 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Consumed 15.052s CPU time. 2024-04-25T16:41:17.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:16 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:17.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:16 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:17.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:16 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:18.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:18.118+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:18.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:18.119+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:41:18.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:18.119+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:41:18.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:18.119+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:41:18.117 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:18.119+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:41:18.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:17 smithi203 ceph-mon[33115]: pgmap v629: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 898 op/s 2024-04-25T16:41:18.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:17 smithi098 ceph-mon[29162]: pgmap v629: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 898 op/s 2024-04-25T16:41:18.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:17 smithi173 ceph-mon[32556]: pgmap v629: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 898 op/s 2024-04-25T16:41:20.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:19 smithi203 ceph-mon[33115]: pgmap v630: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 938 op/s 2024-04-25T16:41:20.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:19 smithi098 ceph-mon[29162]: pgmap v630: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 938 op/s 2024-04-25T16:41:20.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:19 smithi173 ceph-mon[32556]: pgmap v630: 129 pgs: 129 active+clean; 1.4 GiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 938 op/s 2024-04-25T16:41:22.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:21 smithi203 ceph-mon[33115]: pgmap v631: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.2 MiB/s wr, 826 op/s 2024-04-25T16:41:22.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:21 smithi098 ceph-mon[29162]: pgmap v631: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.2 MiB/s wr, 826 op/s 2024-04-25T16:41:22.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:21 smithi173 ceph-mon[32556]: pgmap v631: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 2.2 MiB/s wr, 826 op/s 2024-04-25T16:41:22.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:22.428+0000 7fdebd7fa640 1 -- 172.21.15.98:0/2834070656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fdea0005850 con 0x7fdec8101760 2024-04-25T16:41:22.427 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:22.428+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fdea0005850 con 0x7fdec8101760 2024-04-25T16:41:22.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:22.433+0000 7fdea77fe640 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fde98005820 con 0x55e32644d4d0 2024-04-25T16:41:22.431 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:22.433+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fde98005820 con 0x55e32644d4d0 2024-04-25T16:41:24.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:23 smithi203 ceph-mon[33115]: pgmap v632: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 677 op/s 2024-04-25T16:41:24.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:23 smithi098 ceph-mon[29162]: pgmap v632: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 677 op/s 2024-04-25T16:41:24.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:23 smithi173 ceph-mon[32556]: pgmap v632: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.8 MiB/s wr, 677 op/s 2024-04-25T16:41:26.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:25 smithi203 ceph-mon[33115]: pgmap v633: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 642 op/s 2024-04-25T16:41:26.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:25 smithi098 ceph-mon[29162]: pgmap v633: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 642 op/s 2024-04-25T16:41:26.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:25 smithi173 ceph-mon[32556]: pgmap v633: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.9 MiB/s wr, 642 op/s 2024-04-25T16:41:27.018 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:26 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:41:27.018 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:26 smithi098 systemd[1]: Stopped Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:41:27.018 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:26 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Consumed 15.052s CPU time. 2024-04-25T16:41:27.018 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:26 smithi098 systemd[1]: Starting Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:41:27.018 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:26 smithi098 podman[88740]: 2024-04-25 16:41:26.890302562 +0000 UTC m=+0.018271597 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:41:27.018 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:27 smithi098 podman[88740]: 2024-04-25 16:41:27.020427017 +0000 UTC m=+0.148396041 container create ab3b17b7be703e895cf80e1c91015f38433ec57b487aac8c0399c2e1766f0b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:41:27.578 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:27 smithi098 podman[88740]: 2024-04-25 16:41:27.281717172 +0000 UTC m=+0.409686197 container init ab3b17b7be703e895cf80e1c91015f38433ec57b487aac8c0399c2e1766f0b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:41:27.578 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:27 smithi098 podman[88740]: 2024-04-25 16:41:27.284476969 +0000 UTC m=+0.412445999 container start ab3b17b7be703e895cf80e1c91015f38433ec57b487aac8c0399c2e1766f0b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T16:41:27.578 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mds[88759]: -- 172.21.15.98:0/1174385260 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562154605a40 con 0x56215464d600 2024-04-25T16:41:27.578 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: starting mds.g at 2024-04-25T16:41:27.578 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:27 smithi098 bash[88740]: ab3b17b7be703e895cf80e1c91015f38433ec57b487aac8c0399c2e1766f0b80 2024-04-25T16:41:27.578 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:41:27 smithi098 systemd[1]: Started Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:41:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: pgmap v634: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 985 KiB/s wr, 321 op/s 2024-04-25T16:41:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: daemon mds.g restarted 2024-04-25T16:41:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T16:41:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:boot 2024-04-25T16:41:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: Standby daemon mds.c assigned to filesystem cephfs as rank 4 2024-04-25T16:41:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,2=l=up:active,3=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:28.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:41:28.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:27 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: pgmap v634: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 985 KiB/s wr, 321 op/s 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: daemon mds.g restarted 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:boot 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: Standby daemon mds.c assigned to filesystem cephfs as rank 4 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,2=l=up:active,3=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:41:28.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:27 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: pgmap v634: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 985 KiB/s wr, 321 op/s 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: daemon mds.g restarted 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: osdmap e96: 12 total, 12 up, 12 in 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:boot 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: Standby daemon mds.c assigned to filesystem cephfs as rank 4 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,2=l=up:active,3=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:41:28.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:27 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:41:29.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:29 smithi203 ceph-mon[33115]: pgmap v636: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 716 KiB/s wr, 151 op/s 2024-04-25T16:41:29.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:29 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:resolve 2024-04-25T16:41:29.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:29 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:41:29.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:29 smithi098 ceph-mon[29162]: pgmap v636: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 716 KiB/s wr, 151 op/s 2024-04-25T16:41:29.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:29 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:resolve 2024-04-25T16:41:29.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:29 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:41:29.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:29 smithi173 ceph-mon[32556]: pgmap v636: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 716 KiB/s wr, 151 op/s 2024-04-25T16:41:29.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:29 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:resolve 2024-04-25T16:41:29.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:29 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:41:30.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:30 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:reconnect 2024-04-25T16:41:30.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:30 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00299993 2024-04-25T16:41:30.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:30 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:41:30.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:30 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:reconnect 2024-04-25T16:41:30.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:30 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00299993 2024-04-25T16:41:30.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:30 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:41:30.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:30 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:reconnect 2024-04-25T16:41:30.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:30 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.00299993 2024-04-25T16:41:30.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:30 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:41:31.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:31 smithi203 ceph-mon[33115]: pgmap v637: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 175 KiB/s wr, 16 op/s 2024-04-25T16:41:31.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:31 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:rejoin 2024-04-25T16:41:31.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:31 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:41:31.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:31 smithi203 ceph-mon[33115]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-25T16:41:31.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:31 smithi098 ceph-mon[29162]: pgmap v637: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 175 KiB/s wr, 16 op/s 2024-04-25T16:41:31.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:31 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:rejoin 2024-04-25T16:41:31.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:31 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:41:31.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:31 smithi098 ceph-mon[29162]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-25T16:41:31.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:31 smithi173 ceph-mon[32556]: pgmap v637: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 175 KiB/s wr, 16 op/s 2024-04-25T16:41:31.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:31 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:rejoin 2024-04-25T16:41:31.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:31 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:41:31.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:31 smithi173 ceph-mon[32556]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-25T16:41:32.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:32 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:32.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:32 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:41:32.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:32 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:active 2024-04-25T16:41:32.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:32 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:41:32.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:32 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:32.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:32 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:32.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:32 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:41:32.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:32 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:active 2024-04-25T16:41:32.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:32 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:41:32.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:32 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:32.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:32 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:41:32.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:32 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:41:32.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:32 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/3458796538,v1:172.21.15.203:6837/3458796538] up:active 2024-04-25T16:41:32.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:32 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:41:32.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:32 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:41:33.132 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:33.134+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:33.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:33.134+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:41:33.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:33.134+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:41:33.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:33.134+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:41:33.133 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:33.134+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:41:33.651 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:33 smithi098 ceph-mon[29162]: pgmap v638: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 197 KiB/s wr, 39 op/s 2024-04-25T16:41:33.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:33 smithi203 ceph-mon[33115]: pgmap v638: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 197 KiB/s wr, 39 op/s 2024-04-25T16:41:33.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:33 smithi173 ceph-mon[32556]: pgmap v638: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 197 KiB/s wr, 39 op/s 2024-04-25T16:41:35.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:35 smithi203 ceph-mon[33115]: pgmap v639: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 632 KiB/s wr, 315 op/s 2024-04-25T16:41:35.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:35 smithi098 ceph-mon[29162]: pgmap v639: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 632 KiB/s wr, 315 op/s 2024-04-25T16:41:35.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:35 smithi173 ceph-mon[32556]: pgmap v639: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 632 KiB/s wr, 315 op/s 2024-04-25T16:41:35.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:41:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:41:35.615+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:41:37.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:37 smithi203 ceph-mon[33115]: pgmap v640: 129 pgs: 129 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 825 KiB/s wr, 392 op/s 2024-04-25T16:41:37.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:37 smithi098 ceph-mon[29162]: pgmap v640: 129 pgs: 129 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 825 KiB/s wr, 392 op/s 2024-04-25T16:41:37.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:37 smithi173 ceph-mon[32556]: pgmap v640: 129 pgs: 129 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 825 KiB/s wr, 392 op/s 2024-04-25T16:41:39.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:39 smithi203 ceph-mon[33115]: pgmap v641: 129 pgs: 129 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 433 op/s 2024-04-25T16:41:39.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:39 smithi098 ceph-mon[29162]: pgmap v641: 129 pgs: 129 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 433 op/s 2024-04-25T16:41:39.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:39 smithi173 ceph-mon[32556]: pgmap v641: 129 pgs: 129 active+clean; 1.5 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.0 MiB/s wr, 433 op/s 2024-04-25T16:41:41.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:41 smithi203 ceph-mon[33115]: pgmap v642: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 575 op/s 2024-04-25T16:41:41.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:41 smithi098 ceph-mon[29162]: pgmap v642: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 575 op/s 2024-04-25T16:41:41.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:41 smithi173 ceph-mon[32556]: pgmap v642: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.7 MiB/s wr, 575 op/s 2024-04-25T16:41:43.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:43 smithi203 ceph-mon[33115]: pgmap v643: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 599 op/s 2024-04-25T16:41:43.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:43 smithi098 ceph-mon[29162]: pgmap v643: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 599 op/s 2024-04-25T16:41:43.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:43 smithi173 ceph-mon[32556]: pgmap v643: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.8 MiB/s wr, 599 op/s 2024-04-25T16:41:45.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:45 smithi203 ceph-mon[33115]: pgmap v644: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 841 op/s 2024-04-25T16:41:45.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:45 smithi098 ceph-mon[29162]: pgmap v644: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 841 op/s 2024-04-25T16:41:45.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:45 smithi173 ceph-mon[32556]: pgmap v644: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 841 op/s 2024-04-25T16:41:47.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:47 smithi173 ceph-mon[32556]: pgmap v645: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 683 op/s 2024-04-25T16:41:47.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:47 smithi203 ceph-mon[33115]: pgmap v645: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 683 op/s 2024-04-25T16:41:47.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:47 smithi098 ceph-mon[29162]: pgmap v645: 129 pgs: 129 active+clean; 1.4 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 683 op/s 2024-04-25T16:41:48.147 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:48.149+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:41:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:48.149+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:41:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:48.149+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:41:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:48.149+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:41:48.148 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:41:48.150+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:41:49.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:49 smithi203 ceph-mon[33115]: pgmap v646: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 2.4 MiB/s wr, 765 op/s 2024-04-25T16:41:49.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:49 smithi098 ceph-mon[29162]: pgmap v646: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 2.4 MiB/s wr, 765 op/s 2024-04-25T16:41:49.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:49 smithi173 ceph-mon[32556]: pgmap v646: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 2.4 MiB/s wr, 765 op/s 2024-04-25T16:41:51.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:51 smithi203 ceph-mon[33115]: pgmap v647: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 947 op/s 2024-04-25T16:41:51.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:51 smithi098 ceph-mon[29162]: pgmap v647: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 947 op/s 2024-04-25T16:41:51.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:51 smithi173 ceph-mon[32556]: pgmap v647: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 947 op/s 2024-04-25T16:41:53.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:53 smithi203 ceph-mon[33115]: pgmap v648: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 2.8 MiB/s wr, 779 op/s 2024-04-25T16:41:53.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:53 smithi098 ceph-mon[29162]: pgmap v648: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 2.8 MiB/s wr, 779 op/s 2024-04-25T16:41:53.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:53 smithi173 ceph-mon[32556]: pgmap v648: 129 pgs: 129 active+clean; 1.5 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 2.8 MiB/s wr, 779 op/s 2024-04-25T16:41:55.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:55 smithi203 ceph-mon[33115]: pgmap v649: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 968 op/s 2024-04-25T16:41:55.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:55 smithi098 ceph-mon[29162]: pgmap v649: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 968 op/s 2024-04-25T16:41:55.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:55 smithi173 ceph-mon[32556]: pgmap v649: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 968 op/s 2024-04-25T16:41:57.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:57 smithi203 ceph-mon[33115]: pgmap v650: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 3.6 MiB/s wr, 781 op/s 2024-04-25T16:41:57.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:57 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:41:57.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:57 smithi098 ceph-mon[29162]: pgmap v650: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 3.6 MiB/s wr, 781 op/s 2024-04-25T16:41:57.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:57 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:41:57.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:57 smithi173 ceph-mon[32556]: pgmap v650: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 3.6 MiB/s wr, 781 op/s 2024-04-25T16:41:57.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:57 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:41:58.663 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:41:58.663 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:41:58.663 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:41:58.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:41:58.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:41:58.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:41:58.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:41:58.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:41:58.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:41:59.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:41:59 smithi203 ceph-mon[33115]: pgmap v651: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 821 op/s 2024-04-25T16:42:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:41:59 smithi098 ceph-mon[29162]: pgmap v651: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 821 op/s 2024-04-25T16:42:00.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:41:59 smithi173 ceph-mon[32556]: pgmap v651: 129 pgs: 129 active+clean; 1.5 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 821 op/s 2024-04-25T16:42:01.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:01 smithi203 ceph-mon[33115]: pgmap v652: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 894 op/s 2024-04-25T16:42:01.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:01 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:01 smithi098 ceph-mon[29162]: pgmap v652: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 894 op/s 2024-04-25T16:42:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:01 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:02.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:01 smithi173 ceph-mon[32556]: pgmap v652: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.2 MiB/s wr, 894 op/s 2024-04-25T16:42:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:01 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:03.163 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:03.165+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:03.166+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:42:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:03.166+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:42:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:03.166+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:42:03.164 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:03.166+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:42:03.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:03 smithi203 ceph-mon[33115]: pgmap v653: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 656 op/s 2024-04-25T16:42:04.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:03 smithi098 ceph-mon[29162]: pgmap v653: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 656 op/s 2024-04-25T16:42:04.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:03 smithi173 ceph-mon[32556]: pgmap v653: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 656 op/s 2024-04-25T16:42:05.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:05 smithi203 ceph-mon[33115]: pgmap v654: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 759 op/s 2024-04-25T16:42:06.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:05 smithi098 ceph-mon[29162]: pgmap v654: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 759 op/s 2024-04-25T16:42:06.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:05 smithi173 ceph-mon[32556]: pgmap v654: 129 pgs: 129 active+clean; 1.5 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 5.1 MiB/s wr, 759 op/s 2024-04-25T16:42:07.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:07 smithi203 ceph-mon[33115]: pgmap v655: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 570 op/s 2024-04-25T16:42:08.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:07 smithi098 ceph-mon[29162]: pgmap v655: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 570 op/s 2024-04-25T16:42:08.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:07 smithi173 ceph-mon[32556]: pgmap v655: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.9 MiB/s wr, 570 op/s 2024-04-25T16:42:09.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:09 smithi203 ceph-mon[33115]: pgmap v656: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 578 op/s 2024-04-25T16:42:10.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:09 smithi098 ceph-mon[29162]: pgmap v656: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 578 op/s 2024-04-25T16:42:10.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:09 smithi173 ceph-mon[32556]: pgmap v656: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 4.3 MiB/s wr, 578 op/s 2024-04-25T16:42:11.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:11 smithi203 ceph-mon[33115]: pgmap v657: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 564 op/s 2024-04-25T16:42:12.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:11 smithi098 ceph-mon[29162]: pgmap v657: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 564 op/s 2024-04-25T16:42:12.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:11 smithi173 ceph-mon[32556]: pgmap v657: 129 pgs: 129 active+clean; 1.5 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.8 MiB/s wr, 564 op/s 2024-04-25T16:42:13.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:13 smithi203 ceph-mon[33115]: pgmap v658: 129 pgs: 129 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-25T16:42:14.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:13 smithi098 ceph-mon[29162]: pgmap v658: 129 pgs: 129 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-25T16:42:14.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:13 smithi173 ceph-mon[32556]: pgmap v658: 129 pgs: 129 active+clean; 1.6 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 3.7 MiB/s wr, 355 op/s 2024-04-25T16:42:15.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:15 smithi203 ceph-mon[33115]: pgmap v659: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 545 op/s 2024-04-25T16:42:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:15 smithi098 ceph-mon[29162]: pgmap v659: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 545 op/s 2024-04-25T16:42:16.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:15 smithi173 ceph-mon[32556]: pgmap v659: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 5.4 MiB/s wr, 545 op/s 2024-04-25T16:42:17.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:17 smithi203 ceph-mon[33115]: pgmap v660: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 471 op/s 2024-04-25T16:42:18.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:17 smithi098 ceph-mon[29162]: pgmap v660: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 471 op/s 2024-04-25T16:42:18.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:17 smithi173 ceph-mon[32556]: pgmap v660: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.3 MiB/s wr, 471 op/s 2024-04-25T16:42:18.179 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:18.181+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:18.181+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:42:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:18.181+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:42:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:18.181+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:42:18.180 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:18.181+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:42:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:19 smithi203 ceph-mon[33115]: pgmap v661: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 565 op/s 2024-04-25T16:42:20.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:19 smithi098 ceph-mon[29162]: pgmap v661: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 565 op/s 2024-04-25T16:42:20.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:19 smithi173 ceph-mon[32556]: pgmap v661: 129 pgs: 129 active+clean; 1.6 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.8 MiB/s wr, 565 op/s 2024-04-25T16:42:21.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:21 smithi203 ceph-mon[33115]: pgmap v662: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 727 op/s 2024-04-25T16:42:22.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:21 smithi098 ceph-mon[29162]: pgmap v662: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 727 op/s 2024-04-25T16:42:22.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:21 smithi173 ceph-mon[32556]: pgmap v662: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.9 MiB/s wr, 727 op/s 2024-04-25T16:42:23.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:23 smithi203 ceph-mon[33115]: pgmap v663: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 686 op/s 2024-04-25T16:42:24.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:23 smithi098 ceph-mon[29162]: pgmap v663: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 686 op/s 2024-04-25T16:42:24.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:23 smithi173 ceph-mon[32556]: pgmap v663: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.1 MiB/s wr, 686 op/s 2024-04-25T16:42:25.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:25 smithi203 ceph-mon[33115]: pgmap v664: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 973 op/s 2024-04-25T16:42:25.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:25 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:26.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:25 smithi098 ceph-mon[29162]: pgmap v664: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 973 op/s 2024-04-25T16:42:26.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:25 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:26.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:25 smithi173 ceph-mon[32556]: pgmap v664: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 973 op/s 2024-04-25T16:42:26.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:25 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:27.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:27 smithi203 ceph-mon[33115]: pgmap v665: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.9 MiB/s wr, 833 op/s 2024-04-25T16:42:28.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:27 smithi098 ceph-mon[29162]: pgmap v665: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.9 MiB/s wr, 833 op/s 2024-04-25T16:42:28.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:27 smithi173 ceph-mon[32556]: pgmap v665: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 2.9 MiB/s wr, 833 op/s 2024-04-25T16:42:29.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:29 smithi203 ceph-mon[33115]: pgmap v666: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 3.4 MiB/s wr, 928 op/s 2024-04-25T16:42:30.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:29 smithi098 ceph-mon[29162]: pgmap v666: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 3.4 MiB/s wr, 928 op/s 2024-04-25T16:42:30.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:29 smithi173 ceph-mon[32556]: pgmap v666: 129 pgs: 129 active+clean; 1.6 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 3.4 MiB/s wr, 928 op/s 2024-04-25T16:42:30.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:30 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:31.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:30 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:31.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:30 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:42:31.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:31 smithi203 ceph-mon[33115]: pgmap v667: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 1.09k op/s 2024-04-25T16:42:32.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:31 smithi098 ceph-mon[29162]: pgmap v667: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 1.09k op/s 2024-04-25T16:42:32.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:31 smithi173 ceph-mon[32556]: pgmap v667: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 4.2 MiB/s wr, 1.09k op/s 2024-04-25T16:42:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:33.196+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:33.194 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:33.196+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:42:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:33.196+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:42:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:33.196+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:42:33.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:33.196+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:42:33.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:33 smithi203 ceph-mon[33115]: pgmap v668: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 884 op/s 2024-04-25T16:42:34.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:33 smithi098 ceph-mon[29162]: pgmap v668: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 884 op/s 2024-04-25T16:42:34.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:33 smithi173 ceph-mon[32556]: pgmap v668: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 884 op/s 2024-04-25T16:42:35.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:35 smithi203 ceph-mon[33115]: pgmap v669: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 1.09k op/s 2024-04-25T16:42:36.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:35 smithi098 ceph-mon[29162]: pgmap v669: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 1.09k op/s 2024-04-25T16:42:36.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:35 smithi173 ceph-mon[32556]: pgmap v669: 129 pgs: 129 active+clean; 1.6 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 4.0 MiB/s wr, 1.09k op/s 2024-04-25T16:42:37.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:37 smithi203 ceph-mon[33115]: pgmap v670: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 856 op/s 2024-04-25T16:42:38.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:37 smithi098 ceph-mon[29162]: pgmap v670: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 856 op/s 2024-04-25T16:42:38.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:37 smithi173 ceph-mon[32556]: pgmap v670: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.6 MiB/s wr, 856 op/s 2024-04-25T16:42:39.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:39 smithi203 ceph-mon[33115]: pgmap v671: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 885 op/s 2024-04-25T16:42:40.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:39 smithi098 ceph-mon[29162]: pgmap v671: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 885 op/s 2024-04-25T16:42:40.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:39 smithi173 ceph-mon[32556]: pgmap v671: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.5 MiB/s wr, 885 op/s 2024-04-25T16:42:41.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:41 smithi203 ceph-mon[33115]: pgmap v672: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 916 op/s 2024-04-25T16:42:42.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:41 smithi098 ceph-mon[29162]: pgmap v672: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 916 op/s 2024-04-25T16:42:42.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:41 smithi173 ceph-mon[32556]: pgmap v672: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.6 MiB/s wr, 916 op/s 2024-04-25T16:42:43.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:43 smithi203 ceph-mon[33115]: pgmap v673: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 656 op/s 2024-04-25T16:42:44.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:43 smithi098 ceph-mon[29162]: pgmap v673: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 656 op/s 2024-04-25T16:42:44.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:43 smithi173 ceph-mon[32556]: pgmap v673: 129 pgs: 129 active+clean; 1.6 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 2.1 MiB/s wr, 656 op/s 2024-04-25T16:42:45.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:45 smithi203 ceph-mon[33115]: pgmap v674: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 829 op/s 2024-04-25T16:42:46.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:45 smithi098 ceph-mon[29162]: pgmap v674: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 829 op/s 2024-04-25T16:42:46.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:45 smithi173 ceph-mon[32556]: pgmap v674: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 829 op/s 2024-04-25T16:42:47.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:47 smithi203 ceph-mon[33115]: pgmap v675: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.1 MiB/s wr, 597 op/s 2024-04-25T16:42:48.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:47 smithi098 ceph-mon[29162]: pgmap v675: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.1 MiB/s wr, 597 op/s 2024-04-25T16:42:48.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:47 smithi173 ceph-mon[32556]: pgmap v675: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.1 MiB/s wr, 597 op/s 2024-04-25T16:42:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:48.211+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:42:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:48.211+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:42:48.209 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:48.211+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:42:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:48.211+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:42:48.210 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:42:48.211+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:42:48.580 INFO:tasks.workunit.client.0.smithi098.stderr:+ cd linux-6.5.11 2024-04-25T16:42:48.581 INFO:tasks.workunit.client.0.smithi098.stderr:+ make defconfig 2024-04-25T16:42:49.057 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/basic/fixdep 2024-04-25T16:42:49.335 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/conf.o 2024-04-25T16:42:49.920 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/confdata.o 2024-04-25T16:42:49.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:49 smithi203 ceph-mon[33115]: pgmap v676: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 594 op/s 2024-04-25T16:42:50.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:49 smithi098 ceph-mon[29162]: pgmap v676: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 594 op/s 2024-04-25T16:42:50.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:49 smithi173 ceph-mon[32556]: pgmap v676: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.6 MiB/s wr, 594 op/s 2024-04-25T16:42:50.219 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/expr.o 2024-04-25T16:42:50.459 INFO:tasks.workunit.client.0.smithi098.stdout: LEX scripts/kconfig/lexer.lex.c 2024-04-25T16:42:50.588 INFO:tasks.workunit.client.0.smithi098.stdout: YACC scripts/kconfig/parser.tab.[ch] 2024-04-25T16:42:50.684 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/lexer.lex.o 2024-04-25T16:42:51.158 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/menu.o 2024-04-25T16:42:51.363 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/parser.tab.o 2024-04-25T16:42:51.736 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/preprocess.o 2024-04-25T16:42:51.873 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/symbol.o 2024-04-25T16:42:51.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:51 smithi203 ceph-mon[33115]: pgmap v677: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 604 op/s 2024-04-25T16:42:52.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:51 smithi098 ceph-mon[29162]: pgmap v677: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 604 op/s 2024-04-25T16:42:52.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:51 smithi173 ceph-mon[32556]: pgmap v677: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 604 op/s 2024-04-25T16:42:52.251 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kconfig/util.o 2024-04-25T16:42:52.332 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTLD scripts/kconfig/conf 2024-04-25T16:42:52.417 INFO:tasks.workunit.client.0.smithi098.stdout:*** Default configuration is based on 'x86_64_defconfig' 2024-04-25T16:42:53.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:53 smithi203 ceph-mon[33115]: pgmap v678: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 444 op/s 2024-04-25T16:42:54.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:53 smithi098 ceph-mon[29162]: pgmap v678: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 444 op/s 2024-04-25T16:42:54.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:53 smithi173 ceph-mon[32556]: pgmap v678: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 444 op/s 2024-04-25T16:42:56.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:55 smithi098 ceph-mon[29162]: pgmap v679: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 670 op/s 2024-04-25T16:42:56.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:55 smithi173 ceph-mon[32556]: pgmap v679: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 670 op/s 2024-04-25T16:42:56.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:55 smithi203 ceph-mon[33115]: pgmap v679: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.4 MiB/s wr, 670 op/s 2024-04-25T16:42:57.989 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:57 smithi203 ceph-mon[33115]: pgmap v680: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 1.5 MiB/s wr, 522 op/s 2024-04-25T16:42:58.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:57 smithi098 ceph-mon[29162]: pgmap v680: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 1.5 MiB/s wr, 522 op/s 2024-04-25T16:42:58.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:57 smithi173 ceph-mon[32556]: pgmap v680: 129 pgs: 129 active+clean; 1.6 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 1.5 MiB/s wr, 522 op/s 2024-04-25T16:42:58.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:58 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:42:59.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:58 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:42:59.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:58 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:43:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:59 smithi098 ceph-mon[29162]: pgmap v681: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.4 MiB/s wr, 577 op/s 2024-04-25T16:43:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:42:59 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:59 smithi173 ceph-mon[32556]: pgmap v681: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.4 MiB/s wr, 577 op/s 2024-04-25T16:43:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:42:59 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:59 smithi203 ceph-mon[33115]: pgmap v681: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 1.4 MiB/s wr, 577 op/s 2024-04-25T16:43:00.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:00.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:42:59 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:01 smithi098 ceph-mon[29162]: pgmap v682: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 864 KiB/s wr, 593 op/s 2024-04-25T16:43:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:01 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:02.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:01 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:02.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:01 smithi173 ceph-mon[32556]: pgmap v682: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 864 KiB/s wr, 593 op/s 2024-04-25T16:43:02.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:02.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:01 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:02.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:01 smithi203 ceph-mon[33115]: pgmap v682: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 864 KiB/s wr, 593 op/s 2024-04-25T16:43:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:01 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:02.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:01 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:02.999 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:43:03.000 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:43:03.000 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:03.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:43:03.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:43:03.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:03.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:43:03.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:43:03.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:43:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:03.226+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:03.226+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:43:03.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:03.226+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:43:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:03.226+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:43:03.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:03.226+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:43:03.362 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f67243c2700 time 2024-04-25T16:43:03.001414+0000 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f672ebf6ba6] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f672ebf6dd1] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x564579c7626b] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564579c15428] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564579c168a9] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564579c189c3] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564579ae485d] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564579ae6e8e] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564579ae7894] 2024-04-25T16:43:03.363 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564579ad205b] 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f672ed73ac6] 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 12: (DispatchQueue::entry()+0xb27) [0x7f672ed710b9] 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f672ee5bccb] 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 14: (Thread::entry_wrapper()+0x43) [0x7f672ebc572d] 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 15: (Thread::_entry_func(void*)+0xd) [0x7f672ebc5749] 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f672d7691ca] 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 17: clone() 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: *** Caught signal (Aborted) ** 2024-04-25T16:43:03.364 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: in thread 7f67243c2700 thread_name:ms_dispatch 2024-04-25T16:43:03.365 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.003+0000 7f67243c2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f67243c2700 time 2024-04-25T16:43:03.001414+0000 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f672ebf6ba6] 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f672ebf6dd1] 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x564579c7626b] 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564579c15428] 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564579c168a9] 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564579c189c3] 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564579ae485d] 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564579ae6e8e] 2024-04-25T16:43:03.366 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564579ae7894] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564579ad205b] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f672ed73ac6] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 12: (DispatchQueue::entry()+0xb27) [0x7f672ed710b9] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f672ee5bccb] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 14: (Thread::entry_wrapper()+0x43) [0x7f672ebc572d] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 15: (Thread::_entry_func(void*)+0xd) [0x7f672ebc5749] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f672d7691ca] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 17: clone() 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f672d773d20] 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2: gsignal() 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 3: abort() 2024-04-25T16:43:03.367 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f672ebf6ced] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f672ebf6dd1] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x564579c7626b] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564579c15428] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564579c168a9] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564579c189c3] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564579ae485d] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564579ae6e8e] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564579ae7894] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564579ad205b] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f672ed73ac6] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 15: (DispatchQueue::entry()+0xb27) [0x7f672ed710b9] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f672ee5bccb] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 17: (Thread::entry_wrapper()+0x43) [0x7f672ebc572d] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 18: (Thread::_entry_func(void*)+0xd) [0x7f672ebc5749] 2024-04-25T16:43:03.368 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f672d7691ca] 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 20: clone() 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.006+0000 7f67243c2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: in thread 7f67243c2700 thread_name:ms_dispatch 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f672d773d20] 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2: gsignal() 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 3: abort() 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f672ebf6ced] 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f672ebf6dd1] 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x564579c7626b] 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564579c15428] 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564579c168a9] 2024-04-25T16:43:03.369 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564579c189c3] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564579ae485d] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564579ae6e8e] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564579ae7894] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564579ad205b] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f672ed73ac6] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 15: (DispatchQueue::entry()+0xb27) [0x7f672ed710b9] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f672ee5bccb] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 17: (Thread::entry_wrapper()+0x43) [0x7f672ebc572d] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 18: (Thread::_entry_func(void*)+0xd) [0x7f672ebc5749] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f672d7691ca] 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 20: clone() 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.370 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: -1> 2024-04-25T16:43:03.003+0000 7f67243c2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f67243c2700 time 2024-04-25T16:43:03.001414+0000 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f672ebf6ba6] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f672ebf6dd1] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x564579c7626b] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564579c15428] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564579c168a9] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564579c189c3] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564579ae485d] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564579ae6e8e] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564579ae7894] 2024-04-25T16:43:03.371 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564579ad205b] 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f672ed73ac6] 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 12: (DispatchQueue::entry()+0xb27) [0x7f672ed710b9] 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f672ee5bccb] 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 14: (Thread::entry_wrapper()+0x43) [0x7f672ebc572d] 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 15: (Thread::_entry_func(void*)+0xd) [0x7f672ebc5749] 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f672d7691ca] 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 17: clone() 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 0> 2024-04-25T16:43:03.006+0000 7f67243c2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: in thread 7f67243c2700 thread_name:ms_dispatch 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f672d773d20] 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2: gsignal() 2024-04-25T16:43:03.372 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 3: abort() 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f672ebf6ced] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f672ebf6dd1] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x564579c7626b] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564579c15428] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564579c168a9] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564579c189c3] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564579ae485d] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564579ae6e8e] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564579ae7894] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564579ad205b] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f672ed73ac6] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 15: (DispatchQueue::entry()+0xb27) [0x7f672ed710b9] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f672ee5bccb] 2024-04-25T16:43:03.373 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 17: (Thread::entry_wrapper()+0x43) [0x7f672ebc572d] 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 18: (Thread::_entry_func(void*)+0xd) [0x7f672ebc5749] 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f672d7691ca] 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 20: clone() 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: -1> 2024-04-25T16:43:03.003+0000 7f67243c2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f67243c2700 time 2024-04-25T16:43:03.001414+0000 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f672ebf6ba6] 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f672ebf6dd1] 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x564579c7626b] 2024-04-25T16:43:03.374 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564579c15428] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564579c168a9] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564579c189c3] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564579ae485d] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564579ae6e8e] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564579ae7894] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564579ad205b] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f672ed73ac6] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 12: (DispatchQueue::entry()+0xb27) [0x7f672ed710b9] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f672ee5bccb] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 14: (Thread::entry_wrapper()+0x43) [0x7f672ebc572d] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 15: (Thread::_entry_func(void*)+0xd) [0x7f672ebc5749] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f672d7691ca] 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 17: clone() 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.375 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 0> 2024-04-25T16:43:03.006+0000 7f67243c2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: in thread 7f67243c2700 thread_name:ms_dispatch 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f672d773d20] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2: gsignal() 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 3: abort() 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f672ebf6ced] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f672ebf6dd1] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x564579c7626b] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x564579c15428] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x564579c168a9] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x564579c189c3] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x564579ae485d] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x564579ae6e8e] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x564579ae7894] 2024-04-25T16:43:03.376 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x564579ad205b] 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f672ed73ac6] 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 15: (DispatchQueue::entry()+0xb27) [0x7f672ed710b9] 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f672ee5bccb] 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 17: (Thread::entry_wrapper()+0x43) [0x7f672ebc572d] 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 18: (Thread::_entry_func(void*)+0xd) [0x7f672ebc5749] 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f672d7691ca] 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 20: clone() 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[63560]: 2024-04-25T16:43:03.377 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 podman[69260]: 2024-04-25 16:43:03.07138486 +0000 UTC m=+0.020441346 container died eb5325b0a0ffa330c1a4c214ade5b9b0cb728e2d5d6447bfc48cd958fc4d6043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:43:03.725 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd961b97700 time 2024-04-25T16:43:03.359038+0000 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd96c3cbba6] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd96c3cbdd1] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5582c4d2026b] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5582c4cbf428] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5582c4cc08a9] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5582c4cc29c3] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5582c4b8e85d] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5582c4b90e8e] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5582c4b91894] 2024-04-25T16:43:03.726 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5582c4b7c05b] 2024-04-25T16:43:03.727 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd96c548ac6] 2024-04-25T16:43:03.727 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 12: (DispatchQueue::entry()+0xb27) [0x7fd96c5460b9] 2024-04-25T16:43:03.727 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd96c630ccb] 2024-04-25T16:43:03.727 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 14: (Thread::entry_wrapper()+0x43) [0x7fd96c39a72d] 2024-04-25T16:43:03.727 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd96c39a749] 2024-04-25T16:43:03.727 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd96af3e1ca] 2024-04-25T16:43:03.727 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 17: clone() 2024-04-25T16:43:03.727 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.361+0000 7fd961b97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd961b97700 time 2024-04-25T16:43:03.359038+0000 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd96c3cbba6] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd96c3cbdd1] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5582c4d2026b] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5582c4cbf428] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5582c4cc08a9] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5582c4cc29c3] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5582c4b8e85d] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5582c4b90e8e] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5582c4b91894] 2024-04-25T16:43:03.728 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5582c4b7c05b] 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd96c548ac6] 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 12: (DispatchQueue::entry()+0xb27) [0x7fd96c5460b9] 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd96c630ccb] 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 14: (Thread::entry_wrapper()+0x43) [0x7fd96c39a72d] 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd96c39a749] 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd96af3e1ca] 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 17: clone() 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: *** Caught signal (Aborted) ** 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: in thread 7fd961b97700 thread_name:ms_dispatch 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd96af48d20] 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2: gsignal() 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 3: abort() 2024-04-25T16:43:03.729 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd96c3cbced] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd96c3cbdd1] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5582c4d2026b] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5582c4cbf428] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5582c4cc08a9] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5582c4cc29c3] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5582c4b8e85d] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5582c4b90e8e] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5582c4b91894] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5582c4b7c05b] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd96c548ac6] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 15: (DispatchQueue::entry()+0xb27) [0x7fd96c5460b9] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd96c630ccb] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 17: (Thread::entry_wrapper()+0x43) [0x7fd96c39a72d] 2024-04-25T16:43:03.730 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd96c39a749] 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd96af3e1ca] 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 20: clone() 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.364+0000 7fd961b97700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: in thread 7fd961b97700 thread_name:ms_dispatch 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd96af48d20] 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2: gsignal() 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 3: abort() 2024-04-25T16:43:03.731 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd96c3cbced] 2024-04-25T16:43:03.732 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd96c3cbdd1] 2024-04-25T16:43:03.732 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5582c4d2026b] 2024-04-25T16:43:03.732 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5582c4cbf428] 2024-04-25T16:43:03.732 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5582c4cc08a9] 2024-04-25T16:43:03.732 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5582c4cc29c3] 2024-04-25T16:43:03.732 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5582c4b8e85d] 2024-04-25T16:43:03.732 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5582c4b90e8e] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5582c4b91894] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5582c4b7c05b] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd96c548ac6] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 15: (DispatchQueue::entry()+0xb27) [0x7fd96c5460b9] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd96c630ccb] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 17: (Thread::entry_wrapper()+0x43) [0x7fd96c39a72d] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd96c39a749] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd96af3e1ca] 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 20: clone() 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: -1> 2024-04-25T16:43:03.361+0000 7fd961b97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd961b97700 time 2024-04-25T16:43:03.359038+0000 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:03.733 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd96c3cbba6] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd96c3cbdd1] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5582c4d2026b] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5582c4cbf428] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5582c4cc08a9] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5582c4cc29c3] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5582c4b8e85d] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5582c4b90e8e] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5582c4b91894] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5582c4b7c05b] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd96c548ac6] 2024-04-25T16:43:03.734 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 12: (DispatchQueue::entry()+0xb27) [0x7fd96c5460b9] 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd96c630ccb] 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 14: (Thread::entry_wrapper()+0x43) [0x7fd96c39a72d] 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd96c39a749] 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd96af3e1ca] 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 17: clone() 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 0> 2024-04-25T16:43:03.364+0000 7fd961b97700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: in thread 7fd961b97700 thread_name:ms_dispatch 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd96af48d20] 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2: gsignal() 2024-04-25T16:43:03.735 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 3: abort() 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd96c3cbced] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd96c3cbdd1] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5582c4d2026b] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5582c4cbf428] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5582c4cc08a9] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5582c4cc29c3] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5582c4b8e85d] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5582c4b90e8e] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5582c4b91894] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5582c4b7c05b] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd96c548ac6] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 15: (DispatchQueue::entry()+0xb27) [0x7fd96c5460b9] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd96c630ccb] 2024-04-25T16:43:03.736 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 17: (Thread::entry_wrapper()+0x43) [0x7fd96c39a72d] 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd96c39a749] 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd96af3e1ca] 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 20: clone() 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: -1> 2024-04-25T16:43:03.361+0000 7fd961b97700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fd961b97700 time 2024-04-25T16:43:03.359038+0000 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fd96c3cbba6] 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd96c3cbdd1] 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5582c4d2026b] 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5582c4cbf428] 2024-04-25T16:43:03.737 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5582c4cc08a9] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5582c4cc29c3] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5582c4b8e85d] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5582c4b90e8e] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5582c4b91894] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5582c4b7c05b] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd96c548ac6] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 12: (DispatchQueue::entry()+0xb27) [0x7fd96c5460b9] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd96c630ccb] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 14: (Thread::entry_wrapper()+0x43) [0x7fd96c39a72d] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 15: (Thread::_entry_func(void*)+0xd) [0x7fd96c39a749] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fd96af3e1ca] 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 17: clone() 2024-04-25T16:43:03.738 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 0> 2024-04-25T16:43:03.364+0000 7fd961b97700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: in thread 7fd961b97700 thread_name:ms_dispatch 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd96af48d20] 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2: gsignal() 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 3: abort() 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fd96c3cbced] 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fd96c3cbdd1] 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5582c4d2026b] 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5582c4cbf428] 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5582c4cc08a9] 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5582c4cc29c3] 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5582c4b8e85d] 2024-04-25T16:43:03.739 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5582c4b90e8e] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5582c4b91894] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5582c4b7c05b] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fd96c548ac6] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 15: (DispatchQueue::entry()+0xb27) [0x7fd96c5460b9] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd96c630ccb] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 17: (Thread::entry_wrapper()+0x43) [0x7fd96c39a72d] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 18: (Thread::_entry_func(void*)+0xd) [0x7fd96c39a749] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fd96af3e1ca] 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 20: clone() 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[61854]: 2024-04-25T16:43:03.740 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:03 smithi173 podman[67514]: 2024-04-25 16:43:03.429412554 +0000 UTC m=+0.023858988 container died c50286901662b35cbd3702a2ba6696e4d7ac154791e1e29ef2c63d4fa5a7e133 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-04-25T16:43:04.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:03 smithi173 ceph-mon[32556]: pgmap v683: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 493 KiB/s wr, 502 op/s 2024-04-25T16:43:04.187 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 podman[69260]: 2024-04-25 16:43:03.776332273 +0000 UTC m=+0.725388768 container remove eb5325b0a0ffa330c1a4c214ade5b9b0cb728e2d5d6447bfc48cd958fc4d6043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-04-25T16:43:04.187 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:03 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:03 smithi203 ceph-mon[33115]: pgmap v683: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 493 KiB/s wr, 502 op/s 2024-04-25T16:43:04.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:03 smithi098 ceph-mon[29162]: pgmap v683: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 493 KiB/s wr, 502 op/s 2024-04-25T16:43:04.513 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:04 smithi173 podman[67514]: 2024-04-25 16:43:04.147963945 +0000 UTC m=+0.742410380 container remove c50286901662b35cbd3702a2ba6696e4d7ac154791e1e29ef2c63d4fa5a7e133 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-25T16:43:04.513 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:04 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:04.686 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:04 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:43:04.687 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:04 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Consumed 7.796s CPU time. 2024-04-25T16:43:05.013 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:04 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:43:05.013 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:04 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Consumed 7.839s CPU time. 2024-04-25T16:43:05.590 INFO:tasks.workunit.client.0.smithi098.stdout:# 2024-04-25T16:43:05.590 INFO:tasks.workunit.client.0.smithi098.stdout:# configuration written to .config 2024-04-25T16:43:05.590 INFO:tasks.workunit.client.0.smithi098.stdout:# 2024-04-25T16:43:05.599 INFO:tasks.workunit.client.0.smithi098.stderr:++ grep -c processor /proc/cpuinfo 2024-04-25T16:43:05.600 INFO:tasks.workunit.client.0.smithi098.stderr:+ make -j8 2024-04-25T16:43:06.168 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:05 smithi098 ceph-mon[29162]: pgmap v684: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 795 KiB/s wr, 563 op/s 2024-04-25T16:43:06.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:05 smithi203 ceph-mon[33115]: pgmap v684: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 795 KiB/s wr, 563 op/s 2024-04-25T16:43:06.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:05 smithi173 ceph-mon[32556]: pgmap v684: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 795 KiB/s wr, 563 op/s 2024-04-25T16:43:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:07 smithi203 ceph-mon[33115]: pgmap v685: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 453 KiB/s wr, 369 op/s 2024-04-25T16:43:08.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:07 smithi098 ceph-mon[29162]: pgmap v685: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 453 KiB/s wr, 369 op/s 2024-04-25T16:43:08.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:07 smithi173 ceph-mon[32556]: pgmap v685: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 453 KiB/s wr, 369 op/s 2024-04-25T16:43:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:09 smithi203 ceph-mon[33115]: pgmap v686: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 499 KiB/s wr, 427 op/s 2024-04-25T16:43:10.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:09 smithi098 ceph-mon[29162]: pgmap v686: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 499 KiB/s wr, 427 op/s 2024-04-25T16:43:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:09 smithi173 ceph-mon[32556]: pgmap v686: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 499 KiB/s wr, 427 op/s 2024-04-25T16:43:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:11 smithi203 ceph-mon[33115]: pgmap v687: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 746 KiB/s wr, 475 op/s 2024-04-25T16:43:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:11 smithi098 ceph-mon[29162]: pgmap v687: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 746 KiB/s wr, 475 op/s 2024-04-25T16:43:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:11 smithi173 ceph-mon[32556]: pgmap v687: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 746 KiB/s wr, 475 op/s 2024-04-25T16:43:14.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:13 smithi203 ceph-mon[33115]: pgmap v688: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 687 KiB/s wr, 410 op/s 2024-04-25T16:43:14.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:13 smithi098 ceph-mon[29162]: pgmap v688: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 687 KiB/s wr, 410 op/s 2024-04-25T16:43:14.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:13 smithi173 ceph-mon[32556]: pgmap v688: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 687 KiB/s wr, 410 op/s 2024-04-25T16:43:14.717 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:14 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:43:14.717 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:14 smithi203 systemd[1]: Stopped Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:14.717 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:14 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Consumed 7.796s CPU time. 2024-04-25T16:43:14.717 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:14 smithi203 systemd[1]: Starting Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:43:14.717 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:14 smithi203 podman[69349]: 2024-04-25 16:43:14.584312826 +0000 UTC m=+0.018779815 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:43:14.717 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:14 smithi203 podman[69349]: 2024-04-25 16:43:14.719673662 +0000 UTC m=+0.154140622 container create c42bda7d9a055314ab52d9fe3cb19bce0f99fa110db38416b00d6fb9d5bce38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T16:43:15.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:14 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 1. 2024-04-25T16:43:15.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:14 smithi173 systemd[1]: Stopped Ceph mds.k for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:15.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:14 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Consumed 7.839s CPU time. 2024-04-25T16:43:15.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:14 smithi173 systemd[1]: Starting Ceph mds.k for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:43:15.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:15 smithi173 podman[67611]: 2024-04-25 16:43:14.90742263 +0000 UTC m=+0.018635713 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:43:15.265 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:14 smithi203 podman[69349]: 2024-04-25 16:43:14.99008201 +0000 UTC m=+0.424548970 container init c42bda7d9a055314ab52d9fe3cb19bce0f99fa110db38416b00d6fb9d5bce38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T16:43:15.265 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:14 smithi203 podman[69349]: 2024-04-25 16:43:14.992869937 +0000 UTC m=+0.427336913 container start c42bda7d9a055314ab52d9fe3cb19bce0f99fa110db38416b00d6fb9d5bce38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:43:15.265 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mds[69368]: -- 172.21.15.203:0/2481959959 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55feb11e9a40 con 0x55feb1231180 2024-04-25T16:43:15.265 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: starting mds.l at 2024-04-25T16:43:15.265 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:15 smithi203 bash[69349]: c42bda7d9a055314ab52d9fe3cb19bce0f99fa110db38416b00d6fb9d5bce38a 2024-04-25T16:43:15.265 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:15 smithi203 systemd[1]: Started Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:15.762 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:15 smithi173 podman[67611]: 2024-04-25 16:43:15.481341467 +0000 UTC m=+0.592554536 container create 3ff4772f580a68114b2436de2cedd90e81114c5f68789eafd9854273e16aa0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-25T16:43:15.833 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: pgmap v689: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 868 KiB/s wr, 589 op/s 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: daemon mds.l restarted 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:boot 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: Replacing daemon mds.k as rank 1 with standby daemon mds.f 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T16:43:15.834 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:15 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:replay 3 up:active 2024-04-25T16:43:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: pgmap v689: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 868 KiB/s wr, 589 op/s 2024-04-25T16:43:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: daemon mds.l restarted 2024-04-25T16:43:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T16:43:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:boot 2024-04-25T16:43:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: Replacing daemon mds.k as rank 1 with standby daemon mds.f 2024-04-25T16:43:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-25T16:43:16.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-25T16:43:16.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:43:16.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:43:16.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T16:43:16.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:15 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:replay 3 up:active 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: pgmap v689: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 868 KiB/s wr, 589 op/s 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: daemon mds.l restarted 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: osdmap e97: 12 total, 12 up, 12 in 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:boot 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: Replacing daemon mds.k as rank 1 with standby daemon mds.f 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: MDS daemon mds.k is removed because it is dead or otherwise unavailable. 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: osdmap e98: 12 total, 12 up, 12 in 2024-04-25T16:43:16.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:replay 3 up:active 2024-04-25T16:43:16.033 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:15 smithi173 podman[67611]: 2024-04-25 16:43:15.851562065 +0000 UTC m=+0.962775155 container init 3ff4772f580a68114b2436de2cedd90e81114c5f68789eafd9854273e16aa0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True) 2024-04-25T16:43:16.033 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:15 smithi173 podman[67611]: 2024-04-25 16:43:15.854940787 +0000 UTC m=+0.966153859 container start 3ff4772f580a68114b2436de2cedd90e81114c5f68789eafd9854273e16aa0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T16:43:16.033 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-mds[67630]: -- 172.21.15.173:0/2566566776 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5621ab36da40 con 0x5621ab3b5180 2024-04-25T16:43:16.033 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:15 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: starting mds.k at 2024-04-25T16:43:16.033 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:15 smithi173 bash[67611]: 3ff4772f580a68114b2436de2cedd90e81114c5f68789eafd9854273e16aa0e2 2024-04-25T16:43:16.033 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:15 smithi173 systemd[1]: Started Ceph mds.k for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:17.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:17 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:resolve 2024-04-25T16:43:17.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:17 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:resolve 2024-04-25T16:43:17.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:17 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:boot 2024-04-25T16:43:17.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:17 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:43:17.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:17 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:43:17.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:17 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:resolve 2024-04-25T16:43:17.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:17 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:resolve 2024-04-25T16:43:17.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:17 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:boot 2024-04-25T16:43:17.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:17 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:43:17.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:17 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:43:17.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:17 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:resolve 2024-04-25T16:43:17.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:17 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:resolve 2024-04-25T16:43:17.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:17 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:boot 2024-04-25T16:43:17.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:17 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:43:17.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:17 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:43:18.239 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:18.241+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:18.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:18.242+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:43:18.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:18.242+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:43:18.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:18.242+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:43:18.240 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:18.242+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:43:18.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:18 smithi203 ceph-mon[33115]: pgmap v692: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 672 KiB/s wr, 575 op/s 2024-04-25T16:43:18.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:18 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:reconnect 2024-04-25T16:43:18.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:18 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:reconnect 2024-04-25T16:43:18.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:18 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:43:18.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:18 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0119997 2024-04-25T16:43:18.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:18 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0209995 2024-04-25T16:43:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:18 smithi098 ceph-mon[29162]: pgmap v692: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 672 KiB/s wr, 575 op/s 2024-04-25T16:43:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:18 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:reconnect 2024-04-25T16:43:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:18 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:reconnect 2024-04-25T16:43:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:18 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:43:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:18 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0119997 2024-04-25T16:43:18.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:18 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0209995 2024-04-25T16:43:18.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:18 smithi173 ceph-mon[32556]: pgmap v692: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 672 KiB/s wr, 575 op/s 2024-04-25T16:43:18.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:18 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:reconnect 2024-04-25T16:43:18.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:18 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:reconnect 2024-04-25T16:43:18.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:18 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:43:18.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:18 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0119997 2024-04-25T16:43:18.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:18 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0209995 2024-04-25T16:43:19.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:19 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:rejoin 2024-04-25T16:43:19.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:19 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:rejoin 2024-04-25T16:43:19.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:19 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:43:19.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:19 smithi203 ceph-mon[33115]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:43:19.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:19 smithi203 ceph-mon[33115]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-25T16:43:19.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:19 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:rejoin 2024-04-25T16:43:19.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:19 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:rejoin 2024-04-25T16:43:19.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:19 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:43:19.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:19 smithi098 ceph-mon[29162]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:43:19.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:19 smithi098 ceph-mon[29162]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-25T16:43:19.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:19 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:rejoin 2024-04-25T16:43:19.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:19 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:rejoin 2024-04-25T16:43:19.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:19 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:43:19.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:19 smithi173 ceph-mon[32556]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:43:19.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:19 smithi173 ceph-mon[32556]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-04-25T16:43:20.134 INFO:tasks.workunit.client.0.smithi098.stdout: GEN arch/x86/include/generated/asm/orc_hash.h 2024-04-25T16:43:20.162 INFO:tasks.workunit.client.0.smithi098.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_32.h 2024-04-25T16:43:20.163 INFO:tasks.workunit.client.0.smithi098.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_64.h 2024-04-25T16:43:20.163 INFO:tasks.workunit.client.0.smithi098.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_x32.h 2024-04-25T16:43:20.199 INFO:tasks.workunit.client.0.smithi098.stdout: SYSTBL arch/x86/include/generated/asm/syscalls_32.h 2024-04-25T16:43:20.233 INFO:tasks.workunit.client.0.smithi098.stdout: SYSHDR arch/x86/include/generated/asm/unistd_32_ia32.h 2024-04-25T16:43:20.245 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC arch/x86/tools/relocs_32.o 2024-04-25T16:43:20.269 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/bpf_perf_event.h 2024-04-25T16:43:20.291 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/errno.h 2024-04-25T16:43:20.311 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/fcntl.h 2024-04-25T16:43:20.332 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/ioctl.h 2024-04-25T16:43:20.352 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/ioctls.h 2024-04-25T16:43:20.372 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/ipcbuf.h 2024-04-25T16:43:20.391 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/param.h 2024-04-25T16:43:20.413 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/poll.h 2024-04-25T16:43:20.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:20 smithi203 ceph-mon[33115]: pgmap v693: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 294 KiB/s wr, 400 op/s 2024-04-25T16:43:20.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:20 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:20.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:20 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:43:20.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:20 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:active 2024-04-25T16:43:20.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:20 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:active 2024-04-25T16:43:20.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:20 smithi203 ceph-mon[33115]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:43:20.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:20 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:20.439 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/resource.h 2024-04-25T16:43:20.466 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/socket.h 2024-04-25T16:43:20.489 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/sockios.h 2024-04-25T16:43:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:20 smithi098 ceph-mon[29162]: pgmap v693: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 294 KiB/s wr, 400 op/s 2024-04-25T16:43:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:20 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:20 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:43:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:20 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:active 2024-04-25T16:43:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:20 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:active 2024-04-25T16:43:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:20 smithi098 ceph-mon[29162]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:43:20.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:20 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:20.501 INFO:tasks.workunit.client.0.smithi098.stdout: SYSHDR arch/x86/include/generated/asm/unistd_64_x32.h 2024-04-25T16:43:20.509 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/termbits.h 2024-04-25T16:43:20.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-mon[32556]: pgmap v693: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 294 KiB/s wr, 400 op/s 2024-04-25T16:43:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:43:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3267653510,v1:172.21.15.203:6839/3267653510] up:active 2024-04-25T16:43:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/4038856286,v1:172.21.15.203:6835/4038856286] up:active 2024-04-25T16:43:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-mon[32556]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:43:20.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:20.525 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/termios.h 2024-04-25T16:43:20.543 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/uapi/asm/types.h 2024-04-25T16:43:20.581 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC arch/x86/tools/relocs_64.o 2024-04-25T16:43:20.616 INFO:tasks.workunit.client.0.smithi098.stdout: SYSTBL arch/x86/include/generated/asm/syscalls_64.h 2024-04-25T16:43:20.628 INFO:tasks.workunit.client.0.smithi098.stdout: UPD include/generated/uapi/linux/version.h 2024-04-25T16:43:20.630 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC arch/x86/tools/relocs_common.o 2024-04-25T16:43:20.672 INFO:tasks.workunit.client.0.smithi098.stdout: UPD include/generated/compile.h 2024-04-25T16:43:20.709 INFO:tasks.workunit.client.0.smithi098.stdout: UPD include/config/kernel.release 2024-04-25T16:43:20.786 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/kallsyms 2024-04-25T16:43:20.850 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/selinux/genheaders/genheaders 2024-04-25T16:43:20.897 INFO:tasks.workunit.client.0.smithi098.stdout: DESCEND objtool 2024-04-25T16:43:20.985 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/sorttable 2024-04-25T16:43:21.001 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/selinux/mdp/mdp 2024-04-25T16:43:21.012 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:43:20.622+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:43:21.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:43:20 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:43:20.622+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:43:21.057 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/early_ioremap.h 2024-04-25T16:43:21.073 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/export.h 2024-04-25T16:43:21.084 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTLD arch/x86/tools/relocs 2024-04-25T16:43:21.091 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/mcs_spinlock.h 2024-04-25T16:43:21.107 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/irq_regs.h 2024-04-25T16:43:21.125 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/kmap_size.h 2024-04-25T16:43:21.140 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/local64.h 2024-04-25T16:43:21.160 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/mmiowb.h 2024-04-25T16:43:21.162 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/module.lds.h 2024-04-25T16:43:21.181 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/rwonce.h 2024-04-25T16:43:21.201 INFO:tasks.workunit.client.0.smithi098.stdout: WRAP arch/x86/include/generated/asm/unaligned.h 2024-04-25T16:43:21.241 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/asn1_compiler 2024-04-25T16:43:21.255 INFO:tasks.workunit.client.0.smithi098.stdout: UPD include/generated/utsrelease.h 2024-04-25T16:43:21.769 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep.o 2024-04-25T16:43:21.853 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTLD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep-in.o 2024-04-25T16:43:21.872 INFO:tasks.workunit.client.0.smithi098.stdout: LINK /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/fixdep 2024-04-25T16:43:22.041 INFO:tasks.workunit.client.0.smithi098.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/exec-cmd.h 2024-04-25T16:43:22.042 INFO:tasks.workunit.client.0.smithi098.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/help.h 2024-04-25T16:43:22.045 INFO:tasks.workunit.client.0.smithi098.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/pager.h 2024-04-25T16:43:22.050 INFO:tasks.workunit.client.0.smithi098.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/parse-options.h 2024-04-25T16:43:22.055 INFO:tasks.workunit.client.0.smithi098.stdout: INSTALL /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/include/subcmd/run-command.h 2024-04-25T16:43:22.133 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/exec-cmd.o 2024-04-25T16:43:22.140 INFO:tasks.workunit.client.0.smithi098.stdout: INSTALL libsubcmd_headers 2024-04-25T16:43:22.141 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/help.o 2024-04-25T16:43:22.148 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/pager.o 2024-04-25T16:43:22.156 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/parse-options.o 2024-04-25T16:43:22.166 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/run-command.o 2024-04-25T16:43:22.178 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/sigchain.o 2024-04-25T16:43:22.189 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/subcmd-config.o 2024-04-25T16:43:22.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:22 smithi203 ceph-mon[33115]: pgmap v694: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 304 KiB/s wr, 399 op/s 2024-04-25T16:43:22.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:22 smithi098 ceph-mon[29162]: pgmap v694: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 304 KiB/s wr, 399 op/s 2024-04-25T16:43:22.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:22 smithi173 ceph-mon[32556]: pgmap v694: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 304 KiB/s wr, 399 op/s 2024-04-25T16:43:24.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-mon[33115]: pgmap v695: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 31 KiB/s wr, 126 op/s 2024-04-25T16:43:24.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:24 smithi098 ceph-mon[29162]: pgmap v695: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 31 KiB/s wr, 126 op/s 2024-04-25T16:43:24.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:24 smithi173 ceph-mon[32556]: pgmap v695: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 31 KiB/s wr, 126 op/s 2024-04-25T16:43:24.663 INFO:tasks.workunit.client.0.smithi098.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/libsubcmd-in.o 2024-04-25T16:43:24.715 INFO:tasks.workunit.client.0.smithi098.stdout: AR /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libsubcmd/libsubcmd.a 2024-04-25T16:43:24.859 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f321748d700 time 2024-04-25T16:43:24.513339+0000 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3221cc1ba6] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3221cc1dd1] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b7d098726b] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b7d0926428] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b7d09278a9] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b7d09299c3] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b7d07f585d] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b7d07f7e8e] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b7d07f8894] 2024-04-25T16:43:24.860 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b7d07e305b] 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3221e3eac6] 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 12: (DispatchQueue::entry()+0xb27) [0x7f3221e3c0b9] 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3221f26ccb] 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 14: (Thread::entry_wrapper()+0x43) [0x7f3221c9072d] 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3221c90749] 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f32208341ca] 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 17: clone() 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: *** Caught signal (Aborted) ** 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: in thread 7f321748d700 thread_name:ms_dispatch 2024-04-25T16:43:24.861 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f41f8c83700 time 2024-04-25T16:43:24.578315+0000 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f42034b7ba6] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42034b7dd1] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x561e4779826b] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561e47737428] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561e477388a9] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561e4773a9c3] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561e4760685d] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561e47608e8e] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561e47609894] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561e475f405b] 2024-04-25T16:43:24.862 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4203634ac6] 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 12: (DispatchQueue::entry()+0xb27) [0x7f42036320b9] 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f420371cccb] 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 14: (Thread::entry_wrapper()+0x43) [0x7f420348672d] 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4203486749] 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f420202a1ca] 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 17: clone() 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: *** Caught signal (Aborted) ** 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: in thread 7f41f8c83700 thread_name:ms_dispatch 2024-04-25T16:43:24.863 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.514+0000 7f321748d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f321748d700 time 2024-04-25T16:43:24.513339+0000 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3221cc1ba6] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3221cc1dd1] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b7d098726b] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b7d0926428] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b7d09278a9] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b7d09299c3] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b7d07f585d] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b7d07f7e8e] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b7d07f8894] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b7d07e305b] 2024-04-25T16:43:24.864 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3221e3eac6] 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 12: (DispatchQueue::entry()+0xb27) [0x7f3221e3c0b9] 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3221f26ccb] 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 14: (Thread::entry_wrapper()+0x43) [0x7f3221c9072d] 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3221c90749] 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f32208341ca] 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 17: clone() 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.579+0000 7f41f8c83700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f41f8c83700 time 2024-04-25T16:43:24.578315+0000 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.865 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f42034b7ba6] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42034b7dd1] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x561e4779826b] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561e47737428] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561e477388a9] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561e4773a9c3] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561e4760685d] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561e47608e8e] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561e47609894] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561e475f405b] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4203634ac6] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 12: (DispatchQueue::entry()+0xb27) [0x7f42036320b9] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f420371cccb] 2024-04-25T16:43:24.866 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 14: (Thread::entry_wrapper()+0x43) [0x7f420348672d] 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4203486749] 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f420202a1ca] 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 17: clone() 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f322083ed20] 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2: gsignal() 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 3: abort() 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3221cc1ced] 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3221cc1dd1] 2024-04-25T16:43:24.867 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b7d098726b] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b7d0926428] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b7d09278a9] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b7d09299c3] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b7d07f585d] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b7d07f7e8e] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b7d07f8894] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b7d07e305b] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3221e3eac6] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 15: (DispatchQueue::entry()+0xb27) [0x7f3221e3c0b9] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3221f26ccb] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 17: (Thread::entry_wrapper()+0x43) [0x7f3221c9072d] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3221c90749] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f32208341ca] 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 20: clone() 2024-04-25T16:43:24.868 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.517+0000 7f321748d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: in thread 7f321748d700 thread_name:ms_dispatch 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f322083ed20] 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2: gsignal() 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 3: abort() 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3221cc1ced] 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4202034d20] 2024-04-25T16:43:24.869 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2: gsignal() 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 3: abort() 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f42034b7ced] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42034b7dd1] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x561e4779826b] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561e47737428] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561e477388a9] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561e4773a9c3] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561e4760685d] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561e47608e8e] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561e47609894] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561e475f405b] 2024-04-25T16:43:24.870 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4203634ac6] 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 15: (DispatchQueue::entry()+0xb27) [0x7f42036320b9] 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f420371cccb] 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 17: (Thread::entry_wrapper()+0x43) [0x7f420348672d] 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4203486749] 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f420202a1ca] 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 20: clone() 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.582+0000 7f41f8c83700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: in thread 7f41f8c83700 thread_name:ms_dispatch 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4202034d20] 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2: gsignal() 2024-04-25T16:43:24.871 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 3: abort() 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f42034b7ced] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42034b7dd1] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x561e4779826b] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561e47737428] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561e477388a9] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561e4773a9c3] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561e4760685d] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561e47608e8e] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561e47609894] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561e475f405b] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4203634ac6] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 15: (DispatchQueue::entry()+0xb27) [0x7f42036320b9] 2024-04-25T16:43:24.872 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f420371cccb] 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 17: (Thread::entry_wrapper()+0x43) [0x7f420348672d] 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4203486749] 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f420202a1ca] 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 20: clone() 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: -1> 2024-04-25T16:43:24.579+0000 7f41f8c83700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f41f8c83700 time 2024-04-25T16:43:24.578315+0000 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f42034b7ba6] 2024-04-25T16:43:24.873 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3221cc1dd1] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b7d098726b] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b7d0926428] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b7d09278a9] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b7d09299c3] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b7d07f585d] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b7d07f7e8e] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b7d07f8894] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b7d07e305b] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3221e3eac6] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 15: (DispatchQueue::entry()+0xb27) [0x7f3221e3c0b9] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3221f26ccb] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 17: (Thread::entry_wrapper()+0x43) [0x7f3221c9072d] 2024-04-25T16:43:24.874 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3221c90749] 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f32208341ca] 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 20: clone() 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: -1> 2024-04-25T16:43:24.514+0000 7f321748d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f321748d700 time 2024-04-25T16:43:24.513339+0000 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3221cc1ba6] 2024-04-25T16:43:24.875 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42034b7dd1] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x561e4779826b] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561e47737428] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561e477388a9] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561e4773a9c3] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561e4760685d] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561e47608e8e] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561e47609894] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561e475f405b] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4203634ac6] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 12: (DispatchQueue::entry()+0xb27) [0x7f42036320b9] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f420371cccb] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 14: (Thread::entry_wrapper()+0x43) [0x7f420348672d] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4203486749] 2024-04-25T16:43:24.876 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f420202a1ca] 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 17: clone() 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 0> 2024-04-25T16:43:24.582+0000 7f41f8c83700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: in thread 7f41f8c83700 thread_name:ms_dispatch 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4202034d20] 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2: gsignal() 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 3: abort() 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f42034b7ced] 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42034b7dd1] 2024-04-25T16:43:24.877 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x561e4779826b] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3221cc1dd1] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b7d098726b] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b7d0926428] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b7d09278a9] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b7d09299c3] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b7d07f585d] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b7d07f7e8e] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b7d07f8894] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b7d07e305b] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3221e3eac6] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 12: (DispatchQueue::entry()+0xb27) [0x7f3221e3c0b9] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3221f26ccb] 2024-04-25T16:43:24.878 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 14: (Thread::entry_wrapper()+0x43) [0x7f3221c9072d] 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3221c90749] 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f32208341ca] 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 17: clone() 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 0> 2024-04-25T16:43:24.517+0000 7f321748d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: in thread 7f321748d700 thread_name:ms_dispatch 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f322083ed20] 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2: gsignal() 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 3: abort() 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3221cc1ced] 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3221cc1dd1] 2024-04-25T16:43:24.879 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b7d098726b] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561e47737428] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561e477388a9] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561e4773a9c3] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561e4760685d] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561e47608e8e] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561e47609894] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561e475f405b] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4203634ac6] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 15: (DispatchQueue::entry()+0xb27) [0x7f42036320b9] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f420371cccb] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 17: (Thread::entry_wrapper()+0x43) [0x7f420348672d] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4203486749] 2024-04-25T16:43:24.880 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f420202a1ca] 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 20: clone() 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: -1> 2024-04-25T16:43:24.579+0000 7f41f8c83700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f41f8c83700 time 2024-04-25T16:43:24.578315+0000 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f42034b7ba6] 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42034b7dd1] 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x561e4779826b] 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b7d0926428] 2024-04-25T16:43:24.881 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b7d09278a9] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b7d09299c3] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b7d07f585d] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b7d07f7e8e] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b7d07f8894] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b7d07e305b] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3221e3eac6] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 15: (DispatchQueue::entry()+0xb27) [0x7f3221e3c0b9] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3221f26ccb] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 17: (Thread::entry_wrapper()+0x43) [0x7f3221c9072d] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3221c90749] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f32208341ca] 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 20: clone() 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:24.882 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: -1> 2024-04-25T16:43:24.514+0000 7f321748d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f321748d700 time 2024-04-25T16:43:24.513339+0000 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f3221cc1ba6] 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3221cc1dd1] 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55b7d098726b] 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561e47737428] 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561e477388a9] 2024-04-25T16:43:24.883 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561e4773a9c3] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561e4760685d] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561e47608e8e] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561e47609894] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561e475f405b] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4203634ac6] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 12: (DispatchQueue::entry()+0xb27) [0x7f42036320b9] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f420371cccb] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 14: (Thread::entry_wrapper()+0x43) [0x7f420348672d] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 15: (Thread::_entry_func(void*)+0xd) [0x7f4203486749] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f420202a1ca] 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 17: clone() 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 0> 2024-04-25T16:43:24.582+0000 7f41f8c83700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: in thread 7f41f8c83700 thread_name:ms_dispatch 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.884 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f4202034d20] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2: gsignal() 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 3: abort() 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f42034b7ced] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f42034b7dd1] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x561e4779826b] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x561e47737428] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x561e477388a9] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x561e4773a9c3] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b7d0926428] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b7d09278a9] 2024-04-25T16:43:24.885 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b7d09299c3] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b7d07f585d] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b7d07f7e8e] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b7d07f8894] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b7d07e305b] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3221e3eac6] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 12: (DispatchQueue::entry()+0xb27) [0x7f3221e3c0b9] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3221f26ccb] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 14: (Thread::entry_wrapper()+0x43) [0x7f3221c9072d] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 15: (Thread::_entry_func(void*)+0xd) [0x7f3221c90749] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f32208341ca] 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 17: clone() 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 0> 2024-04-25T16:43:24.517+0000 7f321748d700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:24.886 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: in thread 7f321748d700 thread_name:ms_dispatch 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f322083ed20] 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2: gsignal() 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 3: abort() 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f3221cc1ced] 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f3221cc1dd1] 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55b7d098726b] 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55b7d0926428] 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55b7d09278a9] 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55b7d09299c3] 2024-04-25T16:43:24.887 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x561e4760685d] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x561e47608e8e] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x561e47609894] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x561e475f405b] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f4203634ac6] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 15: (DispatchQueue::entry()+0xb27) [0x7f42036320b9] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f420371cccb] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 17: (Thread::entry_wrapper()+0x43) [0x7f420348672d] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 18: (Thread::_entry_func(void*)+0xd) [0x7f4203486749] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f420202a1ca] 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 20: clone() 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:24.888 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[68061]: 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55b7d07f585d] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55b7d07f7e8e] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55b7d07f8894] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55b7d07e305b] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f3221e3eac6] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 15: (DispatchQueue::entry()+0xb27) [0x7f3221e3c0b9] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3221f26ccb] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 17: (Thread::entry_wrapper()+0x43) [0x7f3221c9072d] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 18: (Thread::_entry_func(void*)+0xd) [0x7f3221c90749] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f32208341ca] 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 20: clone() 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[68388]: 2024-04-25T16:43:24.889 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:24 smithi203 podman[69418]: 2024-04-25 16:43:24.575786949 +0000 UTC m=+0.020298818 container died c8fe897e36cbced92dcf5365bf55e734492246b1669633e5637b5c89e162888a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T16:43:25.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:25 smithi203 podman[69418]: 2024-04-25 16:43:25.215241526 +0000 UTC m=+0.659753385 container remove c8fe897e36cbced92dcf5365bf55e734492246b1669633e5637b5c89e162888a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:43:25.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:25 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:25.687 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:25 smithi203 podman[69430]: 2024-04-25 16:43:25.267282011 +0000 UTC m=+0.648763259 container died 91f363d74dcebfe9ad694abf3d63494b945b268ee9f297ae0dd5fb4dd752d963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T16:43:25.841 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/weak.o 2024-04-25T16:43:25.847 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/check.o 2024-04-25T16:43:25.852 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/special.o 2024-04-25T16:43:25.861 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/builtin-check.o 2024-04-25T16:43:25.870 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/elf.o 2024-04-25T16:43:25.885 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool.o 2024-04-25T16:43:25.911 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/special.o 2024-04-25T16:43:26.383 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:26 smithi203 ceph-mon[33115]: pgmap v696: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 184 KiB/s wr, 118 op/s 2024-04-25T16:43:26.383 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:26 smithi203 podman[69430]: 2024-04-25 16:43:26.25005173 +0000 UTC m=+1.631532978 container remove 91f363d74dcebfe9ad694abf3d63494b945b268ee9f297ae0dd5fb4dd752d963 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T16:43:26.383 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:26 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:26.462 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:26 smithi173 ceph-mon[32556]: pgmap v696: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 184 KiB/s wr, 118 op/s 2024-04-25T16:43:26.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:26 smithi098 ceph-mon[29162]: pgmap v696: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 184 KiB/s wr, 118 op/s 2024-04-25T16:43:26.686 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:26 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Failed with result 'exit-code'. 2024-04-25T16:43:26.686 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:26 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Consumed 2.247s CPU time. 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:26 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:43:26.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:26 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 2.073s CPU time. 2024-04-25T16:43:27.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:27 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:27.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:27 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:27.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:27 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:27.549 INFO:tasks.workunit.client.0.smithi098.stdout: MKDIR /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/lib/ 2024-04-25T16:43:27.565 INFO:tasks.workunit.client.0.smithi098.stdout: GEN /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/lib/inat-tables.c 2024-04-25T16:43:27.631 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/orc_gen.o 2024-04-25T16:43:27.667 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/decode.o 2024-04-25T16:43:27.714 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/orc_dump.o 2024-04-25T16:43:28.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-mon[33115]: pgmap v697: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 178 KiB/s wr, 104 op/s 2024-04-25T16:43:28.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f80909d7700 time 2024-04-25T16:43:28.171272+0000 2024-04-25T16:43:28.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:28.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:28.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f809b20bba6] 2024-04-25T16:43:28.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f809b20bdd1] 2024-04-25T16:43:28.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56499c64726b] 2024-04-25T16:43:28.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56499c5e6428] 2024-04-25T16:43:28.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56499c5e78a9] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56499c5e99c3] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56499c4b585d] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56499c4b7e8e] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56499c4b8894] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56499c4a305b] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f809b388ac6] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 12: (DispatchQueue::entry()+0xb27) [0x7f809b3860b9] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f809b470ccb] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 14: (Thread::entry_wrapper()+0x43) [0x7f809b1da72d] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 15: (Thread::_entry_func(void*)+0xd) [0x7f809b1da749] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8099d7e1ca] 2024-04-25T16:43:28.438 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 17: clone() 2024-04-25T16:43:28.439 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.172+0000 7f80909d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f80909d7700 time 2024-04-25T16:43:28.171272+0000 2024-04-25T16:43:28.439 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f809b20bba6] 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f809b20bdd1] 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56499c64726b] 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56499c5e6428] 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56499c5e78a9] 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56499c5e99c3] 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56499c4b585d] 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56499c4b7e8e] 2024-04-25T16:43:28.440 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56499c4b8894] 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56499c4a305b] 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f809b388ac6] 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 12: (DispatchQueue::entry()+0xb27) [0x7f809b3860b9] 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f809b470ccb] 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 14: (Thread::entry_wrapper()+0x43) [0x7f809b1da72d] 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 15: (Thread::_entry_func(void*)+0xd) [0x7f809b1da749] 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8099d7e1ca] 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 17: clone() 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: *** Caught signal (Aborted) ** 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: in thread 7f80909d7700 thread_name:ms_dispatch 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:28.441 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8099d88d20] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2: gsignal() 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 3: abort() 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f809b20bced] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f809b20bdd1] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56499c64726b] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56499c5e6428] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56499c5e78a9] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56499c5e99c3] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56499c4b585d] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56499c4b7e8e] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56499c4b8894] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56499c4a305b] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f809b388ac6] 2024-04-25T16:43:28.442 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 15: (DispatchQueue::entry()+0xb27) [0x7f809b3860b9] 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f809b470ccb] 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 17: (Thread::entry_wrapper()+0x43) [0x7f809b1da72d] 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 18: (Thread::_entry_func(void*)+0xd) [0x7f809b1da749] 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8099d7e1ca] 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 20: clone() 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.175+0000 7f80909d7700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: in thread 7f80909d7700 thread_name:ms_dispatch 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8099d88d20] 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2: gsignal() 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 3: abort() 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f809b20bced] 2024-04-25T16:43:28.443 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f809b20bdd1] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56499c64726b] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56499c5e6428] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56499c5e78a9] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56499c5e99c3] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56499c4b585d] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56499c4b7e8e] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56499c4b8894] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56499c4a305b] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f809b388ac6] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 15: (DispatchQueue::entry()+0xb27) [0x7f809b3860b9] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f809b470ccb] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 17: (Thread::entry_wrapper()+0x43) [0x7f809b1da72d] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 18: (Thread::_entry_func(void*)+0xd) [0x7f809b1da749] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8099d7e1ca] 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 20: clone() 2024-04-25T16:43:28.444 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: -1> 2024-04-25T16:43:28.172+0000 7f80909d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f80909d7700 time 2024-04-25T16:43:28.171272+0000 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f809b20bba6] 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f809b20bdd1] 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56499c64726b] 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56499c5e6428] 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56499c5e78a9] 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56499c5e99c3] 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56499c4b585d] 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56499c4b7e8e] 2024-04-25T16:43:28.445 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56499c4b8894] 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56499c4a305b] 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f809b388ac6] 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 12: (DispatchQueue::entry()+0xb27) [0x7f809b3860b9] 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f809b470ccb] 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 14: (Thread::entry_wrapper()+0x43) [0x7f809b1da72d] 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 15: (Thread::_entry_func(void*)+0xd) [0x7f809b1da749] 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8099d7e1ca] 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 17: clone() 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 0> 2024-04-25T16:43:28.175+0000 7f80909d7700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: in thread 7f80909d7700 thread_name:ms_dispatch 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:28.446 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8099d88d20] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2: gsignal() 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 3: abort() 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f809b20bced] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f809b20bdd1] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56499c64726b] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56499c5e6428] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56499c5e78a9] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56499c5e99c3] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56499c4b585d] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56499c4b7e8e] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56499c4b8894] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56499c4a305b] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f809b388ac6] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 15: (DispatchQueue::entry()+0xb27) [0x7f809b3860b9] 2024-04-25T16:43:28.447 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f809b470ccb] 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 17: (Thread::entry_wrapper()+0x43) [0x7f809b1da72d] 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 18: (Thread::_entry_func(void*)+0xd) [0x7f809b1da749] 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8099d7e1ca] 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 20: clone() 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: -1> 2024-04-25T16:43:28.172+0000 7f80909d7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f80909d7700 time 2024-04-25T16:43:28.171272+0000 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f809b20bba6] 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f809b20bdd1] 2024-04-25T16:43:28.448 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56499c64726b] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56499c5e6428] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56499c5e78a9] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56499c5e99c3] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56499c4b585d] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56499c4b7e8e] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56499c4b8894] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56499c4a305b] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f809b388ac6] 2024-04-25T16:43:28.449 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 12: (DispatchQueue::entry()+0xb27) [0x7f809b3860b9] 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f809b470ccb] 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 14: (Thread::entry_wrapper()+0x43) [0x7f809b1da72d] 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 15: (Thread::_entry_func(void*)+0xd) [0x7f809b1da749] 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f8099d7e1ca] 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 17: clone() 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 0> 2024-04-25T16:43:28.175+0000 7f80909d7700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: in thread 7f80909d7700 thread_name:ms_dispatch 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8099d88d20] 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2: gsignal() 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 3: abort() 2024-04-25T16:43:28.450 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f809b20bced] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f809b20bdd1] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56499c64726b] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x56499c5e6428] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56499c5e78a9] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56499c5e99c3] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x56499c4b585d] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x56499c4b7e8e] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x56499c4b8894] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x56499c4a305b] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f809b388ac6] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 15: (DispatchQueue::entry()+0xb27) [0x7f809b3860b9] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f809b470ccb] 2024-04-25T16:43:28.451 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 17: (Thread::entry_wrapper()+0x43) [0x7f809b1da72d] 2024-04-25T16:43:28.452 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 18: (Thread::_entry_func(void*)+0xd) [0x7f809b1da749] 2024-04-25T16:43:28.452 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f8099d7e1ca] 2024-04-25T16:43:28.452 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 20: clone() 2024-04-25T16:43:28.452 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:28.452 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[67548]: 2024-04-25T16:43:28.452 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 podman[69505]: 2024-04-25 16:43:28.23686498 +0000 UTC m=+0.018822649 container died 4c1bebd6b08f46c71d5af873dea04bf682c08f71b85b779674c31b328a0cb994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T16:43:28.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:28 smithi098 ceph-mon[29162]: pgmap v697: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 178 KiB/s wr, 104 op/s 2024-04-25T16:43:28.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:28 smithi173 ceph-mon[32556]: pgmap v697: 129 pgs: 129 active+clean; 1.6 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 178 KiB/s wr, 104 op/s 2024-04-25T16:43:29.153 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 podman[69505]: 2024-04-25 16:43:28.901803594 +0000 UTC m=+0.683761262 container remove 4c1bebd6b08f46c71d5af873dea04bf682c08f71b85b779674c31b328a0cb994 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T16:43:29.153 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:28 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:29.402 INFO:tasks.workunit.client.0.smithi098.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/arch/x86/objtool-in.o 2024-04-25T16:43:29.436 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:29 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Failed with result 'exit-code'. 2024-04-25T16:43:29.436 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:29 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Consumed 12.558s CPU time. 2024-04-25T16:43:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:30 smithi203 ceph-mon[33115]: pgmap v698: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 251 KiB/s wr, 104 op/s 2024-04-25T16:43:30.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:30 smithi098 ceph-mon[29162]: pgmap v698: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 251 KiB/s wr, 104 op/s 2024-04-25T16:43:30.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:30 smithi173 ceph-mon[32556]: pgmap v698: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 251 KiB/s wr, 104 op/s 2024-04-25T16:43:31.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:31 smithi203 ceph-mon[33115]: pgmap v699: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 521 KiB/s wr, 112 op/s 2024-04-25T16:43:31.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:31 smithi098 ceph-mon[29162]: pgmap v699: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 521 KiB/s wr, 112 op/s 2024-04-25T16:43:31.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:31 smithi173 ceph-mon[32556]: pgmap v699: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 521 KiB/s wr, 112 op/s 2024-04-25T16:43:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:33.256+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.203:6834/2230353021 2024-04-25T16:43:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:33.256+0000 7fdeb0ff9640 1 client.16329 ms_handle_refused on v2:172.21.15.203:6834/2230353021 2024-04-25T16:43:33.254 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:33.256+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:43:33.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:33 smithi098 ceph-mon[29162]: pgmap v700: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 518 KiB/s wr, 105 op/s 2024-04-25T16:43:33.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:33 smithi173 ceph-mon[32556]: pgmap v700: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 518 KiB/s wr, 105 op/s 2024-04-25T16:43:33.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:33 smithi203 ceph-mon[33115]: pgmap v700: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 518 KiB/s wr, 105 op/s 2024-04-25T16:43:35.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:35 smithi098 ceph-mon[29162]: pgmap v701: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 649 KiB/s wr, 108 op/s 2024-04-25T16:43:35.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:35 smithi173 ceph-mon[32556]: pgmap v701: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 649 KiB/s wr, 108 op/s 2024-04-25T16:43:35.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:35 smithi203 ceph-mon[33115]: pgmap v701: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 649 KiB/s wr, 108 op/s 2024-04-25T16:43:36.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:36 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-04-25T16:43:36.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:36 smithi203 systemd[1]: Stopped Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:36.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:36 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 2.073s CPU time. 2024-04-25T16:43:36.687 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:36 smithi203 systemd[1]: Starting Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:43:36.688 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:36 smithi203 podman[69600]: 2024-04-25 16:43:36.584417898 +0000 UTC m=+0.018809934 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:43:36.948 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:36 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:43:36.948 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:36 smithi203 systemd[1]: Stopped Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:36.948 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:36 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service: Consumed 2.247s CPU time. 2024-04-25T16:43:36.948 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:36 smithi203 systemd[1]: Starting Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:43:36.948 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:36 smithi203 podman[69600]: 2024-04-25 16:43:36.728084145 +0000 UTC m=+0.162476181 container create 144f2147d5cdc92ec3a28abd120fe8636ab27ad2a75275887211e227d48d6f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T16:43:37.235 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:37 smithi203 podman[69600]: 2024-04-25 16:43:37.065302828 +0000 UTC m=+0.499694868 container init 144f2147d5cdc92ec3a28abd120fe8636ab27ad2a75275887211e227d48d6f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:43:37.235 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:37 smithi203 podman[69600]: 2024-04-25 16:43:37.068139901 +0000 UTC m=+0.502531937 container start 144f2147d5cdc92ec3a28abd120fe8636ab27ad2a75275887211e227d48d6f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T16:43:37.235 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mds[69644]: -- 172.21.15.203:0/2816820718 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55befa533a40 con 0x55befa57b180 2024-04-25T16:43:37.235 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: starting mds.f at 2024-04-25T16:43:37.235 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:37 smithi203 bash[69600]: 144f2147d5cdc92ec3a28abd120fe8636ab27ad2a75275887211e227d48d6f96 2024-04-25T16:43:37.235 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:43:37 smithi203 systemd[1]: Started Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:37.438 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:37.440+0000 7fdebd7fa640 1 -- 172.21.15.98:0/2834070656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fdea0005a20 con 0x7fdec8101760 2024-04-25T16:43:37.439 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:37.441+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fdea0005a20 con 0x7fdec8101760 2024-04-25T16:43:37.442 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:37.445+0000 7fdea77fe640 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7fde98005b00 con 0x55e32644d4d0 2024-04-25T16:43:37.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:37.445+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7fde98005b00 con 0x55e32644d4d0 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: pgmap v702: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 527 KiB/s wr, 59 op/s 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: daemon mds.f restarted 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:boot 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:43:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:37 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:43:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: pgmap v702: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 527 KiB/s wr, 59 op/s 2024-04-25T16:43:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: daemon mds.f restarted 2024-04-25T16:43:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T16:43:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:boot 2024-04-25T16:43:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-25T16:43:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:43:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:43:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:37 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:43:37.592 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:37 smithi203 podman[69690]: 2024-04-25 16:43:37.240406406 +0000 UTC m=+0.017957446 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:43:37.592 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:37 smithi203 podman[69690]: 2024-04-25 16:43:37.349959432 +0000 UTC m=+0.127510462 container create e7ce0df81dab0f3e856e890e8300912bb73a1092e0f4ed17b99618f23805dd4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T16:43:37.592 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: pgmap v702: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 527 KiB/s wr, 59 op/s 2024-04-25T16:43:37.592 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: daemon mds.f restarted 2024-04-25T16:43:37.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:43:37.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: osdmap e99: 12 total, 12 up, 12 in 2024-04-25T16:43:37.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:boot 2024-04-25T16:43:37.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: Standby daemon mds.k assigned to filesystem cephfs as rank 1 2024-04-25T16:43:37.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=h=up:active,2=i=up:active,3=e=up:active,4=c=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:43:37.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:43:37.593 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:43:37.936 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:37 smithi203 podman[69690]: 2024-04-25 16:43:37.595071163 +0000 UTC m=+0.372622191 container init e7ce0df81dab0f3e856e890e8300912bb73a1092e0f4ed17b99618f23805dd4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:37 smithi203 podman[69690]: 2024-04-25 16:43:37.597737094 +0000 UTC m=+0.375288123 container start e7ce0df81dab0f3e856e890e8300912bb73a1092e0f4ed17b99618f23805dd4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-mds[69709]: -- 172.21.15.203:0/2073004329 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556c6657a40 con 0x5556c669fa80 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[69705]: starting mds.i at 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:37 smithi203 bash[69690]: e7ce0df81dab0f3e856e890e8300912bb73a1092e0f4ed17b99618f23805dd4f 2024-04-25T16:43:37.937 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:43:37 smithi203 systemd[1]: Started Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:38 smithi098 ceph-mon[29162]: daemon mds.i restarted 2024-04-25T16:43:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:38 smithi098 ceph-mon[29162]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T16:43:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:38 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:resolve 2024-04-25T16:43:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:38 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:boot 2024-04-25T16:43:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:38 smithi098 ceph-mon[29162]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:43:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:38 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:resolve,3=e=up:active,4=c=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:43:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:38 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:43:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:38 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:43:38.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:38 smithi173 ceph-mon[32556]: daemon mds.i restarted 2024-04-25T16:43:38.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:38 smithi173 ceph-mon[32556]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T16:43:38.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:38 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:resolve 2024-04-25T16:43:38.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:38 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:boot 2024-04-25T16:43:38.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:38 smithi173 ceph-mon[32556]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:43:38.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:38 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:resolve,3=e=up:active,4=c=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:43:38.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:38 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:43:38.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:38 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:43:38.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:38 smithi203 ceph-mon[33115]: daemon mds.i restarted 2024-04-25T16:43:38.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:38 smithi203 ceph-mon[33115]: osdmap e100: 12 total, 12 up, 12 in 2024-04-25T16:43:38.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:38 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:resolve 2024-04-25T16:43:38.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:38 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:boot 2024-04-25T16:43:38.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:38 smithi203 ceph-mon[33115]: Standby daemon mds.l assigned to filesystem cephfs as rank 2 2024-04-25T16:43:38.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:38 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:resolve,3=e=up:active,4=c=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:43:38.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:38 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:43:38.687 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:38 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:43:39.527 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:43:39.527 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 systemd[1]: Stopped Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:39.527 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service: Consumed 12.558s CPU time. 2024-04-25T16:43:39.528 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 systemd[1]: Starting Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:43:39.528 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 podman[69783]: 2024-04-25 16:43:39.330985557 +0000 UTC m=+0.018442456 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:43:39.528 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 podman[69783]: 2024-04-25 16:43:39.530302793 +0000 UTC m=+0.217759677 container create 6bdeb6453932c551c48b9a94c6801663a192304d9a4d2e772fe26f8e3e1a3cb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T16:43:39.798 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:39 smithi203 ceph-mon[33115]: pgmap v705: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 219 KiB/s wr, 24 op/s 2024-04-25T16:43:39.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:39 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:resolve 2024-04-25T16:43:39.799 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:39 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:43:39.799 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 podman[69783]: 2024-04-25 16:43:39.766792965 +0000 UTC m=+0.454249848 container init 6bdeb6453932c551c48b9a94c6801663a192304d9a4d2e772fe26f8e3e1a3cb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T16:43:39.799 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 podman[69783]: 2024-04-25 16:43:39.769602579 +0000 UTC m=+0.457059473 container start 6bdeb6453932c551c48b9a94c6801663a192304d9a4d2e772fe26f8e3e1a3cb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T16:43:40.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:39 smithi098 ceph-mon[29162]: pgmap v705: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 219 KiB/s wr, 24 op/s 2024-04-25T16:43:40.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:39 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:resolve 2024-04-25T16:43:40.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:39 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:43:40.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:39 smithi173 ceph-mon[32556]: pgmap v705: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 219 KiB/s wr, 24 op/s 2024-04-25T16:43:40.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:39 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:resolve 2024-04-25T16:43:40.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:39 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:43:40.186 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 ceph-mds[69802]: -- 172.21.15.203:0/1814001340 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5598d4113a40 con 0x5598d415b180 2024-04-25T16:43:40.187 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:39 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[69798]: starting mds.c at 2024-04-25T16:43:40.187 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:40 smithi203 bash[69783]: 6bdeb6453932c551c48b9a94c6801663a192304d9a4d2e772fe26f8e3e1a3cb3 2024-04-25T16:43:40.187 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:43:40 smithi203 systemd[1]: Started Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:43:40.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:40 smithi203 ceph-mon[33115]: daemon mds.c restarted 2024-04-25T16:43:40.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:40 smithi203 ceph-mon[33115]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T16:43:40.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:40 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:boot 2024-04-25T16:43:40.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:40 smithi203 ceph-mon[33115]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-25T16:43:40.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:40 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:resolve,2=l=up:resolve,3=e=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:43:40.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:40 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:43:40.937 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:40 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-04-25T16:43:41.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:40 smithi098 ceph-mon[29162]: daemon mds.c restarted 2024-04-25T16:43:41.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:40 smithi098 ceph-mon[29162]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T16:43:41.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:40 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:boot 2024-04-25T16:43:41.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:40 smithi098 ceph-mon[29162]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-25T16:43:41.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:40 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:resolve,2=l=up:resolve,3=e=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:43:41.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:40 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:43:41.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:40 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-04-25T16:43:41.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:40 smithi173 ceph-mon[32556]: daemon mds.c restarted 2024-04-25T16:43:41.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:40 smithi173 ceph-mon[32556]: osdmap e101: 12 total, 12 up, 12 in 2024-04-25T16:43:41.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:40 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:boot 2024-04-25T16:43:41.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:40 smithi173 ceph-mon[32556]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-25T16:43:41.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:40 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:resolve,2=l=up:resolve,3=e=up:active} 3 up:standby-replay 5 up:standby, 1 failed 2024-04-25T16:43:41.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:40 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:43:41.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:40 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 1 up:replay 2 up:resolve 2 up:active 2024-04-25T16:43:42.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:41 smithi203 ceph-mon[33115]: pgmap v707: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-04-25T16:43:42.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:41 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:resolve 2024-04-25T16:43:42.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:41 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-04-25T16:43:42.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:41 smithi098 ceph-mon[29162]: pgmap v707: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-04-25T16:43:42.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:41 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:resolve 2024-04-25T16:43:42.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:41 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-04-25T16:43:42.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:41 smithi173 ceph-mon[32556]: pgmap v707: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.2 KiB/s wr, 15 op/s 2024-04-25T16:43:42.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:41 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:resolve 2024-04-25T16:43:42.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:41 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:resolve 2 up:active 2024-04-25T16:43:42.916 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libstring.o 2024-04-25T16:43:42.921 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/libctype.o 2024-04-25T16:43:42.927 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/str_error_r.o 2024-04-25T16:43:42.934 INFO:tasks.workunit.client.0.smithi098.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/librbtree.o 2024-04-25T16:43:43.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:42 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:reconnect 2024-04-25T16:43:43.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:42 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:reconnect 2024-04-25T16:43:43.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:42 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:reconnect 2024-04-25T16:43:43.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:42 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-04-25T16:43:43.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:42 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0099998 2024-04-25T16:43:43.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:42 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0179996 2024-04-25T16:43:43.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:42 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0279996 2024-04-25T16:43:43.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:42 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:reconnect 2024-04-25T16:43:43.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:42 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:reconnect 2024-04-25T16:43:43.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:42 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:reconnect 2024-04-25T16:43:43.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:42 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-04-25T16:43:43.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:42 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0099998 2024-04-25T16:43:43.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:42 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0179996 2024-04-25T16:43:43.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:42 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0279996 2024-04-25T16:43:43.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:42 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:reconnect 2024-04-25T16:43:43.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:42 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:reconnect 2024-04-25T16:43:43.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:42 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:reconnect 2024-04-25T16:43:43.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:42 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:reconnect 2 up:active 2024-04-25T16:43:43.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:42 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0099998 2024-04-25T16:43:43.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:42 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0179996 2024-04-25T16:43:43.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:42 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0279996 2024-04-25T16:43:43.663 INFO:tasks.workunit.client.0.smithi098.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool-in.o 2024-04-25T16:43:43.734 INFO:tasks.workunit.client.0.smithi098.stdout: LINK /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/tools/objtool/objtool 2024-04-25T16:43:44.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:43 smithi203 ceph-mon[33115]: pgmap v708: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 KiB/s wr, 15 op/s 2024-04-25T16:43:44.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:43 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:rejoin 2024-04-25T16:43:44.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:43 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:rejoin 2024-04-25T16:43:44.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:43 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:rejoin 2024-04-25T16:43:44.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:43 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-04-25T16:43:44.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:43 smithi203 ceph-mon[33115]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:43:44.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:43 smithi203 ceph-mon[33115]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:43:44.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:43 smithi203 ceph-mon[33115]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-25T16:43:44.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:43 smithi098 ceph-mon[29162]: pgmap v708: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 KiB/s wr, 15 op/s 2024-04-25T16:43:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:43 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:rejoin 2024-04-25T16:43:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:43 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:rejoin 2024-04-25T16:43:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:43 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:rejoin 2024-04-25T16:43:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:43 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-04-25T16:43:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:43 smithi098 ceph-mon[29162]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:43:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:43 smithi098 ceph-mon[29162]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:43:44.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:43 smithi098 ceph-mon[29162]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-25T16:43:44.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:43 smithi173 ceph-mon[32556]: pgmap v708: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.5 KiB/s wr, 15 op/s 2024-04-25T16:43:44.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:43 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:rejoin 2024-04-25T16:43:44.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:43 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:rejoin 2024-04-25T16:43:44.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:43 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:rejoin 2024-04-25T16:43:44.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:43 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 2 up:standby-replay 5 up:standby 3 up:rejoin 2 up:active 2024-04-25T16:43:44.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:43 smithi173 ceph-mon[32556]: daemon mds.l is now active in filesystem cephfs as rank 2 2024-04-25T16:43:44.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:43 smithi173 ceph-mon[32556]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:43:44.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:43 smithi173 ceph-mon[32556]: daemon mds.k is now active in filesystem cephfs as rank 1 2024-04-25T16:43:45.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:44 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:45.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:44 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:43:45.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:44 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:active 2024-04-25T16:43:45.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:44 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:active 2024-04-25T16:43:45.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:44 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:active 2024-04-25T16:43:45.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:44 smithi203 ceph-mon[33115]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-25T16:43:45.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:44 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:45.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:44 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:45.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:44 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:43:45.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:44 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:active 2024-04-25T16:43:45.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:44 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:active 2024-04-25T16:43:45.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:44 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:active 2024-04-25T16:43:45.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:44 smithi098 ceph-mon[29162]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-25T16:43:45.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:44 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:45.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:44 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:43:45.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:44 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:43:45.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:44 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/2590142748,v1:172.21.15.203:6833/2590142748] up:active 2024-04-25T16:43:45.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:44 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/2689051587,v1:172.21.15.98:6839/2689051587] up:active 2024-04-25T16:43:45.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:44 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/2439698545,v1:172.21.15.173:6833/2439698545] up:active 2024-04-25T16:43:45.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:44 smithi173 ceph-mon[32556]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-04-25T16:43:45.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:44 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:43:46.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:45 smithi173 ceph-mon[32556]: pgmap v709: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 166 KiB/s wr, 32 op/s 2024-04-25T16:43:46.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:43:45 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:43:45.623+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:43:46.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:43:45 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:43:45.623+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:43:46.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:43:45 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:43:45.623+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:43:46.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:45 smithi203 ceph-mon[33115]: pgmap v709: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 166 KiB/s wr, 32 op/s 2024-04-25T16:43:46.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:45 smithi098 ceph-mon[29162]: pgmap v709: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 166 KiB/s wr, 32 op/s 2024-04-25T16:43:48.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:47 smithi203 ceph-mon[33115]: pgmap v710: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 139 KiB/s wr, 31 op/s 2024-04-25T16:43:48.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:47 smithi098 ceph-mon[29162]: pgmap v710: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 139 KiB/s wr, 31 op/s 2024-04-25T16:43:48.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:47 smithi173 ceph-mon[32556]: pgmap v710: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 139 KiB/s wr, 31 op/s 2024-04-25T16:43:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:48.271+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:43:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:48.271+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:43:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:48.271+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:43:48.269 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:48.271+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:43:48.270 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:43:48.271+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:43:50.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:49 smithi203 ceph-mon[33115]: pgmap v711: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 254 KiB/s wr, 43 op/s 2024-04-25T16:43:50.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:49 smithi098 ceph-mon[29162]: pgmap v711: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 254 KiB/s wr, 43 op/s 2024-04-25T16:43:50.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:49 smithi173 ceph-mon[32556]: pgmap v711: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 254 KiB/s wr, 43 op/s 2024-04-25T16:43:52.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-mon[33115]: pgmap v712: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 412 KiB/s wr, 102 op/s 2024-04-25T16:43:52.187 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f599ed56700 time 2024-04-25T16:43:51.838046+0000 2024-04-25T16:43:52.187 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f59a958aba6] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a958add1] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55feae6cc26b] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55feae66b428] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55feae66c8a9] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55feae66e9c3] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55feae53a85d] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55feae53ce8e] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55feae53d894] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55feae52805b] 2024-04-25T16:43:52.188 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f59a9707ac6] 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 12: (DispatchQueue::entry()+0xb27) [0x7f59a97050b9] 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f59a97efccb] 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 14: (Thread::entry_wrapper()+0x43) [0x7f59a955972d] 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 15: (Thread::_entry_func(void*)+0xd) [0x7f59a9559749] 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f59a80fd1ca] 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 17: clone() 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: *** Caught signal (Aborted) ** 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: in thread 7f599ed56700 thread_name:ms_dispatch 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:51.839+0000 7f599ed56700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f599ed56700 time 2024-04-25T16:43:51.838046+0000 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:52.189 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f59a958aba6] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a958add1] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55feae6cc26b] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55feae66b428] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55feae66c8a9] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55feae66e9c3] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55feae53a85d] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55feae53ce8e] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55feae53d894] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55feae52805b] 2024-04-25T16:43:52.190 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f59a9707ac6] 2024-04-25T16:43:52.191 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 12: (DispatchQueue::entry()+0xb27) [0x7f59a97050b9] 2024-04-25T16:43:52.191 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f59a97efccb] 2024-04-25T16:43:52.191 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 14: (Thread::entry_wrapper()+0x43) [0x7f59a955972d] 2024-04-25T16:43:52.191 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 15: (Thread::_entry_func(void*)+0xd) [0x7f59a9559749] 2024-04-25T16:43:52.191 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f59a80fd1ca] 2024-04-25T16:43:52.191 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 17: clone() 2024-04-25T16:43:52.191 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.191 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.192 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59a8107d20] 2024-04-25T16:43:52.192 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2: gsignal() 2024-04-25T16:43:52.192 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 3: abort() 2024-04-25T16:43:52.192 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f59a958aced] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a958add1] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55feae6cc26b] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55feae66b428] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55feae66c8a9] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55feae66e9c3] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55feae53a85d] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55feae53ce8e] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55feae53d894] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55feae52805b] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f59a9707ac6] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 15: (DispatchQueue::entry()+0xb27) [0x7f59a97050b9] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f59a97efccb] 2024-04-25T16:43:52.193 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 17: (Thread::entry_wrapper()+0x43) [0x7f59a955972d] 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 18: (Thread::_entry_func(void*)+0xd) [0x7f59a9559749] 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f59a80fd1ca] 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 20: clone() 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:51.842+0000 7f599ed56700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: in thread 7f599ed56700 thread_name:ms_dispatch 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59a8107d20] 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2: gsignal() 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 3: abort() 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f59a958aced] 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a958add1] 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55feae6cc26b] 2024-04-25T16:43:52.194 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55feae66b428] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55feae66c8a9] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55feae66e9c3] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55feae53a85d] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55feae53ce8e] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55feae53d894] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55feae52805b] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f59a9707ac6] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 15: (DispatchQueue::entry()+0xb27) [0x7f59a97050b9] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f59a97efccb] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 17: (Thread::entry_wrapper()+0x43) [0x7f59a955972d] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 18: (Thread::_entry_func(void*)+0xd) [0x7f59a9559749] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f59a80fd1ca] 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 20: clone() 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:52.195 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: -1> 2024-04-25T16:43:51.839+0000 7f599ed56700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f599ed56700 time 2024-04-25T16:43:51.838046+0000 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f59a958aba6] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a958add1] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55feae6cc26b] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55feae66b428] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55feae66c8a9] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55feae66e9c3] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55feae53a85d] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55feae53ce8e] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55feae53d894] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55feae52805b] 2024-04-25T16:43:52.196 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f59a9707ac6] 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 12: (DispatchQueue::entry()+0xb27) [0x7f59a97050b9] 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f59a97efccb] 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 14: (Thread::entry_wrapper()+0x43) [0x7f59a955972d] 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 15: (Thread::_entry_func(void*)+0xd) [0x7f59a9559749] 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f59a80fd1ca] 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 17: clone() 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 0> 2024-04-25T16:43:51.842+0000 7f599ed56700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: in thread 7f599ed56700 thread_name:ms_dispatch 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59a8107d20] 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2: gsignal() 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 3: abort() 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f59a958aced] 2024-04-25T16:43:52.197 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a958add1] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55feae6cc26b] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55feae66b428] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55feae66c8a9] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55feae66e9c3] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55feae53a85d] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55feae53ce8e] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55feae53d894] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55feae52805b] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f59a9707ac6] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 15: (DispatchQueue::entry()+0xb27) [0x7f59a97050b9] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f59a97efccb] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 17: (Thread::entry_wrapper()+0x43) [0x7f59a955972d] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 18: (Thread::_entry_func(void*)+0xd) [0x7f59a9559749] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f59a80fd1ca] 2024-04-25T16:43:52.198 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 20: clone() 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: -5> 2024-04-25T16:43:51.839+0000 7f599ed56700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f599ed56700 time 2024-04-25T16:43:51.838046+0000 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f59a958aba6] 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a958add1] 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55feae6cc26b] 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55feae66b428] 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55feae66c8a9] 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55feae66e9c3] 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55feae53a85d] 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55feae53ce8e] 2024-04-25T16:43:52.199 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55feae53d894] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55feae52805b] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f59a9707ac6] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 12: (DispatchQueue::entry()+0xb27) [0x7f59a97050b9] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f59a97efccb] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 14: (Thread::entry_wrapper()+0x43) [0x7f59a955972d] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 15: (Thread::_entry_func(void*)+0xd) [0x7f59a9559749] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f59a80fd1ca] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 17: clone() 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: -4> 2024-04-25T16:43:51.842+0000 7f599ed56700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: in thread 7f599ed56700 thread_name:ms_dispatch 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59a8107d20] 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2: gsignal() 2024-04-25T16:43:52.200 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 3: abort() 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f59a958aced] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f59a958add1] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55feae6cc26b] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55feae66b428] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55feae66c8a9] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55feae66e9c3] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55feae53a85d] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55feae53ce8e] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55feae53d894] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55feae52805b] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f59a9707ac6] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 15: (DispatchQueue::entry()+0xb27) [0x7f59a97050b9] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f59a97efccb] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 17: (Thread::entry_wrapper()+0x43) [0x7f59a955972d] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 18: (Thread::_entry_func(void*)+0xd) [0x7f59a9559749] 2024-04-25T16:43:52.201 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f59a80fd1ca] 2024-04-25T16:43:52.202 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 20: clone() 2024-04-25T16:43:52.202 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:52.202 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[69364]: 2024-04-25T16:43:52.202 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:51 smithi203 podman[69868]: 2024-04-25 16:43:51.911800089 +0000 UTC m=+0.020110484 container died c42bda7d9a055314ab52d9fe3cb19bce0f99fa110db38416b00d6fb9d5bce38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-04-25T16:43:52.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:51 smithi098 ceph-mon[29162]: pgmap v712: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 412 KiB/s wr, 102 op/s 2024-04-25T16:43:52.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-mon[32556]: pgmap v712: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 412 KiB/s wr, 102 op/s 2024-04-25T16:43:52.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9a8652a700 time 2024-04-25T16:43:51.910350+0000 2024-04-25T16:43:52.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:52.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9a90d5eba6] 2024-04-25T16:43:52.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a90d5edd1] 2024-04-25T16:43:52.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5621aa27526b] 2024-04-25T16:43:52.263 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5621aa214428] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5621aa2158a9] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5621aa2179c3] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5621aa0e385d] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5621aa0e5e8e] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5621aa0e6894] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5621aa0d105b] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a90edbac6] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 12: (DispatchQueue::entry()+0xb27) [0x7f9a90ed90b9] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a90fc3ccb] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 14: (Thread::entry_wrapper()+0x43) [0x7f9a90d2d72d] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 15: (Thread::_entry_func(void*)+0xd) [0x7f9a90d2d749] 2024-04-25T16:43:52.264 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f9a8f8d11ca] 2024-04-25T16:43:52.265 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 17: clone() 2024-04-25T16:43:52.265 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: *** Caught signal (Aborted) ** 2024-04-25T16:43:52.265 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: in thread 7f9a8652a700 thread_name:ms_dispatch 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:51.912+0000 7f9a8652a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9a8652a700 time 2024-04-25T16:43:51.910350+0000 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9a90d5eba6] 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a90d5edd1] 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5621aa27526b] 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5621aa214428] 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5621aa2158a9] 2024-04-25T16:43:52.266 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5621aa2179c3] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5621aa0e385d] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5621aa0e5e8e] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5621aa0e6894] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5621aa0d105b] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a90edbac6] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 12: (DispatchQueue::entry()+0xb27) [0x7f9a90ed90b9] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a90fc3ccb] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 14: (Thread::entry_wrapper()+0x43) [0x7f9a90d2d72d] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 15: (Thread::_entry_func(void*)+0xd) [0x7f9a90d2d749] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f9a8f8d11ca] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 17: clone() 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9a8f8dbd20] 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2: gsignal() 2024-04-25T16:43:52.267 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 3: abort() 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9a90d5eced] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a90d5edd1] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5621aa27526b] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5621aa214428] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5621aa2158a9] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5621aa2179c3] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5621aa0e385d] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5621aa0e5e8e] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5621aa0e6894] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5621aa0d105b] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a90edbac6] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 15: (DispatchQueue::entry()+0xb27) [0x7f9a90ed90b9] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a90fc3ccb] 2024-04-25T16:43:52.268 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 17: (Thread::entry_wrapper()+0x43) [0x7f9a90d2d72d] 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 18: (Thread::_entry_func(void*)+0xd) [0x7f9a90d2d749] 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f9a8f8d11ca] 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 20: clone() 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:51.914+0000 7f9a8652a700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: in thread 7f9a8652a700 thread_name:ms_dispatch 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9a8f8dbd20] 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2: gsignal() 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 3: abort() 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9a90d5eced] 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a90d5edd1] 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5621aa27526b] 2024-04-25T16:43:52.269 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5621aa214428] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5621aa2158a9] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5621aa2179c3] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5621aa0e385d] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5621aa0e5e8e] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5621aa0e6894] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5621aa0d105b] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a90edbac6] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 15: (DispatchQueue::entry()+0xb27) [0x7f9a90ed90b9] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a90fc3ccb] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 17: (Thread::entry_wrapper()+0x43) [0x7f9a90d2d72d] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 18: (Thread::_entry_func(void*)+0xd) [0x7f9a90d2d749] 2024-04-25T16:43:52.270 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f9a8f8d11ca] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 20: clone() 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: -1> 2024-04-25T16:43:51.912+0000 7f9a8652a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9a8652a700 time 2024-04-25T16:43:51.910350+0000 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9a90d5eba6] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a90d5edd1] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5621aa27526b] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5621aa214428] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5621aa2158a9] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5621aa2179c3] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5621aa0e385d] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5621aa0e5e8e] 2024-04-25T16:43:52.271 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5621aa0e6894] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5621aa0d105b] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a90edbac6] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 12: (DispatchQueue::entry()+0xb27) [0x7f9a90ed90b9] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a90fc3ccb] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 14: (Thread::entry_wrapper()+0x43) [0x7f9a90d2d72d] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 15: (Thread::_entry_func(void*)+0xd) [0x7f9a90d2d749] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f9a8f8d11ca] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 17: clone() 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 0> 2024-04-25T16:43:51.914+0000 7f9a8652a700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: in thread 7f9a8652a700 thread_name:ms_dispatch 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9a8f8dbd20] 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2: gsignal() 2024-04-25T16:43:52.272 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 3: abort() 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9a90d5eced] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a90d5edd1] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5621aa27526b] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5621aa214428] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5621aa2158a9] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5621aa2179c3] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5621aa0e385d] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5621aa0e5e8e] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5621aa0e6894] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5621aa0d105b] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a90edbac6] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 15: (DispatchQueue::entry()+0xb27) [0x7f9a90ed90b9] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a90fc3ccb] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 17: (Thread::entry_wrapper()+0x43) [0x7f9a90d2d72d] 2024-04-25T16:43:52.273 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 18: (Thread::_entry_func(void*)+0xd) [0x7f9a90d2d749] 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f9a8f8d11ca] 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 20: clone() 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: -1> 2024-04-25T16:43:51.912+0000 7f9a8652a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f9a8652a700 time 2024-04-25T16:43:51.910350+0000 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f9a90d5eba6] 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a90d5edd1] 2024-04-25T16:43:52.274 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x5621aa27526b] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5621aa214428] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5621aa2158a9] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5621aa2179c3] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5621aa0e385d] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5621aa0e5e8e] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5621aa0e6894] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5621aa0d105b] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a90edbac6] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 12: (DispatchQueue::entry()+0xb27) [0x7f9a90ed90b9] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a90fc3ccb] 2024-04-25T16:43:52.275 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 14: (Thread::entry_wrapper()+0x43) [0x7f9a90d2d72d] 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 15: (Thread::_entry_func(void*)+0xd) [0x7f9a90d2d749] 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f9a8f8d11ca] 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 17: clone() 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 0> 2024-04-25T16:43:51.914+0000 7f9a8652a700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: in thread 7f9a8652a700 thread_name:ms_dispatch 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9a8f8dbd20] 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2: gsignal() 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 3: abort() 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f9a90d5eced] 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9a90d5edd1] 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x5621aa27526b] 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x5621aa214428] 2024-04-25T16:43:52.276 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x5621aa2158a9] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x5621aa2179c3] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x5621aa0e385d] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x5621aa0e5e8e] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x5621aa0e6894] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x5621aa0d105b] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f9a90edbac6] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 15: (DispatchQueue::entry()+0xb27) [0x7f9a90ed90b9] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a90fc3ccb] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 17: (Thread::entry_wrapper()+0x43) [0x7f9a90d2d72d] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 18: (Thread::_entry_func(void*)+0xd) [0x7f9a90d2d749] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f9a8f8d11ca] 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 20: clone() 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[67625]: 2024-04-25T16:43:52.277 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:51 smithi173 podman[67698]: 2024-04-25 16:43:51.972938365 +0000 UTC m=+0.019617944 container died 3ff4772f580a68114b2436de2cedd90e81114c5f68789eafd9854273e16aa0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T16:43:52.936 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:52 smithi203 podman[69868]: 2024-04-25 16:43:52.481668737 +0000 UTC m=+0.589979119 container remove c42bda7d9a055314ab52d9fe3cb19bce0f99fa110db38416b00d6fb9d5bce38a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T16:43:52.937 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:52 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:52.937 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:52 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Failed with result 'exit-code'. 2024-04-25T16:43:52.937 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:43:52 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.785s CPU time. 2024-04-25T16:43:53.013 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:52 smithi173 podman[67698]: 2024-04-25 16:43:52.724741507 +0000 UTC m=+0.771421142 container remove 3ff4772f580a68114b2436de2cedd90e81114c5f68789eafd9854273e16aa0e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T16:43:53.013 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:52 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:43:53.013 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:52 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Failed with result 'exit-code'. 2024-04-25T16:43:53.013 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:43:52 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Consumed 2.060s CPU time. 2024-04-25T16:43:54.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:53 smithi203 ceph-mon[33115]: pgmap v713: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 364 KiB/s wr, 90 op/s 2024-04-25T16:43:54.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:53 smithi098 ceph-mon[29162]: pgmap v713: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 364 KiB/s wr, 90 op/s 2024-04-25T16:43:54.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:53 smithi173 ceph-mon[32556]: pgmap v713: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 364 KiB/s wr, 90 op/s 2024-04-25T16:43:56.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:55 smithi203 ceph-mon[33115]: pgmap v714: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 393 KiB/s wr, 100 op/s 2024-04-25T16:43:56.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:55 smithi098 ceph-mon[29162]: pgmap v714: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 393 KiB/s wr, 100 op/s 2024-04-25T16:43:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:55 smithi173 ceph-mon[32556]: pgmap v714: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 393 KiB/s wr, 100 op/s 2024-04-25T16:43:58.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:57 smithi203 ceph-mon[33115]: pgmap v715: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 301 KiB/s wr, 89 op/s 2024-04-25T16:43:58.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:57 smithi098 ceph-mon[29162]: pgmap v715: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 301 KiB/s wr, 89 op/s 2024-04-25T16:43:58.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:57 smithi173 ceph-mon[32556]: pgmap v715: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 301 KiB/s wr, 89 op/s 2024-04-25T16:44:00.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:43:59 smithi203 ceph-mon[33115]: pgmap v716: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 300 KiB/s wr, 87 op/s 2024-04-25T16:44:00.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:43:59 smithi098 ceph-mon[29162]: pgmap v716: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 300 KiB/s wr, 87 op/s 2024-04-25T16:44:00.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:43:59 smithi173 ceph-mon[32556]: pgmap v716: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 300 KiB/s wr, 87 op/s 2024-04-25T16:44:02.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:01 smithi203 ceph-mon[33115]: pgmap v717: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 191 KiB/s wr, 76 op/s 2024-04-25T16:44:02.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:01 smithi098 ceph-mon[29162]: pgmap v717: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 191 KiB/s wr, 76 op/s 2024-04-25T16:44:02.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:01 smithi173 ceph-mon[32556]: pgmap v717: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 191 KiB/s wr, 76 op/s 2024-04-25T16:44:03.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:44:03.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.151 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.152 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:02 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:02 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:02 smithi203 systemd[1]: Stopped Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:02 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service: Consumed 1.785s CPU time. 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:02 smithi203 systemd[1]: Starting Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.180 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.181 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.181 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:02 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.240 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:44:03.240 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 systemd[1]: Stopped Ceph mds.k for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:03.240 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service: Consumed 2.060s CPU time. 2024-04-25T16:44:03.240 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 systemd[1]: Starting Ceph mds.k for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:44:03.240 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:44:03.240 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.241 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.241 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.241 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.241 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.241 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:02 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:03.278 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:03.280+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:03.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:03.280+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:44:03.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:03.280+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:44:03.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:03.280+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:44:03.279 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:03.280+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:44:03.436 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:03 smithi203 podman[70152]: 2024-04-25 16:44:03.085841424 +0000 UTC m=+0.018208855 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:44:03.436 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:03 smithi203 podman[70152]: 2024-04-25 16:44:03.208199205 +0000 UTC m=+0.140566635 container create 1e88eb729659811f4aa96843952a943faf477ffaad20126e65e5e48f96b08dd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T16:44:03.513 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 podman[67983]: 2024-04-25 16:44:03.146088547 +0000 UTC m=+0.018244709 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:44:03.513 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 podman[67983]: 2024-04-25 16:44:03.317627525 +0000 UTC m=+0.189783675 container create a61e5ea3dda99a8de771696af8485fe72d8735f5d481fdad2a06159e96a90511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T16:44:03.709 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:03 smithi203 podman[70152]: 2024-04-25 16:44:03.472158526 +0000 UTC m=+0.404525972 container init 1e88eb729659811f4aa96843952a943faf477ffaad20126e65e5e48f96b08dd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:03.709 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:03 smithi203 podman[70152]: 2024-04-25 16:44:03.474880935 +0000 UTC m=+0.407248368 container start 1e88eb729659811f4aa96843952a943faf477ffaad20126e65e5e48f96b08dd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:44:03.709 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mds[70204]: -- 172.21.15.203:0/3606820595 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba656a1a40 con 0x55ba656e9180 2024-04-25T16:44:03.709 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[70199]: starting mds.l at 2024-04-25T16:44:03.709 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:03 smithi203 bash[70152]: 1e88eb729659811f4aa96843952a943faf477ffaad20126e65e5e48f96b08dd1 2024-04-25T16:44:03.709 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:44:03 smithi203 systemd[1]: Started Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:03.796 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 podman[67983]: 2024-04-25 16:44:03.562685017 +0000 UTC m=+0.434841187 container init a61e5ea3dda99a8de771696af8485fe72d8735f5d481fdad2a06159e96a90511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:44:03.796 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 podman[67983]: 2024-04-25 16:44:03.565433618 +0000 UTC m=+0.437589777 container start a61e5ea3dda99a8de771696af8485fe72d8735f5d481fdad2a06159e96a90511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:03.796 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mds[68035]: -- 172.21.15.173:0/98646192 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563fccf5fa40 con 0x563fccfa7180 2024-04-25T16:44:03.796 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[68030]: starting mds.k at 2024-04-25T16:44:03.796 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 bash[67983]: a61e5ea3dda99a8de771696af8485fe72d8735f5d481fdad2a06159e96a90511 2024-04-25T16:44:03.796 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:44:03 smithi173 systemd[1]: Started Ceph mds.k for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:04.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: pgmap v718: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-25T16:44:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:44:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:44:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:04.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: daemon mds.l restarted 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6832/2679969166,v1:172.21.15.203:6833/2679969166] up:boot 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: daemon mds.k restarted 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6832/1861017755,v1:172.21.15.173:6833/1861017755] up:boot 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=h=up:active,2=i=up:replay,3=e=up:active,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:44:04.188 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:03 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: pgmap v718: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: daemon mds.l restarted 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6832/2679969166,v1:172.21.15.203:6833/2679969166] up:boot 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-25T16:44:04.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: daemon mds.k restarted 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6832/1861017755,v1:172.21.15.173:6833/1861017755] up:boot 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=h=up:active,2=i=up:replay,3=e=up:active,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:44:04.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:03 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: pgmap v718: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 42 KiB/s wr, 22 op/s 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: daemon mds.l restarted 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: osdmap e102: 12 total, 12 up, 12 in 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6832/2679969166,v1:172.21.15.203:6833/2679969166] up:boot 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: Standby daemon mds.i assigned to filesystem cephfs as rank 2 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=h=up:active,1=k=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:44:04.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:44:04.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: daemon mds.k restarted 2024-04-25T16:44:04.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: osdmap e103: 12 total, 12 up, 12 in 2024-04-25T16:44:04.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6832/1861017755,v1:172.21.15.173:6833/1861017755] up:boot 2024-04-25T16:44:04.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: Standby daemon mds.c assigned to filesystem cephfs as rank 1 2024-04-25T16:44:04.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=h=up:active,2=i=up:replay,3=e=up:active,4=g=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:44:04.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:44:04.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:03 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-04-25T16:44:06.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:05 smithi203 ceph-mon[33115]: pgmap v721: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-04-25T16:44:06.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:05 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:resolve 2024-04-25T16:44:06.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:05 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:resolve 2024-04-25T16:44:06.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:05 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:44:06.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:05 smithi098 ceph-mon[29162]: pgmap v721: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-04-25T16:44:06.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:05 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:resolve 2024-04-25T16:44:06.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:05 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:resolve 2024-04-25T16:44:06.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:05 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:44:06.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:05 smithi173 ceph-mon[32556]: pgmap v721: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 4.7 KiB/s wr, 13 op/s 2024-04-25T16:44:06.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:05 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:resolve 2024-04-25T16:44:06.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:05 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:resolve 2024-04-25T16:44:06.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:05 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:44:07.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:06 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:reconnect 2024-04-25T16:44:07.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:06 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:reconnect 2024-04-25T16:44:07.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:06 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:44:07.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:06 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0139997 2024-04-25T16:44:07.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:06 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0229995 2024-04-25T16:44:07.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:06 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:reconnect 2024-04-25T16:44:07.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:06 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:reconnect 2024-04-25T16:44:07.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:06 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:44:07.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:06 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0139997 2024-04-25T16:44:07.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:06 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0229995 2024-04-25T16:44:07.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:06 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:reconnect 2024-04-25T16:44:07.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:06 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:reconnect 2024-04-25T16:44:07.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:06 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:44:07.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:06 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0139997 2024-04-25T16:44:07.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:06 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0229995 2024-04-25T16:44:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:07 smithi203 ceph-mon[33115]: pgmap v722: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 14 op/s 2024-04-25T16:44:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:07 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:rejoin 2024-04-25T16:44:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:07 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:rejoin 2024-04-25T16:44:08.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:07 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:44:08.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:07 smithi203 ceph-mon[33115]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-25T16:44:08.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:07 smithi203 ceph-mon[33115]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:44:08.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:07 smithi098 ceph-mon[29162]: pgmap v722: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 14 op/s 2024-04-25T16:44:08.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:07 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:rejoin 2024-04-25T16:44:08.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:07 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:rejoin 2024-04-25T16:44:08.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:07 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:44:08.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:07 smithi098 ceph-mon[29162]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-25T16:44:08.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:07 smithi098 ceph-mon[29162]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:44:08.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:07 smithi173 ceph-mon[32556]: pgmap v722: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.0 KiB/s wr, 14 op/s 2024-04-25T16:44:08.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:07 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:rejoin 2024-04-25T16:44:08.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:07 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:rejoin 2024-04-25T16:44:08.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:07 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:44:08.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:07 smithi173 ceph-mon[32556]: daemon mds.c is now active in filesystem cephfs as rank 1 2024-04-25T16:44:08.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:07 smithi173 ceph-mon[32556]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-25T16:44:09.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:08 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:09.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:08 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:44:09.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:08 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:active 2024-04-25T16:44:09.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:08 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:active 2024-04-25T16:44:09.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:08 smithi203 ceph-mon[33115]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:44:09.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:08 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:09.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:08 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:09.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:08 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:44:09.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:08 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:active 2024-04-25T16:44:09.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:08 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:active 2024-04-25T16:44:09.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:08 smithi098 ceph-mon[29162]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:44:09.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:08 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:09.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:08 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:09.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:08 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:44:09.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:08 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6838/3663834844,v1:172.21.15.203:6839/3663834844] up:active 2024-04-25T16:44:09.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:08 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6836/1270588400,v1:172.21.15.203:6837/1270588400] up:active 2024-04-25T16:44:09.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:08 smithi173 ceph-mon[32556]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:44:09.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:08 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:09 smithi203 ceph-mon[33115]: pgmap v723: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 639 B/s wr, 15 op/s 2024-04-25T16:44:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:09 smithi098 ceph-mon[29162]: pgmap v723: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 639 B/s wr, 15 op/s 2024-04-25T16:44:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:09 smithi173 ceph-mon[32556]: pgmap v723: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 639 B/s wr, 15 op/s 2024-04-25T16:44:11.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:10.625+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:44:11.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:10.625+0000 7f06773a7700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:44:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:11 smithi203 ceph-mon[33115]: pgmap v724: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 19 KiB/s wr, 33 op/s 2024-04-25T16:44:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:11 smithi098 ceph-mon[29162]: pgmap v724: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 19 KiB/s wr, 33 op/s 2024-04-25T16:44:12.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:11 smithi173 ceph-mon[32556]: pgmap v724: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 19 KiB/s wr, 33 op/s 2024-04-25T16:44:14.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:13 smithi203 ceph-mon[33115]: pgmap v725: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 18 KiB/s wr, 32 op/s 2024-04-25T16:44:14.230 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:13 smithi173 ceph-mon[32556]: pgmap v725: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 18 KiB/s wr, 32 op/s 2024-04-25T16:44:14.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:13 smithi098 ceph-mon[29162]: pgmap v725: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 18 KiB/s wr, 32 op/s 2024-04-25T16:44:15.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0679bac700 time 2024-04-25T16:44:14.693458+0000 2024-04-25T16:44:15.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:15.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f06843e0ba6] 2024-04-25T16:44:15.013 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06843e0dd1] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x563e8923d26b] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563e891dc428] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563e891dd8a9] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563e891df9c3] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563e890ab85d] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563e890ade8e] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563e890ae894] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563e8909905b] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f068455dac6] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 12: (DispatchQueue::entry()+0xb27) [0x7f068455b0b9] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0684645ccb] 2024-04-25T16:44:15.014 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 14: (Thread::entry_wrapper()+0x43) [0x7f06843af72d] 2024-04-25T16:44:15.015 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 15: (Thread::_entry_func(void*)+0xd) [0x7f06843af749] 2024-04-25T16:44:15.015 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0682f531ca] 2024-04-25T16:44:15.015 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 17: clone() 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:14.694+0000 7f0679bac700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0679bac700 time 2024-04-25T16:44:14.693458+0000 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f06843e0ba6] 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06843e0dd1] 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x563e8923d26b] 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563e891dc428] 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563e891dd8a9] 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563e891df9c3] 2024-04-25T16:44:15.016 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563e890ab85d] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563e890ade8e] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563e890ae894] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563e8909905b] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f068455dac6] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 12: (DispatchQueue::entry()+0xb27) [0x7f068455b0b9] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0684645ccb] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 14: (Thread::entry_wrapper()+0x43) [0x7f06843af72d] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 15: (Thread::_entry_func(void*)+0xd) [0x7f06843af749] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0682f531ca] 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 17: clone() 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: *** Caught signal (Aborted) ** 2024-04-25T16:44:15.017 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: in thread 7f0679bac700 thread_name:ms_dispatch 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0682f5dd20] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2: gsignal() 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 3: abort() 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f06843e0ced] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06843e0dd1] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x563e8923d26b] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563e891dc428] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563e891dd8a9] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563e891df9c3] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563e890ab85d] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563e890ade8e] 2024-04-25T16:44:15.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563e890ae894] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563e8909905b] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f068455dac6] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 15: (DispatchQueue::entry()+0xb27) [0x7f068455b0b9] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0684645ccb] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 17: (Thread::entry_wrapper()+0x43) [0x7f06843af72d] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 18: (Thread::_entry_func(void*)+0xd) [0x7f06843af749] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0682f531ca] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 20: clone() 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:14.697+0000 7f0679bac700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: in thread 7f0679bac700 thread_name:ms_dispatch 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0682f5dd20] 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2: gsignal() 2024-04-25T16:44:15.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 3: abort() 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f06843e0ced] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06843e0dd1] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x563e8923d26b] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563e891dc428] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563e891dd8a9] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563e891df9c3] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563e890ab85d] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563e890ade8e] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563e890ae894] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563e8909905b] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f068455dac6] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 15: (DispatchQueue::entry()+0xb27) [0x7f068455b0b9] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0684645ccb] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 17: (Thread::entry_wrapper()+0x43) [0x7f06843af72d] 2024-04-25T16:44:15.020 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 18: (Thread::_entry_func(void*)+0xd) [0x7f06843af749] 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0682f531ca] 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 20: clone() 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: -1> 2024-04-25T16:44:14.694+0000 7f0679bac700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0679bac700 time 2024-04-25T16:44:14.693458+0000 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f06843e0ba6] 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06843e0dd1] 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x563e8923d26b] 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563e891dc428] 2024-04-25T16:44:15.021 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563e891dd8a9] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563e891df9c3] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563e890ab85d] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563e890ade8e] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563e890ae894] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563e8909905b] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f068455dac6] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 12: (DispatchQueue::entry()+0xb27) [0x7f068455b0b9] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0684645ccb] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 14: (Thread::entry_wrapper()+0x43) [0x7f06843af72d] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 15: (Thread::_entry_func(void*)+0xd) [0x7f06843af749] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0682f531ca] 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 17: clone() 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.022 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 0> 2024-04-25T16:44:14.697+0000 7f0679bac700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: in thread 7f0679bac700 thread_name:ms_dispatch 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0682f5dd20] 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2: gsignal() 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 3: abort() 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f06843e0ced] 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06843e0dd1] 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x563e8923d26b] 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563e891dc428] 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563e891dd8a9] 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563e891df9c3] 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563e890ab85d] 2024-04-25T16:44:15.023 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563e890ade8e] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563e890ae894] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563e8909905b] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f068455dac6] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 15: (DispatchQueue::entry()+0xb27) [0x7f068455b0b9] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0684645ccb] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 17: (Thread::entry_wrapper()+0x43) [0x7f06843af72d] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 18: (Thread::_entry_func(void*)+0xd) [0x7f06843af749] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0682f531ca] 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 20: clone() 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: -1> 2024-04-25T16:44:14.694+0000 7f0679bac700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f0679bac700 time 2024-04-25T16:44:14.693458+0000 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:15.024 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.025 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.025 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f06843e0ba6] 2024-04-25T16:44:15.025 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06843e0dd1] 2024-04-25T16:44:15.025 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x563e8923d26b] 2024-04-25T16:44:15.025 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563e891dc428] 2024-04-25T16:44:15.025 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563e891dd8a9] 2024-04-25T16:44:15.025 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563e891df9c3] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563e890ab85d] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563e890ade8e] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563e890ae894] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563e8909905b] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f068455dac6] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 12: (DispatchQueue::entry()+0xb27) [0x7f068455b0b9] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0684645ccb] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 14: (Thread::entry_wrapper()+0x43) [0x7f06843af72d] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 15: (Thread::_entry_func(void*)+0xd) [0x7f06843af749] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f0682f531ca] 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 17: clone() 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.026 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 0> 2024-04-25T16:44:14.697+0000 7f0679bac700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: in thread 7f0679bac700 thread_name:ms_dispatch 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0682f5dd20] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2: gsignal() 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 3: abort() 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f06843e0ced] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f06843e0dd1] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x563e8923d26b] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x563e891dc428] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x563e891dd8a9] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x563e891df9c3] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x563e890ab85d] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x563e890ade8e] 2024-04-25T16:44:15.027 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x563e890ae894] 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x563e8909905b] 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f068455dac6] 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 15: (DispatchQueue::entry()+0xb27) [0x7f068455b0b9] 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0684645ccb] 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 17: (Thread::entry_wrapper()+0x43) [0x7f06843af72d] 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 18: (Thread::_entry_func(void*)+0xd) [0x7f06843af749] 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f0682f531ca] 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 20: clone() 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[66496]: 2024-04-25T16:44:15.028 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:14 smithi173 podman[68087]: 2024-04-25 16:44:14.762600347 +0000 UTC m=+0.018902098 container died 33e77d5898bc8a7ed94d6d6ac8b9d06abe63211843bceef294f25337dd286cfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:15.251 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fea42ac2700 time 2024-04-25T16:44:14.999107+0000 2024-04-25T16:44:15.251 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fea4d2f6ba6] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fea4d2f6dd1] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56215209a26b] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562152039428] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56215203a8a9] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56215203c9c3] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x562151f0885d] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x562151f0ae8e] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x562151f0b894] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x562151ef605b] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fea4d473ac6] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 12: (DispatchQueue::entry()+0xb27) [0x7fea4d4710b9] 2024-04-25T16:44:15.252 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea4d55bccb] 2024-04-25T16:44:15.253 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 14: (Thread::entry_wrapper()+0x43) [0x7fea4d2c572d] 2024-04-25T16:44:15.253 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 15: (Thread::_entry_func(void*)+0xd) [0x7fea4d2c5749] 2024-04-25T16:44:15.253 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fea4be691ca] 2024-04-25T16:44:15.253 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 17: clone() 2024-04-25T16:44:15.253 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: *** Caught signal (Aborted) ** 2024-04-25T16:44:15.253 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: in thread 7fea42ac2700 thread_name:ms_dispatch 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.000+0000 7fea42ac2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fea42ac2700 time 2024-04-25T16:44:14.999107+0000 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fea4d2f6ba6] 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fea4d2f6dd1] 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56215209a26b] 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562152039428] 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56215203a8a9] 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56215203c9c3] 2024-04-25T16:44:15.254 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x562151f0885d] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x562151f0ae8e] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x562151f0b894] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x562151ef605b] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fea4d473ac6] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 12: (DispatchQueue::entry()+0xb27) [0x7fea4d4710b9] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea4d55bccb] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 14: (Thread::entry_wrapper()+0x43) [0x7fea4d2c572d] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 15: (Thread::_entry_func(void*)+0xd) [0x7fea4d2c5749] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fea4be691ca] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 17: clone() 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fea4be73d20] 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2: gsignal() 2024-04-25T16:44:15.255 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 3: abort() 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fea4d2f6ced] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fea4d2f6dd1] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56215209a26b] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562152039428] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56215203a8a9] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56215203c9c3] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x562151f0885d] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x562151f0ae8e] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x562151f0b894] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x562151ef605b] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fea4d473ac6] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 15: (DispatchQueue::entry()+0xb27) [0x7fea4d4710b9] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea4d55bccb] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 17: (Thread::entry_wrapper()+0x43) [0x7fea4d2c572d] 2024-04-25T16:44:15.256 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 18: (Thread::_entry_func(void*)+0xd) [0x7fea4d2c5749] 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fea4be691ca] 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 20: clone() 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.003+0000 7fea42ac2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: in thread 7fea42ac2700 thread_name:ms_dispatch 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fea4be73d20] 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2: gsignal() 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 3: abort() 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fea4d2f6ced] 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fea4d2f6dd1] 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56215209a26b] 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562152039428] 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56215203a8a9] 2024-04-25T16:44:15.257 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56215203c9c3] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x562151f0885d] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x562151f0ae8e] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x562151f0b894] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x562151ef605b] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fea4d473ac6] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 15: (DispatchQueue::entry()+0xb27) [0x7fea4d4710b9] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea4d55bccb] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 17: (Thread::entry_wrapper()+0x43) [0x7fea4d2c572d] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 18: (Thread::_entry_func(void*)+0xd) [0x7fea4d2c5749] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fea4be691ca] 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 20: clone() 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: -1> 2024-04-25T16:44:15.000+0000 7fea42ac2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fea42ac2700 time 2024-04-25T16:44:14.999107+0000 2024-04-25T16:44:15.258 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fea4d2f6ba6] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fea4d2f6dd1] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56215209a26b] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562152039428] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56215203a8a9] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56215203c9c3] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x562151f0885d] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x562151f0ae8e] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x562151f0b894] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x562151ef605b] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fea4d473ac6] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 12: (DispatchQueue::entry()+0xb27) [0x7fea4d4710b9] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea4d55bccb] 2024-04-25T16:44:15.259 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 14: (Thread::entry_wrapper()+0x43) [0x7fea4d2c572d] 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 15: (Thread::_entry_func(void*)+0xd) [0x7fea4d2c5749] 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fea4be691ca] 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 17: clone() 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 0> 2024-04-25T16:44:15.003+0000 7fea42ac2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: in thread 7fea42ac2700 thread_name:ms_dispatch 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fea4be73d20] 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2: gsignal() 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 3: abort() 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fea4d2f6ced] 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fea4d2f6dd1] 2024-04-25T16:44:15.260 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56215209a26b] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562152039428] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56215203a8a9] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56215203c9c3] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x562151f0885d] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x562151f0ae8e] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x562151f0b894] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x562151ef605b] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fea4d473ac6] 2024-04-25T16:44:15.261 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 15: (DispatchQueue::entry()+0xb27) [0x7fea4d4710b9] 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea4d55bccb] 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 17: (Thread::entry_wrapper()+0x43) [0x7fea4d2c572d] 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 18: (Thread::_entry_func(void*)+0xd) [0x7fea4d2c5749] 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fea4be691ca] 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 20: clone() 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: -1> 2024-04-25T16:44:15.000+0000 7fea42ac2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7fea42ac2700 time 2024-04-25T16:44:14.999107+0000 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7fea4d2f6ba6] 2024-04-25T16:44:15.262 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fea4d2f6dd1] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x56215209a26b] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562152039428] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56215203a8a9] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56215203c9c3] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x562151f0885d] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x562151f0ae8e] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x562151f0b894] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x562151ef605b] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fea4d473ac6] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 12: (DispatchQueue::entry()+0xb27) [0x7fea4d4710b9] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea4d55bccb] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 14: (Thread::entry_wrapper()+0x43) [0x7fea4d2c572d] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 15: (Thread::_entry_func(void*)+0xd) [0x7fea4d2c5749] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7fea4be691ca] 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 17: clone() 2024-04-25T16:44:15.263 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 0> 2024-04-25T16:44:15.003+0000 7fea42ac2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: in thread 7fea42ac2700 thread_name:ms_dispatch 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fea4be73d20] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2: gsignal() 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 3: abort() 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7fea4d2f6ced] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fea4d2f6dd1] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x56215209a26b] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x562152039428] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x56215203a8a9] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x56215203c9c3] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x562151f0885d] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x562151f0ae8e] 2024-04-25T16:44:15.264 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x562151f0b894] 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x562151ef605b] 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7fea4d473ac6] 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 15: (DispatchQueue::entry()+0xb27) [0x7fea4d4710b9] 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fea4d55bccb] 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 17: (Thread::entry_wrapper()+0x43) [0x7fea4d2c572d] 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 18: (Thread::_entry_func(void*)+0xd) [0x7fea4d2c5749] 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7fea4be691ca] 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 20: clone() 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[88755]: 2024-04-25T16:44:15.265 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 podman[91091]: 2024-04-25 16:44:15.063086917 +0000 UTC m=+0.020308992 container died ab3b17b7be703e895cf80e1c91015f38433ec57b487aac8c0399c2e1766f0b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T16:44:15.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:15 smithi173 podman[68087]: 2024-04-25 16:44:15.427693287 +0000 UTC m=+0.683995028 container remove 33e77d5898bc8a7ed94d6d6ac8b9d06abe63211843bceef294f25337dd286cfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:15.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:15 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:44:15.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:15 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Failed with result 'exit-code'. 2024-04-25T16:44:15.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:15 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Consumed 3min 13.901s CPU time. 2024-04-25T16:44:16.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:15 smithi203 ceph-mon[33115]: pgmap v726: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 90 KiB/s wr, 67 op/s 2024-04-25T16:44:16.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:15 smithi098 ceph-mon[29162]: pgmap v726: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 90 KiB/s wr, 67 op/s 2024-04-25T16:44:16.251 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 podman[91091]: 2024-04-25 16:44:15.760601066 +0000 UTC m=+0.717823186 container remove ab3b17b7be703e895cf80e1c91015f38433ec57b487aac8c0399c2e1766f0b80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0) 2024-04-25T16:44:16.251 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:15 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:44:16.251 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:16 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Failed with result 'exit-code'. 2024-04-25T16:44:16.251 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:16 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Consumed 5.576s CPU time. 2024-04-25T16:44:16.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:15 smithi173 ceph-mon[32556]: pgmap v726: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 90 KiB/s wr, 67 op/s 2024-04-25T16:44:18.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:17 smithi203 ceph-mon[33115]: pgmap v727: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 83 KiB/s wr, 75 op/s 2024-04-25T16:44:18.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:17 smithi098 ceph-mon[29162]: pgmap v727: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 83 KiB/s wr, 75 op/s 2024-04-25T16:44:18.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:17 smithi173 ceph-mon[32556]: pgmap v727: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 83 KiB/s wr, 75 op/s 2024-04-25T16:44:18.291 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:18.293+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:18.294+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:44:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:18.294+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:44:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:18.294+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:44:18.292 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:18.294+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:44:20.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:19 smithi203 ceph-mon[33115]: pgmap v728: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 97 KiB/s wr, 99 op/s 2024-04-25T16:44:20.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:19 smithi098 ceph-mon[29162]: pgmap v728: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 97 KiB/s wr, 99 op/s 2024-04-25T16:44:20.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:19 smithi173 ceph-mon[32556]: pgmap v728: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 97 KiB/s wr, 99 op/s 2024-04-25T16:44:22.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:21 smithi203 ceph-mon[33115]: pgmap v729: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 99 KiB/s wr, 129 op/s 2024-04-25T16:44:22.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:21 smithi098 ceph-mon[29162]: pgmap v729: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 99 KiB/s wr, 129 op/s 2024-04-25T16:44:22.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:21 smithi173 ceph-mon[32556]: pgmap v729: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 99 KiB/s wr, 129 op/s 2024-04-25T16:44:22.443 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:22.445+0000 7fdebd7fa640 1 -- 172.21.15.98:0/2834070656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fdea0005d00 con 0x7fdec8101760 2024-04-25T16:44:22.444 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:22.446+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fdea0005d00 con 0x7fdec8101760 2024-04-25T16:44:22.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:22.449+0000 7fdea77fe640 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fde98005d20 con 0x55e32644d4d0 2024-04-25T16:44:22.447 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:22.449+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fde98005d20 con 0x55e32644d4d0 2024-04-25T16:44:24.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:23 smithi203 ceph-mon[33115]: pgmap v730: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 94 KiB/s wr, 118 op/s 2024-04-25T16:44:24.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:23 smithi098 ceph-mon[29162]: pgmap v730: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 94 KiB/s wr, 118 op/s 2024-04-25T16:44:24.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:23 smithi173 ceph-mon[32556]: pgmap v730: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 94 KiB/s wr, 118 op/s 2024-04-25T16:44:26.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:25 smithi173 ceph-mon[32556]: pgmap v731: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 94 KiB/s wr, 117 op/s 2024-04-25T16:44:26.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:25 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:44:26.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:25 smithi173 systemd[1]: Stopped Ceph mds.h for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:26.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:25 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service: Consumed 3min 13.901s CPU time. 2024-04-25T16:44:26.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:25 smithi173 systemd[1]: Starting Ceph mds.h for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:44:26.018 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:26 smithi173 podman[68183]: 2024-04-25 16:44:25.904701851 +0000 UTC m=+0.018492966 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:44:26.019 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:26 smithi173 podman[68183]: 2024-04-25 16:44:26.020582847 +0000 UTC m=+0.134373939 container create ddc980f52dbd50e2d6b55b122d63cdbbf7d68af1a7cac594eb44be4b59eab257 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T16:44:26.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:25 smithi203 ceph-mon[33115]: pgmap v731: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 94 KiB/s wr, 117 op/s 2024-04-25T16:44:26.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:25 smithi098 ceph-mon[29162]: pgmap v731: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 94 KiB/s wr, 117 op/s 2024-04-25T16:44:26.247 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-04-25T16:44:26.247 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 systemd[1]: Stopped Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:26.247 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service: Consumed 5.576s CPU time. 2024-04-25T16:44:26.247 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 systemd[1]: Starting Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:44:26.501 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 podman[91189]: 2024-04-25 16:44:26.152902697 +0000 UTC m=+0.019130511 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:44:26.501 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 podman[91189]: 2024-04-25 16:44:26.287528652 +0000 UTC m=+0.153756511 container create 6a9cb0b7c61d9462670e9439fddf7100b031df9e56cd9bab086043896edddc2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:44:26.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:26 smithi173 podman[68183]: 2024-04-25 16:44:26.307850937 +0000 UTC m=+0.421642037 container init ddc980f52dbd50e2d6b55b122d63cdbbf7d68af1a7cac594eb44be4b59eab257 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T16:44:26.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:26 smithi173 podman[68183]: 2024-04-25 16:44:26.310487904 +0000 UTC m=+0.424278997 container start ddc980f52dbd50e2d6b55b122d63cdbbf7d68af1a7cac594eb44be4b59eab257 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:44:26.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mds[68202]: -- 172.21.15.173:0/2571729367 <== mon.2 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eaaecf9a40 con 0x55eaaed41180 2024-04-25T16:44:26.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[68198]: starting mds.h at 2024-04-25T16:44:26.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:26 smithi173 bash[68183]: ddc980f52dbd50e2d6b55b122d63cdbbf7d68af1a7cac594eb44be4b59eab257 2024-04-25T16:44:26.763 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:44:26 smithi173 systemd[1]: Started Ceph mds.h for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:26.800 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 podman[91189]: 2024-04-25 16:44:26.532468638 +0000 UTC m=+0.398696454 container init 6a9cb0b7c61d9462670e9439fddf7100b031df9e56cd9bab086043896edddc2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:44:26.800 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 podman[91189]: 2024-04-25 16:44:26.536662942 +0000 UTC m=+0.402890756 container start 6a9cb0b7c61d9462670e9439fddf7100b031df9e56cd9bab086043896edddc2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:44:26.800 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mds[91209]: -- 172.21.15.98:0/2819789193 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556daf51a40 con 0x5556daf99600 2024-04-25T16:44:26.800 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[91203]: starting mds.g at 2024-04-25T16:44:26.800 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 bash[91189]: 6a9cb0b7c61d9462670e9439fddf7100b031df9e56cd9bab086043896edddc2c 2024-04-25T16:44:26.800 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:44:26 smithi098 systemd[1]: Started Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:44:27.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: daemon mds.h restarted 2024-04-25T16:44:27.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:27.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6836/3615606309,v1:172.21.15.173:6837/3615606309] up:boot 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {1=c=up:active,2=i=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: daemon mds.g restarted 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:resolve 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:boot 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: Standby daemon mds.f assigned to filesystem cephfs as rank 4 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=b=up:resolve,1=c=up:active,2=i=up:active,3=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:44:27.252 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:26 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:44:27.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: daemon mds.h restarted 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6836/3615606309,v1:172.21.15.173:6837/3615606309] up:boot 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {1=c=up:active,2=i=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: daemon mds.g restarted 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:resolve 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:boot 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: Standby daemon mds.f assigned to filesystem cephfs as rank 4 2024-04-25T16:44:27.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=b=up:resolve,1=c=up:active,2=i=up:active,3=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:44:27.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:44:27.264 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:26 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:44:27.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: daemon mds.h restarted 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: osdmap e104: 12 total, 12 up, 12 in 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6836/3615606309,v1:172.21.15.173:6837/3615606309] up:boot 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: Standby daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {1=c=up:active,2=i=up:active,3=e=up:active,4=g=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: daemon mds.g restarted 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: osdmap e105: 12 total, 12 up, 12 in 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:resolve 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:boot 2024-04-25T16:44:27.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: Standby daemon mds.f assigned to filesystem cephfs as rank 4 2024-04-25T16:44:27.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=b=up:resolve,1=c=up:active,2=i=up:active,3=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-04-25T16:44:27.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:44:27.438 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:26 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-04-25T16:44:28.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:27 smithi098 ceph-mon[29162]: pgmap v732: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 28 KiB/s wr, 83 op/s 2024-04-25T16:44:28.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:27 smithi173 ceph-mon[32556]: pgmap v732: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 28 KiB/s wr, 83 op/s 2024-04-25T16:44:28.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:27 smithi203 ceph-mon[33115]: pgmap v732: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 28 KiB/s wr, 83 op/s 2024-04-25T16:44:29.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:28 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:resolve 2024-04-25T16:44:29.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:28 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:44:29.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:28 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:resolve 2024-04-25T16:44:29.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:28 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:44:29.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:28 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:resolve 2024-04-25T16:44:29.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:28 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-04-25T16:44:30.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:29 smithi098 ceph-mon[29162]: pgmap v735: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-25T16:44:30.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:29 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:reconnect 2024-04-25T16:44:30.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:29 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:reconnect 2024-04-25T16:44:30.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:29 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:44:30.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:29 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0239995 2024-04-25T16:44:30.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:29 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0249994 2024-04-25T16:44:30.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:29 smithi173 ceph-mon[32556]: pgmap v735: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-25T16:44:30.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:29 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:reconnect 2024-04-25T16:44:30.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:29 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:reconnect 2024-04-25T16:44:30.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:29 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:44:30.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:29 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0239995 2024-04-25T16:44:30.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:29 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0249994 2024-04-25T16:44:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:29 smithi203 ceph-mon[33115]: pgmap v735: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-04-25T16:44:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:29 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:reconnect 2024-04-25T16:44:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:29 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:reconnect 2024-04-25T16:44:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:29 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-04-25T16:44:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:29 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0239995 2024-04-25T16:44:30.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:29 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0249994 2024-04-25T16:44:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:30 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:rejoin 2024-04-25T16:44:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:30 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:rejoin 2024-04-25T16:44:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:30 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:44:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:30 smithi098 ceph-mon[29162]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T16:44:31.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:30 smithi098 ceph-mon[29162]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:44:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:30 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:rejoin 2024-04-25T16:44:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:30 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:rejoin 2024-04-25T16:44:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:30 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:44:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:30 smithi173 ceph-mon[32556]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T16:44:31.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:30 smithi173 ceph-mon[32556]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:44:31.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:30 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:rejoin 2024-04-25T16:44:31.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:30 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:rejoin 2024-04-25T16:44:31.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:30 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-04-25T16:44:31.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:30 smithi203 ceph-mon[33115]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-04-25T16:44:31.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:30 smithi203 ceph-mon[33115]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T16:44:32.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:31 smithi098 ceph-mon[29162]: pgmap v736: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.7 KiB/s wr, 44 op/s 2024-04-25T16:44:32.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:31 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:32.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:31 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:44:32.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:31 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:active 2024-04-25T16:44:32.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:31 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:active 2024-04-25T16:44:32.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:31 smithi098 ceph-mon[29162]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:44:32.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:31 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:32.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:31 smithi173 ceph-mon[32556]: pgmap v736: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.7 KiB/s wr, 44 op/s 2024-04-25T16:44:32.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:31 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:32.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:31 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:44:32.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:31 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:active 2024-04-25T16:44:32.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:31 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:active 2024-04-25T16:44:32.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:31 smithi173 ceph-mon[32556]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:44:32.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:31 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:32.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:31 smithi203 ceph-mon[33115]: pgmap v736: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 1.7 KiB/s wr, 44 op/s 2024-04-25T16:44:32.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:31 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:44:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:31 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:44:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:31 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/3629133920,v1:172.21.15.203:6835/3629133920] up:active 2024-04-25T16:44:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:31 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.173:6838/3695246205,v1:172.21.15.173:6839/3695246205] up:active 2024-04-25T16:44:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:31 smithi203 ceph-mon[33115]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-04-25T16:44:32.437 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:31 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:33.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:33.309+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:33.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:33.309+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:44:33.307 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:33.309+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:44:33.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:33.309+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:44:33.308 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:33.309+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:44:34.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:33 smithi098 ceph-mon[29162]: pgmap v737: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 KiB/s wr, 50 op/s 2024-04-25T16:44:34.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:33 smithi173 ceph-mon[32556]: pgmap v737: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 KiB/s wr, 50 op/s 2024-04-25T16:44:34.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:33 smithi203 ceph-mon[33115]: pgmap v737: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.7 KiB/s wr, 50 op/s 2024-04-25T16:44:36.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:35 smithi173 ceph-mon[32556]: pgmap v738: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 184 KiB/s wr, 738 op/s 2024-04-25T16:44:36.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:44:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: 2024-04-25T16:44:35.740+0000 7f2e273c4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T16:44:36.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:44:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: 2024-04-25T16:44:35.740+0000 7f2e273c4700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T16:44:36.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:44:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: 2024-04-25T16:44:35.740+0000 7f2e273c4700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-25T16:44:36.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:44:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: 2024-04-25T16:44:35.740+0000 7f2e273c4700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T16:44:36.013 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:44:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: 2024-04-25T16:44:35.740+0000 7f2e273c4700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:44:36.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:35 smithi098 ceph-mon[29162]: pgmap v738: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 184 KiB/s wr, 738 op/s 2024-04-25T16:44:36.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:35 smithi203 ceph-mon[33115]: pgmap v738: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 184 KiB/s wr, 738 op/s 2024-04-25T16:44:37.249 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:36 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:37.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:36 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:36 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:44:38.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:37 smithi098 ceph-mon[29162]: pgmap v739: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 170 KiB/s wr, 810 op/s 2024-04-25T16:44:38.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:37 smithi173 ceph-mon[32556]: pgmap v739: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 170 KiB/s wr, 810 op/s 2024-04-25T16:44:38.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:37 smithi203 ceph-mon[33115]: pgmap v739: 129 pgs: 129 active+clean; 1.6 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 170 KiB/s wr, 810 op/s 2024-04-25T16:44:40.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:39 smithi098 ceph-mon[29162]: pgmap v740: 129 pgs: 129 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 227 KiB/s wr, 959 op/s 2024-04-25T16:44:40.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:39 smithi173 ceph-mon[32556]: pgmap v740: 129 pgs: 129 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 227 KiB/s wr, 959 op/s 2024-04-25T16:44:40.375 INFO:tasks.workunit.client.0.smithi098.stdout: CC scripts/mod/empty.o 2024-04-25T16:44:40.375 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/mod/mk_elfconfig 2024-04-25T16:44:40.375 INFO:tasks.workunit.client.0.smithi098.stdout: CC scripts/mod/devicetable-offsets.s 2024-04-25T16:44:40.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:39 smithi203 ceph-mon[33115]: pgmap v740: 129 pgs: 129 active+clean; 1.6 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 227 KiB/s wr, 959 op/s 2024-04-25T16:44:40.541 INFO:tasks.workunit.client.0.smithi098.stdout: MKELF scripts/mod/elfconfig.h 2024-04-25T16:44:40.555 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/mod/modpost.o 2024-04-25T16:44:40.555 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/mod/sumversion.o 2024-04-25T16:44:40.935 INFO:tasks.workunit.client.0.smithi098.stdout: UPD scripts/mod/devicetable-offsets.h 2024-04-25T16:44:40.954 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC scripts/mod/file2alias.o 2024-04-25T16:44:41.422 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTLD scripts/mod/modpost 2024-04-25T16:44:41.736 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/bounds.s 2024-04-25T16:44:41.750 INFO:tasks.workunit.client.0.smithi098.stdout: CHKSHA1 include/linux/atomic/atomic-instrumented.h 2024-04-25T16:44:41.751 INFO:tasks.workunit.client.0.smithi098.stdout: CHKSHA1 include/linux/atomic/atomic-arch-fallback.h 2024-04-25T16:44:41.751 INFO:tasks.workunit.client.0.smithi098.stdout: CHKSHA1 include/linux/atomic/atomic-long.h 2024-04-25T16:44:41.988 INFO:tasks.workunit.client.0.smithi098.stdout: UPD include/generated/timeconst.h 2024-04-25T16:44:42.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:41 smithi098 ceph-mon[29162]: pgmap v741: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 383 KiB/s wr, 1.27k op/s 2024-04-25T16:44:42.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:41 smithi173 ceph-mon[32556]: pgmap v741: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 383 KiB/s wr, 1.27k op/s 2024-04-25T16:44:42.324 INFO:tasks.workunit.client.0.smithi098.stdout: UPD include/generated/bounds.h 2024-04-25T16:44:42.341 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/asm-offsets.s 2024-04-25T16:44:42.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:41 smithi203 ceph-mon[33115]: pgmap v741: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 383 KiB/s wr, 1.27k op/s 2024-04-25T16:44:44.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:43 smithi098 ceph-mon[29162]: pgmap v742: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 481 KiB/s wr, 1.28k op/s 2024-04-25T16:44:44.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:43 smithi173 ceph-mon[32556]: pgmap v742: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 481 KiB/s wr, 1.28k op/s 2024-04-25T16:44:44.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:43 smithi203 ceph-mon[33115]: pgmap v742: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 481 KiB/s wr, 1.28k op/s 2024-04-25T16:44:45.389 INFO:tasks.workunit.client.0.smithi098.stdout: UPD include/generated/asm-offsets.h 2024-04-25T16:44:45.404 INFO:tasks.workunit.client.0.smithi098.stdout: CALL scripts/checksyscalls.sh 2024-04-25T16:44:45.531 INFO:tasks.workunit.client.0.smithi098.stdout: LDS scripts/module.lds 2024-04-25T16:44:45.935 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC usr/gen_init_cpio 2024-04-25T16:44:45.935 INFO:tasks.workunit.client.0.smithi098.stdout: CC certs/system_keyring.o 2024-04-25T16:44:45.940 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/compat.o 2024-04-25T16:44:45.943 INFO:tasks.workunit.client.0.smithi098.stdout: CC init/main.o 2024-04-25T16:44:46.174 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/filemap.o 2024-04-25T16:44:46.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:46 smithi098 ceph-mon[29162]: pgmap v743: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 508 KiB/s wr, 1.31k op/s 2024-04-25T16:44:46.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:46 smithi173 ceph-mon[32556]: pgmap v743: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 508 KiB/s wr, 1.31k op/s 2024-04-25T16:44:46.289 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/sched/core.o 2024-04-25T16:44:46.291 INFO:tasks.workunit.client.0.smithi098.stdout: GEN usr/initramfs_data.cpio 2024-04-25T16:44:46.297 INFO:tasks.workunit.client.0.smithi098.stdout: CC fs/notify/dnotify/dnotify.o 2024-04-25T16:44:46.363 INFO:tasks.workunit.client.0.smithi098.stdout: COPY usr/initramfs_inc_data 2024-04-25T16:44:46.364 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/vma.o 2024-04-25T16:44:46.387 INFO:tasks.workunit.client.0.smithi098.stdout: AS usr/initramfs_data.o 2024-04-25T16:44:46.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:46 smithi203 ceph-mon[33115]: pgmap v743: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s rd, 508 KiB/s wr, 1.31k op/s 2024-04-25T16:44:46.479 INFO:tasks.workunit.client.0.smithi098.stdout: AR usr/built-in.a 2024-04-25T16:44:46.563 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC certs/extract-cert 2024-04-25T16:44:47.006 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/gc.o 2024-04-25T16:44:48.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:48 smithi098 ceph-mon[29162]: pgmap v744: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 403 KiB/s wr, 861 op/s 2024-04-25T16:44:48.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:48 smithi173 ceph-mon[32556]: pgmap v744: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 403 KiB/s wr, 861 op/s 2024-04-25T16:44:48.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:48.325+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:44:48.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:48.325+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:44:48.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:48.325+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:44:48.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:48.325+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:44:48.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:44:48.325+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:44:48.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:48 smithi203 ceph-mon[33115]: pgmap v744: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 403 KiB/s wr, 861 op/s 2024-04-25T16:44:49.937 INFO:tasks.workunit.client.0.smithi098.stdout: CERT certs/x509_certificate_list 2024-04-25T16:44:49.972 INFO:tasks.workunit.client.0.smithi098.stdout: CERT certs/signing_key.x509 2024-04-25T16:44:50.024 INFO:tasks.workunit.client.0.smithi098.stdout: AS certs/system_certificates.o 2024-04-25T16:44:50.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:50 smithi173 ceph-mon[32556]: pgmap v745: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 440 KiB/s wr, 717 op/s 2024-04-25T16:44:50.351 INFO:tasks.workunit.client.0.smithi098.stdout: AR certs/built-in.a 2024-04-25T16:44:50.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:50 smithi203 ceph-mon[33115]: pgmap v745: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 440 KiB/s wr, 717 op/s 2024-04-25T16:44:50.480 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/key.o 2024-04-25T16:44:50.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:50 smithi098 ceph-mon[29162]: pgmap v745: 129 pgs: 129 active+clean; 1.6 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 440 KiB/s wr, 717 op/s 2024-04-25T16:44:51.553 INFO:tasks.workunit.client.0.smithi098.stdout: AR fs/notify/dnotify/built-in.a 2024-04-25T16:44:51.729 INFO:tasks.workunit.client.0.smithi098.stdout: CC fs/notify/inotify/inotify_fsnotify.o 2024-04-25T16:44:52.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:52 smithi173 ceph-mon[32556]: pgmap v746: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 374 KiB/s wr, 479 op/s 2024-04-25T16:44:52.276 INFO:tasks.workunit.client.0.smithi098.stdout: CC fs/notify/inotify/inotify_user.o 2024-04-25T16:44:52.372 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/util.o 2024-04-25T16:44:52.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:52 smithi203 ceph-mon[33115]: pgmap v746: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 374 KiB/s wr, 479 op/s 2024-04-25T16:44:52.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:52 smithi098 ceph-mon[29162]: pgmap v746: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 374 KiB/s wr, 479 op/s 2024-04-25T16:44:52.661 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/extable.o 2024-04-25T16:44:54.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:54 smithi173 ceph-mon[32556]: pgmap v747: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 232 KiB/s wr, 142 op/s 2024-04-25T16:44:54.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:54 smithi203 ceph-mon[33115]: pgmap v747: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 232 KiB/s wr, 142 op/s 2024-04-25T16:44:54.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:54 smithi098 ceph-mon[29162]: pgmap v747: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 232 KiB/s wr, 142 op/s 2024-04-25T16:44:54.542 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mempool.o 2024-04-25T16:44:55.038 INFO:tasks.workunit.client.0.smithi098.stdout: UPD init/utsversion-tmp.h 2024-04-25T16:44:55.071 INFO:tasks.workunit.client.0.smithi098.stdout: CC init/do_mounts.o 2024-04-25T16:44:56.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:56 smithi173 ceph-mon[32556]: pgmap v748: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 169 KiB/s wr, 142 op/s 2024-04-25T16:44:56.362 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/keyring.o 2024-04-25T16:44:56.383 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/keyctl.o 2024-04-25T16:44:56.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:56 smithi203 ceph-mon[33115]: pgmap v748: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 169 KiB/s wr, 142 op/s 2024-04-25T16:44:56.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:56 smithi098 ceph-mon[29162]: pgmap v748: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 169 KiB/s wr, 142 op/s 2024-04-25T16:44:58.244 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/sched/fair.o 2024-04-25T16:44:58.265 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/msgutil.o 2024-04-25T16:44:58.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:44:58 smithi203 ceph-mon[33115]: pgmap v749: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 159 KiB/s wr, 119 op/s 2024-04-25T16:44:58.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:44:58 smithi098 ceph-mon[29162]: pgmap v749: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 159 KiB/s wr, 119 op/s 2024-04-25T16:44:58.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:44:58 smithi173 ceph-mon[32556]: pgmap v749: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 159 KiB/s wr, 119 op/s 2024-04-25T16:44:58.806 INFO:tasks.workunit.client.0.smithi098.stdout: AR fs/notify/inotify/built-in.a 2024-04-25T16:44:58.812 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/vdso32-setup.o 2024-04-25T16:44:58.824 INFO:tasks.workunit.client.0.smithi098.stderr:ar: fs/notify/inotify/built-in.a: Bad file descriptor 2024-04-25T16:44:58.824 INFO:tasks.workunit.client.0.smithi098.stderr:make[5]: *** [scripts/Makefile.build:403: fs/notify/inotify/built-in.a] Error 123 2024-04-25T16:44:58.826 INFO:tasks.workunit.client.0.smithi098.stderr:make[5]: *** Deleting file 'fs/notify/inotify/built-in.a' 2024-04-25T16:44:58.832 INFO:tasks.workunit.client.0.smithi098.stderr:make[4]: *** [scripts/Makefile.build:480: fs/notify/inotify] Error 2 2024-04-25T16:44:58.834 INFO:tasks.workunit.client.0.smithi098.stderr:make[3]: *** [scripts/Makefile.build:480: fs/notify] Error 2 2024-04-25T16:44:58.836 INFO:tasks.workunit.client.0.smithi098.stderr:make[2]: *** [scripts/Makefile.build:480: fs] Error 2 2024-04-25T16:44:58.836 INFO:tasks.workunit.client.0.smithi098.stderr:make[2]: *** Waiting for unfinished jobs.... 2024-04-25T16:44:58.911 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vsyscall/vsyscall_64.o 2024-04-25T16:45:00.296 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/oom_kill.o 2024-04-25T16:45:00.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:00 smithi173 ceph-mon[32556]: pgmap v750: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 209 KiB/s wr, 120 op/s 2024-04-25T16:45:00.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:00 smithi203 ceph-mon[33115]: pgmap v750: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 209 KiB/s wr, 120 op/s 2024-04-25T16:45:00.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:00 smithi098 ceph-mon[29162]: pgmap v750: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 209 KiB/s wr, 120 op/s 2024-04-25T16:45:01.626 INFO:tasks.workunit.client.0.smithi098.stdout: LDS arch/x86/entry/vdso/vdso.lds 2024-04-25T16:45:01.995 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/vdso/vdso-note.o 2024-04-25T16:45:02.167 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/vclock_gettime.o 2024-04-25T16:45:02.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:02 smithi203 ceph-mon[33115]: pgmap v751: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 185 KiB/s wr, 102 op/s 2024-04-25T16:45:02.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:02 smithi098 ceph-mon[29162]: pgmap v751: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 185 KiB/s wr, 102 op/s 2024-04-25T16:45:02.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:02 smithi173 ceph-mon[32556]: pgmap v751: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 185 KiB/s wr, 102 op/s 2024-04-25T16:45:03.098 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/vgetcpu.o 2024-04-25T16:45:03.236 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/permission.o 2024-04-25T16:45:03.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:03.340+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:03.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:03.340+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:45:03.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:03.340+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:45:03.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:03.340+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:45:03.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:03.340+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:45:03.600 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/msg.o 2024-04-25T16:45:04.300 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/process_keys.o 2024-04-25T16:45:04.301 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:04 smithi098 ceph-mon[29162]: pgmap v752: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 184 KiB/s wr, 80 op/s 2024-04-25T16:45:04.302 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:04 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:45:04.426 INFO:tasks.workunit.client.0.smithi098.stdout: CC init/do_mounts_initrd.o 2024-04-25T16:45:04.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:04 smithi203 ceph-mon[33115]: pgmap v752: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 184 KiB/s wr, 80 op/s 2024-04-25T16:45:04.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:04 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:45:04.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:04 smithi173 ceph-mon[32556]: pgmap v752: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 184 KiB/s wr, 80 op/s 2024-04-25T16:45:04.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:04 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:45:05.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:05 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:45:05.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:05 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:45:05.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:05 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:45:05.464 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/vsyscall/vsyscall_emu_64.o 2024-04-25T16:45:05.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:05 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:45:05.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:05 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:45:05.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:05 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:45:05.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:05 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:45:05.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:05 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:45:05.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:05 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:45:05.847 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/entry/vsyscall/built-in.a 2024-04-25T16:45:06.035 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/entry.o 2024-04-25T16:45:06.384 INFO:tasks.workunit.client.0.smithi098.stdout: CC init/initramfs.o 2024-04-25T16:45:06.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:06 smithi203 ceph-mon[33115]: pgmap v753: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 195 KiB/s wr, 86 op/s 2024-04-25T16:45:06.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:06 smithi098 ceph-mon[29162]: pgmap v753: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 195 KiB/s wr, 86 op/s 2024-04-25T16:45:06.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:06 smithi173 ceph-mon[32556]: pgmap v753: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 195 KiB/s wr, 86 op/s 2024-04-25T16:45:06.557 INFO:tasks.workunit.client.0.smithi098.stdout: HOSTCC arch/x86/entry/vdso/vdso2c 2024-04-25T16:45:07.287 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/sched/build_policy.o 2024-04-25T16:45:07.299 INFO:tasks.workunit.client.0.smithi098.stdout: LDS arch/x86/entry/vdso/vdso32/vdso32.lds 2024-04-25T16:45:07.672 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/fadvise.o 2024-04-25T16:45:07.745 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/vdso/vdso32/note.o 2024-04-25T16:45:07.920 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/vdso/vdso32/system_call.o 2024-04-25T16:45:08.272 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/vdso/vdso32/sigreturn.o 2024-04-25T16:45:08.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:08 smithi203 ceph-mon[33115]: pgmap v754: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 168 KiB/s wr, 57 op/s 2024-04-25T16:45:08.482 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/maccess.o 2024-04-25T16:45:08.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:08 smithi098 ceph-mon[29162]: pgmap v754: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 168 KiB/s wr, 57 op/s 2024-04-25T16:45:08.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:08 smithi173 ceph-mon[32556]: pgmap v754: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 168 KiB/s wr, 57 op/s 2024-04-25T16:45:08.595 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/vdso32/vclock_gettime.o 2024-04-25T16:45:10.376 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/sem.o 2024-04-25T16:45:10.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:10 smithi203 ceph-mon[33115]: pgmap v755: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 205 KiB/s wr, 58 op/s 2024-04-25T16:45:10.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:10 smithi098 ceph-mon[29162]: pgmap v755: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 205 KiB/s wr, 58 op/s 2024-04-25T16:45:10.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:10 smithi173 ceph-mon[32556]: pgmap v755: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 205 KiB/s wr, 58 op/s 2024-04-25T16:45:10.814 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/shm.o 2024-04-25T16:45:11.100 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/request_key.o 2024-04-25T16:45:12.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:12 smithi203 ceph-mon[33115]: pgmap v756: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 183 KiB/s wr, 59 op/s 2024-04-25T16:45:12.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:12 smithi098 ceph-mon[29162]: pgmap v756: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 183 KiB/s wr, 59 op/s 2024-04-25T16:45:12.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:12 smithi173 ceph-mon[32556]: pgmap v756: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 183 KiB/s wr, 59 op/s 2024-04-25T16:45:13.301 INFO:tasks.workunit.client.0.smithi098.stdout: CC init/calibrate.o 2024-04-25T16:45:13.317 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/vdso32/vgetcpu.o 2024-04-25T16:45:14.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:14 smithi203 ceph-mon[33115]: pgmap v757: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 165 KiB/s wr, 52 op/s 2024-04-25T16:45:14.471 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/sched/build_utility.o 2024-04-25T16:45:14.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:14 smithi098 ceph-mon[29162]: pgmap v757: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 165 KiB/s wr, 52 op/s 2024-04-25T16:45:14.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:14 smithi173 ceph-mon[32556]: pgmap v757: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 165 KiB/s wr, 52 op/s 2024-04-25T16:45:14.589 INFO:tasks.workunit.client.0.smithi098.stdout: VDSO arch/x86/entry/vdso/vdso64.so.dbg 2024-04-25T16:45:14.660 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/page-writeback.o 2024-04-25T16:45:15.114 INFO:tasks.workunit.client.0.smithi098.stdout: VDSO arch/x86/entry/vdso/vdso32.so.dbg 2024-04-25T16:45:15.260 INFO:tasks.workunit.client.0.smithi098.stdout: OBJCOPY arch/x86/entry/vdso/vdso64.so 2024-04-25T16:45:15.299 INFO:tasks.workunit.client.0.smithi098.stdout: OBJCOPY arch/x86/entry/vdso/vdso32.so 2024-04-25T16:45:15.343 INFO:tasks.workunit.client.0.smithi098.stdout: VDSO2C arch/x86/entry/vdso/vdso-image-64.c 2024-04-25T16:45:15.377 INFO:tasks.workunit.client.0.smithi098.stdout: VDSO2C arch/x86/entry/vdso/vdso-image-32.c 2024-04-25T16:45:15.415 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/vdso-image-64.o 2024-04-25T16:45:16.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:16 smithi203 ceph-mon[33115]: pgmap v758: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 201 KiB/s wr, 135 op/s 2024-04-25T16:45:16.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:16 smithi098 ceph-mon[29162]: pgmap v758: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 201 KiB/s wr, 135 op/s 2024-04-25T16:45:16.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:16 smithi173 ceph-mon[32556]: pgmap v758: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 201 KiB/s wr, 135 op/s 2024-04-25T16:45:16.631 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/request_key_auth.o 2024-04-25T16:45:16.710 INFO:tasks.workunit.client.0.smithi098.stdout: CC init/init_task.o 2024-04-25T16:45:17.855 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/user_defined.o 2024-04-25T16:45:17.960 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/vdso/vdso-image-32.o 2024-04-25T16:45:18.249 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/syscall.o 2024-04-25T16:45:18.327 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/ipc_sysctl.o 2024-04-25T16:45:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:18.355+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:18.355+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:45:18.353 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:18.355+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:45:18.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:18.355+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:45:18.354 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:18.355+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:45:18.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:18 smithi203 ceph-mon[33115]: pgmap v759: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 172 KiB/s wr, 144 op/s 2024-04-25T16:45:18.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:18 smithi098 ceph-mon[29162]: pgmap v759: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 172 KiB/s wr, 144 op/s 2024-04-25T16:45:18.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:18 smithi173 ceph-mon[32556]: pgmap v759: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 172 KiB/s wr, 144 op/s 2024-04-25T16:45:20.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:20 smithi203 ceph-mon[33115]: pgmap v760: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 198 KiB/s wr, 158 op/s 2024-04-25T16:45:20.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:20 smithi098 ceph-mon[29162]: pgmap v760: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 198 KiB/s wr, 158 op/s 2024-04-25T16:45:20.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:20 smithi173 ceph-mon[32556]: pgmap v760: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 198 KiB/s wr, 158 op/s 2024-04-25T16:45:20.822 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/entry/vdso/built-in.a 2024-04-25T16:45:21.147 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/entry_64.o 2024-04-25T16:45:21.703 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/folio-compat.o 2024-04-25T16:45:21.966 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/syscall_64.o 2024-04-25T16:45:21.966 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/mqueue.o 2024-04-25T16:45:22.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:22 smithi203 ceph-mon[33115]: pgmap v761: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 213 KiB/s wr, 158 op/s 2024-04-25T16:45:22.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:22 smithi098 ceph-mon[29162]: pgmap v761: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 213 KiB/s wr, 158 op/s 2024-04-25T16:45:22.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:22 smithi173 ceph-mon[32556]: pgmap v761: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 213 KiB/s wr, 158 op/s 2024-04-25T16:45:23.157 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/compat.o 2024-04-25T16:45:23.454 INFO:tasks.workunit.client.0.smithi098.stdout: CC init/version.o 2024-04-25T16:45:23.676 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/readahead.o 2024-04-25T16:45:23.823 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/sched/built-in.a 2024-04-25T16:45:24.092 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/mutex.o 2024-04-25T16:45:24.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:24 smithi203 ceph-mon[33115]: pgmap v762: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 179 KiB/s wr, 147 op/s 2024-04-25T16:45:24.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:24 smithi098 ceph-mon[29162]: pgmap v762: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 179 KiB/s wr, 147 op/s 2024-04-25T16:45:24.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:24 smithi173 ceph-mon[32556]: pgmap v762: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 179 KiB/s wr, 147 op/s 2024-04-25T16:45:24.707 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/qos.o 2024-04-25T16:45:26.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:26 smithi203 ceph-mon[33115]: pgmap v763: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 218 KiB/s wr, 158 op/s 2024-04-25T16:45:26.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:26 smithi098 ceph-mon[29162]: pgmap v763: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 218 KiB/s wr, 158 op/s 2024-04-25T16:45:26.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:26 smithi173 ceph-mon[32556]: pgmap v763: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 218 KiB/s wr, 158 op/s 2024-04-25T16:45:26.932 INFO:tasks.workunit.client.0.smithi098.stdout: AR init/built-in.a 2024-04-25T16:45:27.136 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/namespace.o 2024-04-25T16:45:27.751 INFO:tasks.workunit.client.0.smithi098.stdout: CC ipc/mq_sysctl.o 2024-04-25T16:45:28.158 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/common.o 2024-04-25T16:45:28.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:28 smithi203 ceph-mon[33115]: pgmap v764: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 182 KiB/s wr, 74 op/s 2024-04-25T16:45:28.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:28 smithi098 ceph-mon[29162]: pgmap v764: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 182 KiB/s wr, 74 op/s 2024-04-25T16:45:28.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:28 smithi173 ceph-mon[32556]: pgmap v764: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 182 KiB/s wr, 74 op/s 2024-04-25T16:45:29.601 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/proc.o 2024-04-25T16:45:30.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:30 smithi173 ceph-mon[32556]: pgmap v765: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 216 KiB/s wr, 57 op/s 2024-04-25T16:45:30.413 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/swap.o 2024-04-25T16:45:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:30 smithi203 ceph-mon[33115]: pgmap v765: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 216 KiB/s wr, 57 op/s 2024-04-25T16:45:30.446 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/truncate.o 2024-04-25T16:45:30.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:30 smithi098 ceph-mon[29162]: pgmap v765: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 216 KiB/s wr, 57 op/s 2024-04-25T16:45:30.777 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/semaphore.o 2024-04-25T16:45:31.147 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/main.o 2024-04-25T16:45:31.721 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/console.o 2024-04-25T16:45:32.276 INFO:tasks.workunit.client.0.smithi098.stdout: AR ipc/built-in.a 2024-04-25T16:45:32.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:32 smithi203 ceph-mon[33115]: pgmap v766: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 210 KiB/s wr, 47 op/s 2024-04-25T16:45:32.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:32 smithi098 ceph-mon[29162]: pgmap v766: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 210 KiB/s wr, 47 op/s 2024-04-25T16:45:32.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:32 smithi173 ceph-mon[32556]: pgmap v766: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 210 KiB/s wr, 47 op/s 2024-04-25T16:45:32.528 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/vmscan.o 2024-04-25T16:45:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:33.370+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:33.370+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:45:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:33.370+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:45:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:33.370+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:45:33.368 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:33.370+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:45:34.433 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/thunk_64.o 2024-04-25T16:45:34.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:34 smithi203 ceph-mon[33115]: pgmap v767: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 161 KiB/s wr, 40 op/s 2024-04-25T16:45:34.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:34 smithi098 ceph-mon[29162]: pgmap v767: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 161 KiB/s wr, 40 op/s 2024-04-25T16:45:34.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:34 smithi173 ceph-mon[32556]: pgmap v767: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 161 KiB/s wr, 40 op/s 2024-04-25T16:45:34.936 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/sysctl.o 2024-04-25T16:45:34.945 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/entry/entry_64_compat.o 2024-04-25T16:45:35.595 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/entry/syscall_32.o 2024-04-25T16:45:36.264 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/rwsem.o 2024-04-25T16:45:36.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:36 smithi203 ceph-mon[33115]: pgmap v768: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 192 KiB/s wr, 45 op/s 2024-04-25T16:45:36.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:36 smithi098 ceph-mon[29162]: pgmap v768: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 192 KiB/s wr, 45 op/s 2024-04-25T16:45:36.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:36 smithi173 ceph-mon[32556]: pgmap v768: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 192 KiB/s wr, 45 op/s 2024-04-25T16:45:36.652 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/shmem.o 2024-04-25T16:45:37.608 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/percpu-rwsem.o 2024-04-25T16:45:38.111 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/spinlock.o 2024-04-25T16:45:38.216 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/process.o 2024-04-25T16:45:38.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:38 smithi203 ceph-mon[33115]: pgmap v769: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 146 KiB/s wr, 34 op/s 2024-04-25T16:45:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:38 smithi098 ceph-mon[29162]: pgmap v769: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 146 KiB/s wr, 34 op/s 2024-04-25T16:45:38.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:38 smithi173 ceph-mon[32556]: pgmap v769: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 146 KiB/s wr, 34 op/s 2024-04-25T16:45:40.040 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/keys/keyctl_pkey.o 2024-04-25T16:45:40.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:40 smithi203 ceph-mon[33115]: pgmap v770: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 167 KiB/s wr, 39 op/s 2024-04-25T16:45:40.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:40 smithi098 ceph-mon[29162]: pgmap v770: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 167 KiB/s wr, 39 op/s 2024-04-25T16:45:40.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:40 smithi173 ceph-mon[32556]: pgmap v770: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 167 KiB/s wr, 39 op/s 2024-04-25T16:45:41.148 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/util.o 2024-04-25T16:45:41.271 INFO:tasks.workunit.client.0.smithi098.stdout: GEN security/selinux/flask.h security/selinux/av_permissions.h 2024-04-25T16:45:41.336 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/avc.o 2024-04-25T16:45:41.599 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/hooks.o 2024-04-25T16:45:41.741 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/entry/built-in.a 2024-04-25T16:45:42.218 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/osq_lock.o 2024-04-25T16:45:42.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:42 smithi203 ceph-mon[33115]: pgmap v771: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 170 KiB/s wr, 41 op/s 2024-04-25T16:45:42.500 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/amd/core.o 2024-04-25T16:45:42.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:42 smithi098 ceph-mon[29162]: pgmap v771: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 170 KiB/s wr, 41 op/s 2024-04-25T16:45:42.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:42 smithi173 ceph-mon[32556]: pgmap v771: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 170 KiB/s wr, 41 op/s 2024-04-25T16:45:44.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:44 smithi203 ceph-mon[33115]: pgmap v772: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 171 KiB/s wr, 36 op/s 2024-04-25T16:45:44.483 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/qspinlock.o 2024-04-25T16:45:44.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:44 smithi098 ceph-mon[29162]: pgmap v772: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 171 KiB/s wr, 36 op/s 2024-04-25T16:45:44.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:44 smithi173 ceph-mon[32556]: pgmap v772: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 171 KiB/s wr, 36 op/s 2024-04-25T16:45:44.924 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/suspend.o 2024-04-25T16:45:45.348 INFO:tasks.workunit.client.0.smithi098.stdout: AR security/keys/built-in.a 2024-04-25T16:45:45.549 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mmzone.o 2024-04-25T16:45:45.752 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/vmstat.o 2024-04-25T16:45:46.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:46 smithi203 ceph-mon[33115]: pgmap v773: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 202 KiB/s wr, 42 op/s 2024-04-25T16:45:46.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:46 smithi098 ceph-mon[29162]: pgmap v773: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 202 KiB/s wr, 42 op/s 2024-04-25T16:45:46.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:46 smithi173 ceph-mon[32556]: pgmap v773: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 202 KiB/s wr, 42 op/s 2024-04-25T16:45:47.641 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/printk/printk.o 2024-04-25T16:45:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:48.385+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:45:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:48.385+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:45:48.383 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:48.385+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:45:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:48.385+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:45:48.384 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:45:48.385+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:45:48.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:48 smithi203 ceph-mon[33115]: pgmap v774: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 165 KiB/s wr, 36 op/s 2024-04-25T16:45:48.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:48 smithi098 ceph-mon[29162]: pgmap v774: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 165 KiB/s wr, 36 op/s 2024-04-25T16:45:48.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:48 smithi173 ceph-mon[32556]: pgmap v774: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 165 KiB/s wr, 36 op/s 2024-04-25T16:45:48.901 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/amd/lbr.o 2024-04-25T16:45:48.933 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/rtmutex_api.o 2024-04-25T16:45:50.368 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/locking/qrwlock.o 2024-04-25T16:45:50.408 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/printk/printk_safe.o 2024-04-25T16:45:50.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:50 smithi203 ceph-mon[33115]: pgmap v775: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 190 KiB/s wr, 41 op/s 2024-04-25T16:45:50.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:50 smithi098 ceph-mon[29162]: pgmap v775: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 190 KiB/s wr, 41 op/s 2024-04-25T16:45:50.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:50 smithi173 ceph-mon[32556]: pgmap v775: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 190 KiB/s wr, 41 op/s 2024-04-25T16:45:51.684 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/hibernate.o 2024-04-25T16:45:52.395 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/backing-dev.o 2024-04-25T16:45:52.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:52 smithi203 ceph-mon[33115]: pgmap v776: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 196 KiB/s wr, 49 op/s 2024-04-25T16:45:52.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:52 smithi098 ceph-mon[29162]: pgmap v776: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 196 KiB/s wr, 49 op/s 2024-04-25T16:45:52.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:52 smithi173 ceph-mon[32556]: pgmap v776: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 196 KiB/s wr, 49 op/s 2024-04-25T16:45:53.678 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/selinuxfs.o 2024-04-25T16:45:53.873 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/netlink.o 2024-04-25T16:45:54.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:54 smithi203 ceph-mon[33115]: pgmap v777: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 170 KiB/s wr, 44 op/s 2024-04-25T16:45:54.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:54 smithi098 ceph-mon[29162]: pgmap v777: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 170 KiB/s wr, 44 op/s 2024-04-25T16:45:54.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:54 smithi173 ceph-mon[32556]: pgmap v777: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 170 KiB/s wr, 44 op/s 2024-04-25T16:45:54.721 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/locking/built-in.a 2024-04-25T16:45:54.992 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/amd/ibs.o 2024-04-25T16:45:55.061 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/nlmsgtab.o 2024-04-25T16:45:55.432 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/irqdesc.o 2024-04-25T16:45:55.687 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/printk/printk_ringbuffer.o 2024-04-25T16:45:56.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:56 smithi203 ceph-mon[33115]: pgmap v778: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 275 KiB/s wr, 54 op/s 2024-04-25T16:45:56.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:56 smithi098 ceph-mon[29162]: pgmap v778: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 275 KiB/s wr, 54 op/s 2024-04-25T16:45:56.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:56 smithi173 ceph-mon[32556]: pgmap v778: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 275 KiB/s wr, 54 op/s 2024-04-25T16:45:58.356 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mm_init.o 2024-04-25T16:45:58.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:45:58 smithi203 ceph-mon[33115]: pgmap v779: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 235 KiB/s wr, 47 op/s 2024-04-25T16:45:58.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:45:58 smithi098 ceph-mon[29162]: pgmap v779: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 235 KiB/s wr, 47 op/s 2024-04-25T16:45:58.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:45:58 smithi173 ceph-mon[32556]: pgmap v779: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 235 KiB/s wr, 47 op/s 2024-04-25T16:45:58.806 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/snapshot.o 2024-04-25T16:45:59.209 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/printk/sysctl.o 2024-04-25T16:46:00.418 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/handle.o 2024-04-25T16:46:00.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:00 smithi203 ceph-mon[33115]: pgmap v780: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 247 KiB/s wr, 50 op/s 2024-04-25T16:46:00.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:00 smithi098 ceph-mon[29162]: pgmap v780: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 247 KiB/s wr, 50 op/s 2024-04-25T16:46:00.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:00 smithi173 ceph-mon[32556]: pgmap v780: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 247 KiB/s wr, 50 op/s 2024-04-25T16:46:00.583 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/manage.o 2024-04-25T16:46:01.834 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/amd/uncore.o 2024-04-25T16:46:02.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:02 smithi203 ceph-mon[33115]: pgmap v781: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 255 KiB/s wr, 52 op/s 2024-04-25T16:46:02.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:02 smithi098 ceph-mon[29162]: pgmap v781: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 255 KiB/s wr, 52 op/s 2024-04-25T16:46:02.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:02 smithi173 ceph-mon[32556]: pgmap v781: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 255 KiB/s wr, 52 op/s 2024-04-25T16:46:02.726 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/percpu.o 2024-04-25T16:46:02.760 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/netif.o 2024-04-25T16:46:02.856 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/printk/built-in.a 2024-04-25T16:46:03.030 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/netnode.o 2024-04-25T16:46:03.398 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:03.400+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:03.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:03.400+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:46:03.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:03.400+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:46:03.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:03.400+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:46:03.399 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:03.400+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:46:04.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:04 smithi203 ceph-mon[33115]: pgmap v782: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 222 KiB/s wr, 42 op/s 2024-04-25T16:46:04.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:04 smithi098 ceph-mon[29162]: pgmap v782: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 222 KiB/s wr, 42 op/s 2024-04-25T16:46:04.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:04 smithi173 ceph-mon[32556]: pgmap v782: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 222 KiB/s wr, 42 op/s 2024-04-25T16:46:05.290 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/rcu/update.o 2024-04-25T16:46:05.397 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/rcu/sync.o 2024-04-25T16:46:05.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:05 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:46:05.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:05 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:46:05.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:05 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:46:06.369 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/spurious.o 2024-04-25T16:46:06.408 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/swap.o 2024-04-25T16:46:06.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:06 smithi203 ceph-mon[33115]: pgmap v783: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 272 KiB/s wr, 51 op/s 2024-04-25T16:46:06.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:46:06.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:46:06.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:46:06.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:06 smithi098 ceph-mon[29162]: pgmap v783: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 272 KiB/s wr, 51 op/s 2024-04-25T16:46:06.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:46:06.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:46:06.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:46:06.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:06 smithi173 ceph-mon[32556]: pgmap v783: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 272 KiB/s wr, 51 op/s 2024-04-25T16:46:06.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:46:06.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:46:06.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:46:08.223 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/amd/iommu.o 2024-04-25T16:46:08.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:08 smithi203 ceph-mon[33115]: pgmap v784: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 151 KiB/s wr, 40 op/s 2024-04-25T16:46:08.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:08 smithi098 ceph-mon[29162]: pgmap v784: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 151 KiB/s wr, 40 op/s 2024-04-25T16:46:08.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:08 smithi173 ceph-mon[32556]: pgmap v784: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 151 KiB/s wr, 40 op/s 2024-04-25T16:46:08.817 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/slab_common.o 2024-04-25T16:46:09.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:09 smithi203 ceph-mon[33115]: pgmap v785: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 160 KiB/s wr, 43 op/s 2024-04-25T16:46:09.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:09 smithi098 ceph-mon[29162]: pgmap v785: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 160 KiB/s wr, 43 op/s 2024-04-25T16:46:09.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:09 smithi173 ceph-mon[32556]: pgmap v785: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 160 KiB/s wr, 43 op/s 2024-04-25T16:46:10.694 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/user.o 2024-04-25T16:46:11.093 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/resend.o 2024-04-25T16:46:11.303 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/chip.o 2024-04-25T16:46:11.540 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/netport.o 2024-04-25T16:46:11.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:11 smithi203 ceph-mon[33115]: pgmap v786: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 177 KiB/s wr, 45 op/s 2024-04-25T16:46:11.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:11 smithi098 ceph-mon[29162]: pgmap v786: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 177 KiB/s wr, 45 op/s 2024-04-25T16:46:11.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:11 smithi173 ceph-mon[32556]: pgmap v786: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 177 KiB/s wr, 45 op/s 2024-04-25T16:46:12.814 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/rcu/srcutree.o 2024-04-25T16:46:12.846 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/status.o 2024-04-25T16:46:13.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:13 smithi203 ceph-mon[33115]: pgmap v787: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 144 KiB/s wr, 38 op/s 2024-04-25T16:46:13.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:13 smithi098 ceph-mon[29162]: pgmap v787: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 144 KiB/s wr, 38 op/s 2024-04-25T16:46:13.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:13 smithi173 ceph-mon[32556]: pgmap v787: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 144 KiB/s wr, 38 op/s 2024-04-25T16:46:14.359 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/events/amd/built-in.a 2024-04-25T16:46:14.832 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/core.o 2024-04-25T16:46:15.643 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/ebitmap.o 2024-04-25T16:46:15.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:15 smithi203 ceph-mon[33115]: pgmap v788: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:46:16.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:15 smithi098 ceph-mon[29162]: pgmap v788: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:46:16.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:15 smithi173 ceph-mon[32556]: pgmap v788: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:46:16.150 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/compaction.o 2024-04-25T16:46:16.572 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/dummychip.o 2024-04-25T16:46:16.983 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/power/poweroff.o 2024-04-25T16:46:17.765 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/rcu/tree.o 2024-04-25T16:46:17.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:17 smithi203 ceph-mon[33115]: pgmap v789: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 152 KiB/s wr, 34 op/s 2024-04-25T16:46:18.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:17 smithi098 ceph-mon[29162]: pgmap v789: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 152 KiB/s wr, 34 op/s 2024-04-25T16:46:18.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:17 smithi173 ceph-mon[32556]: pgmap v789: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 152 KiB/s wr, 34 op/s 2024-04-25T16:46:18.413 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:18.416+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:18.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:18.416+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:46:18.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:18.416+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:46:18.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:18.416+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:46:18.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:18.416+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:46:19.821 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/power/built-in.a 2024-04-25T16:46:19.835 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/show_mem.o 2024-04-25T16:46:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:19 smithi203 ceph-mon[33115]: pgmap v790: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 153 KiB/s wr, 38 op/s 2024-04-25T16:46:20.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:19 smithi098 ceph-mon[29162]: pgmap v790: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 153 KiB/s wr, 38 op/s 2024-04-25T16:46:20.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:19 smithi173 ceph-mon[32556]: pgmap v790: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 153 KiB/s wr, 38 op/s 2024-04-25T16:46:20.185 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/interval_tree.o 2024-04-25T16:46:21.004 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/list_lru.o 2024-04-25T16:46:21.178 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/devres.o 2024-04-25T16:46:21.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:21 smithi203 ceph-mon[33115]: pgmap v791: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 182 KiB/s wr, 43 op/s 2024-04-25T16:46:22.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:21 smithi098 ceph-mon[29162]: pgmap v791: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 182 KiB/s wr, 43 op/s 2024-04-25T16:46:22.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:21 smithi173 ceph-mon[32556]: pgmap v791: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 182 KiB/s wr, 43 op/s 2024-04-25T16:46:23.022 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/bts.o 2024-04-25T16:46:23.359 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/hashtab.o 2024-04-25T16:46:23.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:23 smithi203 ceph-mon[33115]: pgmap v792: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 150 KiB/s wr, 38 op/s 2024-04-25T16:46:23.954 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/symtab.o 2024-04-25T16:46:24.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:23 smithi098 ceph-mon[29162]: pgmap v792: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 150 KiB/s wr, 38 op/s 2024-04-25T16:46:24.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:23 smithi173 ceph-mon[32556]: pgmap v792: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 150 KiB/s wr, 38 op/s 2024-04-25T16:46:25.618 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/autoprobe.o 2024-04-25T16:46:25.916 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/sidtab.o 2024-04-25T16:46:25.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:25 smithi203 ceph-mon[33115]: pgmap v793: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 213 KiB/s wr, 48 op/s 2024-04-25T16:46:26.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:25 smithi098 ceph-mon[29162]: pgmap v793: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 213 KiB/s wr, 48 op/s 2024-04-25T16:46:26.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:25 smithi173 ceph-mon[32556]: pgmap v793: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 213 KiB/s wr, 48 op/s 2024-04-25T16:46:26.043 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/avtab.o 2024-04-25T16:46:26.585 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/rcu/rcu_segcblist.o 2024-04-25T16:46:26.769 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/workingset.o 2024-04-25T16:46:27.326 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/debug.o 2024-04-25T16:46:27.359 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/ds.o 2024-04-25T16:46:27.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:27 smithi203 ceph-mon[33115]: pgmap v794: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 165 KiB/s wr, 40 op/s 2024-04-25T16:46:28.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:27 smithi098 ceph-mon[29162]: pgmap v794: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 165 KiB/s wr, 40 op/s 2024-04-25T16:46:28.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:27 smithi173 ceph-mon[32556]: pgmap v794: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 165 KiB/s wr, 40 op/s 2024-04-25T16:46:29.172 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/livepatch/built-in.a 2024-04-25T16:46:29.242 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/knc.o 2024-04-25T16:46:29.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:29 smithi203 ceph-mon[33115]: pgmap v795: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 179 KiB/s wr, 42 op/s 2024-04-25T16:46:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:29 smithi098 ceph-mon[29162]: pgmap v795: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 179 KiB/s wr, 42 op/s 2024-04-25T16:46:30.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:29 smithi173 ceph-mon[32556]: pgmap v795: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 179 KiB/s wr, 42 op/s 2024-04-25T16:46:30.143 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/irqdomain.o 2024-04-25T16:46:31.075 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/rcu/built-in.a 2024-04-25T16:46:31.329 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/realmode/init.o 2024-04-25T16:46:31.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:31 smithi173 ceph-mon[32556]: pgmap v796: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 220 KiB/s wr, 46 op/s 2024-04-25T16:46:31.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:31 smithi203 ceph-mon[33115]: pgmap v796: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 220 KiB/s wr, 46 op/s 2024-04-25T16:46:32.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:31 smithi098 ceph-mon[29162]: pgmap v796: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 220 KiB/s wr, 46 op/s 2024-04-25T16:46:32.905 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/realmode/rm/header.o 2024-04-25T16:46:33.316 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/realmode/rm/trampoline_64.o 2024-04-25T16:46:33.429 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:33.431+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:33.432+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:46:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:33.432+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:46:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:33.432+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:46:33.430 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:33.432+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:46:33.683 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/realmode/rm/stack.o 2024-04-25T16:46:33.826 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/policydb.o 2024-04-25T16:46:33.865 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/gup.o 2024-04-25T16:46:33.882 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/zhaoxin/core.o 2024-04-25T16:46:33.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:33 smithi203 ceph-mon[33115]: pgmap v797: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 171 KiB/s wr, 37 op/s 2024-04-25T16:46:33.994 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/realmode/rm/reboot.o 2024-04-25T16:46:34.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:33 smithi098 ceph-mon[29162]: pgmap v797: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 171 KiB/s wr, 37 op/s 2024-04-25T16:46:34.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:33 smithi173 ceph-mon[32556]: pgmap v797: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 171 KiB/s wr, 37 op/s 2024-04-25T16:46:34.247 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mmap_lock.o 2024-04-25T16:46:34.418 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/realmode/rm/wakeup_asm.o 2024-04-25T16:46:34.836 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/realmode/rm/wakemain.o 2024-04-25T16:46:35.340 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/lbr.o 2024-04-25T16:46:35.677 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/realmode/rm/video-mode.o 2024-04-25T16:46:35.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:35 smithi203 ceph-mon[33115]: pgmap v798: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 220 KiB/s wr, 47 op/s 2024-04-25T16:46:36.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:35 smithi098 ceph-mon[29162]: pgmap v798: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 220 KiB/s wr, 47 op/s 2024-04-25T16:46:36.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:35 smithi173 ceph-mon[32556]: pgmap v798: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 220 KiB/s wr, 47 op/s 2024-04-25T16:46:36.308 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/proc.o 2024-04-25T16:46:36.503 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/migration.o 2024-04-25T16:46:36.535 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/realmode/rm/copy.o 2024-04-25T16:46:36.838 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/realmode/rm/bioscall.o 2024-04-25T16:46:36.976 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/realmode/rm/regs.o 2024-04-25T16:46:37.753 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/realmode/rm/video-vga.o 2024-04-25T16:46:37.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:37 smithi203 ceph-mon[33115]: pgmap v799: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 160 KiB/s wr, 38 op/s 2024-04-25T16:46:38.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:37 smithi098 ceph-mon[29162]: pgmap v799: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 160 KiB/s wr, 38 op/s 2024-04-25T16:46:38.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:37 smithi173 ceph-mon[32556]: pgmap v799: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 160 KiB/s wr, 38 op/s 2024-04-25T16:46:38.546 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/realmode/rm/video-vesa.o 2024-04-25T16:46:39.327 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/realmode/rm/video-bios.o 2024-04-25T16:46:39.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:39 smithi203 ceph-mon[33115]: pgmap v800: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 167 KiB/s wr, 43 op/s 2024-04-25T16:46:40.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:39 smithi098 ceph-mon[29162]: pgmap v800: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 167 KiB/s wr, 43 op/s 2024-04-25T16:46:40.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:39 smithi173 ceph-mon[32556]: pgmap v800: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 167 KiB/s wr, 43 op/s 2024-04-25T16:46:40.061 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/events/zhaoxin/built-in.a 2024-04-25T16:46:40.127 INFO:tasks.workunit.client.0.smithi098.stdout: PASYMS arch/x86/realmode/rm/pasyms.h 2024-04-25T16:46:40.204 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/highmem.o 2024-04-25T16:46:40.327 INFO:tasks.workunit.client.0.smithi098.stdout: LDS arch/x86/realmode/rm/realmode.lds 2024-04-25T16:46:40.594 INFO:tasks.workunit.client.0.smithi098.stdout: LD arch/x86/realmode/rm/realmode.elf 2024-04-25T16:46:40.671 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/p4.o 2024-04-25T16:46:40.711 INFO:tasks.workunit.client.0.smithi098.stdout: RELOCS arch/x86/realmode/rm/realmode.relocs 2024-04-25T16:46:40.748 INFO:tasks.workunit.client.0.smithi098.stdout: OBJCOPY arch/x86/realmode/rm/realmode.bin 2024-04-25T16:46:40.827 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/realmode/rmpiggy.o 2024-04-25T16:46:41.038 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/core.o 2024-04-25T16:46:41.086 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/p6.o 2024-04-25T16:46:41.182 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/realmode/built-in.a 2024-04-25T16:46:41.354 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/cpuhotplug.o 2024-04-25T16:46:41.750 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/pm.o 2024-04-25T16:46:41.812 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/services.o 2024-04-25T16:46:41.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:41 smithi203 ceph-mon[33115]: pgmap v801: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 181 KiB/s wr, 52 op/s 2024-04-25T16:46:42.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:41 smithi098 ceph-mon[29162]: pgmap v801: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 181 KiB/s wr, 52 op/s 2024-04-25T16:46:42.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:41 smithi173 ceph-mon[32556]: pgmap v801: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 181 KiB/s wr, 52 op/s 2024-04-25T16:46:42.552 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/dma/mapping.o 2024-04-25T16:46:43.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:43 smithi203 ceph-mon[33115]: pgmap v802: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 160 KiB/s wr, 46 op/s 2024-04-25T16:46:44.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:43 smithi098 ceph-mon[29162]: pgmap v802: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 160 KiB/s wr, 46 op/s 2024-04-25T16:46:44.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:43 smithi173 ceph-mon[32556]: pgmap v802: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 160 KiB/s wr, 46 op/s 2024-04-25T16:46:45.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:45 smithi203 ceph-mon[33115]: pgmap v803: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 223 KiB/s wr, 57 op/s 2024-04-25T16:46:46.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:45 smithi098 ceph-mon[29162]: pgmap v803: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 223 KiB/s wr, 57 op/s 2024-04-25T16:46:46.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:45 smithi173 ceph-mon[32556]: pgmap v803: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 223 KiB/s wr, 57 op/s 2024-04-25T16:46:46.111 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/dma/direct.o 2024-04-25T16:46:46.234 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/memory.o 2024-04-25T16:46:46.966 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/dma/ops_helpers.o 2024-04-25T16:46:47.068 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/pt.o 2024-04-25T16:46:47.744 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/msi.o 2024-04-25T16:46:47.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:47 smithi203 ceph-mon[33115]: pgmap v804: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 189 KiB/s wr, 50 op/s 2024-04-25T16:46:48.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:47 smithi098 ceph-mon[29162]: pgmap v804: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 189 KiB/s wr, 50 op/s 2024-04-25T16:46:48.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:47 smithi173 ceph-mon[32556]: pgmap v804: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 189 KiB/s wr, 50 op/s 2024-04-25T16:46:48.372 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/dma/dummy.o 2024-04-25T16:46:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:48.447+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:46:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:48.447+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:46:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:48.447+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:46:48.445 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:48.447+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:46:48.446 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:46:48.447+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:46:48.603 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/probe.o 2024-04-25T16:46:49.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:49 smithi203 ceph-mon[33115]: pgmap v805: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 193 KiB/s wr, 51 op/s 2024-04-25T16:46:50.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:49 smithi098 ceph-mon[29162]: pgmap v805: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 193 KiB/s wr, 51 op/s 2024-04-25T16:46:50.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:49 smithi173 ceph-mon[32556]: pgmap v805: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 193 KiB/s wr, 51 op/s 2024-04-25T16:46:51.536 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/conditional.o 2024-04-25T16:46:51.752 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/uncore.o 2024-04-25T16:46:51.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:51 smithi203 ceph-mon[33115]: pgmap v806: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 216 KiB/s wr, 54 op/s 2024-04-25T16:46:52.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:51 smithi098 ceph-mon[29162]: pgmap v806: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 216 KiB/s wr, 54 op/s 2024-04-25T16:46:52.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:51 smithi173 ceph-mon[32556]: pgmap v806: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 216 KiB/s wr, 54 op/s 2024-04-25T16:46:52.300 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/dma/swiotlb.o 2024-04-25T16:46:52.386 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/dma/remap.o 2024-04-25T16:46:53.561 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/utils.o 2024-04-25T16:46:53.726 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/rapl.o 2024-04-25T16:46:53.835 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/affinity.o 2024-04-25T16:46:53.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:53 smithi203 ceph-mon[33115]: pgmap v807: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 186 KiB/s wr, 41 op/s 2024-04-25T16:46:54.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:53 smithi098 ceph-mon[29162]: pgmap v807: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 186 KiB/s wr, 41 op/s 2024-04-25T16:46:54.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:53 smithi173 ceph-mon[32556]: pgmap v807: 129 pgs: 129 active+clean; 1.7 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 186 KiB/s wr, 41 op/s 2024-04-25T16:46:54.812 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mincore.o 2024-04-25T16:46:55.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:55 smithi203 ceph-mon[33115]: pgmap v808: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 KiB/s wr, 47 op/s 2024-04-25T16:46:56.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:55 smithi098 ceph-mon[29162]: pgmap v808: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 KiB/s wr, 47 op/s 2024-04-25T16:46:56.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:55 smithi173 ceph-mon[32556]: pgmap v808: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 199 KiB/s wr, 47 op/s 2024-04-25T16:46:57.759 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq/matrix.o 2024-04-25T16:46:57.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:57 smithi203 ceph-mon[33115]: pgmap v809: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 156 KiB/s wr, 36 op/s 2024-04-25T16:46:58.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:57 smithi098 ceph-mon[29162]: pgmap v809: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 156 KiB/s wr, 36 op/s 2024-04-25T16:46:58.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:57 smithi173 ceph-mon[32556]: pgmap v809: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 156 KiB/s wr, 36 op/s 2024-04-25T16:46:58.679 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/uncore_nhmex.o 2024-04-25T16:46:58.861 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/fpu/init.o 2024-04-25T16:46:59.377 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/dma/built-in.a 2024-04-25T16:46:59.407 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/mls.o 2024-04-25T16:46:59.896 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/msr.o 2024-04-25T16:46:59.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:46:59 smithi203 ceph-mon[33115]: pgmap v810: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 157 KiB/s wr, 37 op/s 2024-04-25T16:47:00.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:46:59 smithi098 ceph-mon[29162]: pgmap v810: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 157 KiB/s wr, 37 op/s 2024-04-25T16:47:00.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:46:59 smithi173 ceph-mon[32556]: pgmap v810: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 157 KiB/s wr, 37 op/s 2024-04-25T16:47:00.132 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mce/core.o 2024-04-25T16:47:00.367 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/pat/set_memory.o 2024-04-25T16:47:01.527 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mlock.o 2024-04-25T16:47:01.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:01 smithi203 ceph-mon[33115]: pgmap v811: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 190 KiB/s wr, 41 op/s 2024-04-25T16:47:02.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:01 smithi098 ceph-mon[29162]: pgmap v811: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 190 KiB/s wr, 41 op/s 2024-04-25T16:47:02.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:01 smithi173 ceph-mon[32556]: pgmap v811: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 190 KiB/s wr, 41 op/s 2024-04-25T16:47:02.901 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/fpu/bugs.o 2024-04-25T16:47:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:03.462+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:47:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:03.462+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:47:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:03.462+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:47:03.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:03.462+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:47:03.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:03.462+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:47:03.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:03 smithi203 ceph-mon[33115]: pgmap v812: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 153 KiB/s wr, 33 op/s 2024-04-25T16:47:04.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:03 smithi098 ceph-mon[29162]: pgmap v812: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 153 KiB/s wr, 33 op/s 2024-04-25T16:47:04.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:03 smithi173 ceph-mon[32556]: pgmap v812: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 153 KiB/s wr, 33 op/s 2024-04-25T16:47:04.646 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/irq/built-in.a 2024-04-25T16:47:05.086 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/fpu/core.o 2024-04-25T16:47:05.271 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/entry/common.o 2024-04-25T16:47:05.413 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/uncore_snb.o 2024-04-25T16:47:05.867 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/uncore_snbep.o 2024-04-25T16:47:05.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:05 smithi203 ceph-mon[33115]: pgmap v813: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 186 KiB/s wr, 42 op/s 2024-04-25T16:47:06.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:05 smithi098 ceph-mon[29162]: pgmap v813: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 186 KiB/s wr, 42 op/s 2024-04-25T16:47:06.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:05 smithi173 ceph-mon[32556]: pgmap v813: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 186 KiB/s wr, 42 op/s 2024-04-25T16:47:06.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:06 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:47:07.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:06 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:47:07.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:06 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:47:07.237 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/pat/memtype.o 2024-04-25T16:47:07.275 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/ss/context.o 2024-04-25T16:47:07.646 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mce/severity.o 2024-04-25T16:47:07.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:07 smithi203 ceph-mon[33115]: pgmap v814: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 140 KiB/s wr, 34 op/s 2024-04-25T16:47:07.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:47:07.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:47:07.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:47:08.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:07 smithi098 ceph-mon[29162]: pgmap v814: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 140 KiB/s wr, 34 op/s 2024-04-25T16:47:08.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:47:08.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:47:08.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:47:08.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:07 smithi173 ceph-mon[32556]: pgmap v814: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 140 KiB/s wr, 34 op/s 2024-04-25T16:47:08.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:47:08.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:47:08.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:47:08.399 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mmap.o 2024-04-25T16:47:09.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:09 smithi203 ceph-mon[33115]: pgmap v815: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 153 KiB/s wr, 39 op/s 2024-04-25T16:47:10.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:09 smithi098 ceph-mon[29162]: pgmap v815: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 153 KiB/s wr, 39 op/s 2024-04-25T16:47:10.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:09 smithi173 ceph-mon[32556]: pgmap v815: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 153 KiB/s wr, 39 op/s 2024-04-25T16:47:11.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:11 smithi203 ceph-mon[33115]: pgmap v816: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 188 KiB/s wr, 45 op/s 2024-04-25T16:47:12.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:11 smithi098 ceph-mon[29162]: pgmap v816: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 188 KiB/s wr, 45 op/s 2024-04-25T16:47:12.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:11 smithi173 ceph-mon[32556]: pgmap v816: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 188 KiB/s wr, 45 op/s 2024-04-25T16:47:12.049 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mmu_gather.o 2024-04-25T16:47:12.345 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/entry/syscall_user_dispatch.o 2024-04-25T16:47:12.351 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/fpu/regset.o 2024-04-25T16:47:12.827 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/pat/memtype_interval.o 2024-04-25T16:47:12.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:12 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:47:13.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:12 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:47:13.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:12 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:47:13.219 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/uncore_discovery.o 2024-04-25T16:47:13.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:13 smithi203 ceph-mon[33115]: pgmap v817: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 154 KiB/s wr, 40 op/s 2024-04-25T16:47:13.992 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mce/genpool.o 2024-04-25T16:47:14.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:13 smithi098 ceph-mon[29162]: pgmap v817: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 154 KiB/s wr, 40 op/s 2024-04-25T16:47:14.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:13 smithi173 ceph-mon[32556]: pgmap v817: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 154 KiB/s wr, 40 op/s 2024-04-25T16:47:14.716 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/selinux/netlabel.o 2024-04-25T16:47:15.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:15 smithi203 ceph-mon[33115]: pgmap v818: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 198 KiB/s wr, 48 op/s 2024-04-25T16:47:16.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:15 smithi098 ceph-mon[29162]: pgmap v818: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 198 KiB/s wr, 48 op/s 2024-04-25T16:47:16.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:15 smithi173 ceph-mon[32556]: pgmap v818: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 198 KiB/s wr, 48 op/s 2024-04-25T16:47:16.269 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/fpu/signal.o 2024-04-25T16:47:17.433 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/entry/built-in.a 2024-04-25T16:47:17.634 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/module/main.o 2024-04-25T16:47:17.780 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/mm/pat/built-in.a 2024-04-25T16:47:17.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:17 smithi203 ceph-mon[33115]: pgmap v819: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 174 KiB/s wr, 39 op/s 2024-04-25T16:47:17.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:17 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:47:18.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:17 smithi098 ceph-mon[29162]: pgmap v819: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 174 KiB/s wr, 39 op/s 2024-04-25T16:47:18.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:17 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:47:18.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:17 smithi173 ceph-mon[32556]: pgmap v819: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 174 KiB/s wr, 39 op/s 2024-04-25T16:47:18.013 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:17 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:47:18.016 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/init.o 2024-04-25T16:47:18.039 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mprotect.o 2024-04-25T16:47:18.078 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mremap.o 2024-04-25T16:47:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:18.477+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:47:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:18.477+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:47:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:18.477+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:47:18.475 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:18.477+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:47:18.476 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:18.477+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:47:19.241 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mce/intel.o 2024-04-25T16:47:19.686 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/events/intel/cstate.o 2024-04-25T16:47:19.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:19 smithi203 ceph-mon[33115]: pgmap v820: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 187 KiB/s wr, 42 op/s 2024-04-25T16:47:20.002 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:19 smithi098 ceph-mon[29162]: pgmap v820: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 187 KiB/s wr, 42 op/s 2024-04-25T16:47:20.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:19 smithi173 ceph-mon[32556]: pgmap v820: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 187 KiB/s wr, 42 op/s 2024-04-25T16:47:21.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:21 smithi203 ceph-mon[33115]: pgmap v821: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:47:22.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:21 smithi098 ceph-mon[29162]: pgmap v821: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:47:22.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:21 smithi173 ceph-mon[32556]: pgmap v821: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:47:22.282 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/fpu/xstate.o 2024-04-25T16:47:22.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:22.460+0000 7fdebd7fa640 1 -- 172.21.15.98:0/2834070656 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fdea0005f20 con 0x7fdec8101760 2024-04-25T16:47:22.458 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:22.461+0000 7fdebe7fc640 1 -- 172.21.15.98:0/2834070656 <== mon.0 v2:172.21.15.98:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fdea0005f20 con 0x7fdec8101760 2024-04-25T16:47:22.460 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:22.463+0000 7fdea77fe640 1 -- 172.21.15.98:0/2691354310 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fde98006000 con 0x55e32644d4d0 2024-04-25T16:47:22.461 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:22.463+0000 7fdeb0ff9640 1 -- 172.21.15.98:0/2691354310 <== mon.0 v2:172.21.15.98:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7fde98006000 con 0x55e32644d4d0 2024-04-25T16:47:23.224 INFO:tasks.workunit.client.0.smithi098.stdout: AR security/selinux/built-in.a 2024-04-25T16:47:23.709 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mce/amd.o 2024-04-25T16:47:23.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:23 smithi203 ceph-mon[33115]: pgmap v822: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 155 KiB/s wr, 37 op/s 2024-04-25T16:47:23.973 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/integrity/iint.o 2024-04-25T16:47:24.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:23 smithi098 ceph-mon[29162]: pgmap v822: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 155 KiB/s wr, 37 op/s 2024-04-25T16:47:24.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:23 smithi173 ceph-mon[32556]: pgmap v822: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 155 KiB/s wr, 37 op/s 2024-04-25T16:47:24.521 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/init_64.o 2024-04-25T16:47:24.929 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/fault.o 2024-04-25T16:47:25.116 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/msync.o 2024-04-25T16:47:25.222 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/module/strict_rwx.o 2024-04-25T16:47:25.907 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/events/intel/built-in.a 2024-04-25T16:47:25.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:25 smithi203 ceph-mon[33115]: pgmap v823: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 187 KiB/s wr, 44 op/s 2024-04-25T16:47:26.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:25 smithi098 ceph-mon[29162]: pgmap v823: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 187 KiB/s wr, 44 op/s 2024-04-25T16:47:26.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:25 smithi173 ceph-mon[32556]: pgmap v823: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 187 KiB/s wr, 44 op/s 2024-04-25T16:47:26.689 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/events/built-in.a 2024-04-25T16:47:27.812 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/page_vma_mapped.o 2024-04-25T16:47:27.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:27 smithi203 ceph-mon[33115]: pgmap v824: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 150 KiB/s wr, 38 op/s 2024-04-25T16:47:28.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:27 smithi098 ceph-mon[29162]: pgmap v824: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 150 KiB/s wr, 38 op/s 2024-04-25T16:47:28.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:27 smithi173 ceph-mon[32556]: pgmap v824: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 150 KiB/s wr, 38 op/s 2024-04-25T16:47:28.656 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/fpu/built-in.a 2024-04-25T16:47:29.067 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/pagewalk.o 2024-04-25T16:47:29.231 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/integrity/integrity_audit.o 2024-04-25T16:47:29.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:29 smithi203 ceph-mon[33115]: pgmap v825: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 157 KiB/s wr, 43 op/s 2024-04-25T16:47:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:29 smithi098 ceph-mon[29162]: pgmap v825: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 157 KiB/s wr, 43 op/s 2024-04-25T16:47:30.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:29 smithi173 ceph-mon[32556]: pgmap v825: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 157 KiB/s wr, 43 op/s 2024-04-25T16:47:30.696 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/module/kmod.o 2024-04-25T16:47:31.091 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mce/threshold.o 2024-04-25T16:47:31.653 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/module/tree_lookup.o 2024-04-25T16:47:31.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:31 smithi203 ceph-mon[33115]: pgmap v826: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 188 KiB/s wr, 68 op/s 2024-04-25T16:47:31.955 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/module/kallsyms.o 2024-04-25T16:47:32.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:31 smithi098 ceph-mon[29162]: pgmap v826: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 188 KiB/s wr, 68 op/s 2024-04-25T16:47:32.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:31 smithi173 ceph-mon[32556]: pgmap v826: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 188 KiB/s wr, 68 op/s 2024-04-25T16:47:32.806 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/ioremap.o 2024-04-25T16:47:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:33.492+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:47:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:33.492+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:47:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:33.492+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:47:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:33.492+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:47:33.490 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:33.492+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:47:33.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:33 smithi203 ceph-mon[33115]: pgmap v827: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 164 KiB/s wr, 78 op/s 2024-04-25T16:47:34.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:33 smithi098 ceph-mon[29162]: pgmap v827: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 164 KiB/s wr, 78 op/s 2024-04-25T16:47:34.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:33 smithi173 ceph-mon[32556]: pgmap v827: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 164 KiB/s wr, 78 op/s 2024-04-25T16:47:34.046 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/extable.o 2024-04-25T16:47:34.660 INFO:tasks.workunit.client.0.smithi098.stdout: AR security/integrity/built-in.a 2024-04-25T16:47:34.790 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/commoncap.o 2024-04-25T16:47:35.558 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/pgtable-generic.o 2024-04-25T16:47:35.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:35 smithi203 ceph-mon[33115]: pgmap v828: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 217 KiB/s wr, 182 op/s 2024-04-25T16:47:36.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:35 smithi098 ceph-mon[29162]: pgmap v828: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 217 KiB/s wr, 182 op/s 2024-04-25T16:47:36.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:35 smithi173 ceph-mon[32556]: pgmap v828: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 217 KiB/s wr, 182 op/s 2024-04-25T16:47:36.756 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/rmap.o 2024-04-25T16:47:37.219 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/vmalloc.o 2024-04-25T16:47:37.302 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/module/procfs.o 2024-04-25T16:47:37.700 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/cpu/mce/built-in.a 2024-04-25T16:47:37.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:37 smithi203 ceph-mon[33115]: pgmap v829: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 180 KiB/s wr, 179 op/s 2024-04-25T16:47:38.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:37 smithi098 ceph-mon[29162]: pgmap v829: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 180 KiB/s wr, 179 op/s 2024-04-25T16:47:38.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:37 smithi173 ceph-mon[32556]: pgmap v829: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 180 KiB/s wr, 179 op/s 2024-04-25T16:47:38.251 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mtrr/mtrr.o 2024-04-25T16:47:39.401 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mtrr/if.o 2024-04-25T16:47:39.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:39 smithi203 ceph-mon[33115]: pgmap v830: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 184 KiB/s wr, 179 op/s 2024-04-25T16:47:40.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:39 smithi098 ceph-mon[29162]: pgmap v830: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 184 KiB/s wr, 179 op/s 2024-04-25T16:47:40.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:39 smithi173 ceph-mon[32556]: pgmap v830: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 184 KiB/s wr, 179 op/s 2024-04-25T16:47:40.564 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/mmap.o 2024-04-25T16:47:41.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:41 smithi203 ceph-mon[33115]: pgmap v831: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 195 KiB/s wr, 183 op/s 2024-04-25T16:47:42.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:41 smithi098 ceph-mon[29162]: pgmap v831: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 195 KiB/s wr, 183 op/s 2024-04-25T16:47:42.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:41 smithi173 ceph-mon[32556]: pgmap v831: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 195 KiB/s wr, 183 op/s 2024-04-25T16:47:42.167 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/pgtable.o 2024-04-25T16:47:42.396 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/min_addr.o 2024-04-25T16:47:42.595 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/module/sysfs.o 2024-04-25T16:47:43.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:43 smithi203 ceph-mon[33115]: pgmap v832: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 160 KiB/s wr, 157 op/s 2024-04-25T16:47:44.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:43 smithi098 ceph-mon[29162]: pgmap v832: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 160 KiB/s wr, 157 op/s 2024-04-25T16:47:44.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:43 smithi173 ceph-mon[32556]: pgmap v832: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 160 KiB/s wr, 157 op/s 2024-04-25T16:47:44.262 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/security.o 2024-04-25T16:47:44.326 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/physaddr.o 2024-04-25T16:47:44.569 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mtrr/generic.o 2024-04-25T16:47:45.497 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/process_vm_access.o 2024-04-25T16:47:46.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:45 smithi098 ceph-mon[29162]: pgmap v833: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 181 KiB/s wr, 143 op/s 2024-04-25T16:47:46.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:45 smithi173 ceph-mon[32556]: pgmap v833: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 181 KiB/s wr, 143 op/s 2024-04-25T16:47:46.095 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/page_alloc.o 2024-04-25T16:47:46.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:45 smithi203 ceph-mon[33115]: pgmap v833: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 181 KiB/s wr, 143 op/s 2024-04-25T16:47:47.400 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mtrr/cleanup.o 2024-04-25T16:47:47.999 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/module/built-in.a 2024-04-25T16:47:48.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:47 smithi098 ceph-mon[29162]: pgmap v834: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 135 KiB/s wr, 40 op/s 2024-04-25T16:47:48.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:47 smithi173 ceph-mon[32556]: pgmap v834: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 135 KiB/s wr, 40 op/s 2024-04-25T16:47:48.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:47 smithi203 ceph-mon[33115]: pgmap v834: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 135 KiB/s wr, 40 op/s 2024-04-25T16:47:48.428 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/time.o 2024-04-25T16:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:48.507+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:48.507+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:48.507+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:48.507+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:47:48.505 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:47:48.507+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:47:48.722 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/futex/core.o 2024-04-25T16:47:49.485 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/tlb.o 2024-04-25T16:47:50.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:49 smithi098 ceph-mon[29162]: pgmap v835: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 144 KiB/s wr, 38 op/s 2024-04-25T16:47:50.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:49 smithi173 ceph-mon[32556]: pgmap v835: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 144 KiB/s wr, 38 op/s 2024-04-25T16:47:50.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:49 smithi203 ceph-mon[33115]: pgmap v835: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 144 KiB/s wr, 38 op/s 2024-04-25T16:47:50.642 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/futex/syscalls.o 2024-04-25T16:47:51.942 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/timer.o 2024-04-25T16:47:52.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:51 smithi098 ceph-mon[29162]: pgmap v836: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 193 KiB/s wr, 45 op/s 2024-04-25T16:47:52.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:51 smithi173 ceph-mon[32556]: pgmap v836: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 193 KiB/s wr, 45 op/s 2024-04-25T16:47:52.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:51 smithi203 ceph-mon[33115]: pgmap v836: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 193 KiB/s wr, 45 op/s 2024-04-25T16:47:52.344 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/lsm_audit.o 2024-04-25T16:47:53.348 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/cpu/mtrr/built-in.a 2024-04-25T16:47:53.761 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/microcode/core.o 2024-04-25T16:47:54.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:53 smithi098 ceph-mon[29162]: pgmap v837: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 156 KiB/s wr, 37 op/s 2024-04-25T16:47:54.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:53 smithi173 ceph-mon[32556]: pgmap v837: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 156 KiB/s wr, 37 op/s 2024-04-25T16:47:54.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:53 smithi203 ceph-mon[33115]: pgmap v837: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 156 KiB/s wr, 37 op/s 2024-04-25T16:47:54.303 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/microcode/intel.o 2024-04-25T16:47:54.423 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/init-mm.o 2024-04-25T16:47:55.042 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/memblock.o 2024-04-25T16:47:55.325 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/cpu_entry_area.o 2024-04-25T16:47:56.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:55 smithi098 ceph-mon[29162]: pgmap v838: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:47:56.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:55 smithi173 ceph-mon[32556]: pgmap v838: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:47:56.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:55 smithi203 ceph-mon[33115]: pgmap v838: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 200 KiB/s wr, 45 op/s 2024-04-25T16:47:57.055 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/futex/pi.o 2024-04-25T16:47:58.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:57 smithi098 ceph-mon[29162]: pgmap v839: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 169 KiB/s wr, 39 op/s 2024-04-25T16:47:58.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:57 smithi173 ceph-mon[32556]: pgmap v839: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 169 KiB/s wr, 39 op/s 2024-04-25T16:47:58.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:57 smithi203 ceph-mon[33115]: pgmap v839: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 169 KiB/s wr, 39 op/s 2024-04-25T16:47:59.529 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/hrtimer.o 2024-04-25T16:47:59.592 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/timekeeping.o 2024-04-25T16:47:59.856 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/microcode/amd.o 2024-04-25T16:47:59.987 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:47:59 smithi203 ceph-mon[33115]: pgmap v840: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 172 KiB/s wr, 42 op/s 2024-04-25T16:48:00.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:47:59 smithi098 ceph-mon[29162]: pgmap v840: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 172 KiB/s wr, 42 op/s 2024-04-25T16:48:00.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:47:59 smithi173 ceph-mon[32556]: pgmap v840: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 172 KiB/s wr, 42 op/s 2024-04-25T16:48:00.064 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/madvise.o 2024-04-25T16:48:00.556 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/maccess.o 2024-04-25T16:48:00.579 INFO:tasks.workunit.client.0.smithi098.stdout: CC security/device_cgroup.o 2024-04-25T16:48:01.225 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/futex/requeue.o 2024-04-25T16:48:01.336 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/futex/waitwake.o 2024-04-25T16:48:02.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:01 smithi098 ceph-mon[29162]: pgmap v841: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 192 KiB/s wr, 44 op/s 2024-04-25T16:48:02.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:01 smithi173 ceph-mon[32556]: pgmap v841: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 192 KiB/s wr, 44 op/s 2024-04-25T16:48:02.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:01 smithi203 ceph-mon[33115]: pgmap v841: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 192 KiB/s wr, 44 op/s 2024-04-25T16:48:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:03.522+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:48:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:03.522+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:48:03.520 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:03.522+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:48:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:03.522+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:48:03.521 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:03.523+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:48:04.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:03 smithi098 ceph-mon[29162]: pgmap v842: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 131 KiB/s wr, 35 op/s 2024-04-25T16:48:04.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:03 smithi173 ceph-mon[32556]: pgmap v842: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 131 KiB/s wr, 35 op/s 2024-04-25T16:48:04.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:03 smithi203 ceph-mon[33115]: pgmap v842: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 131 KiB/s wr, 35 op/s 2024-04-25T16:48:04.205 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/pgprot.o 2024-04-25T16:48:05.236 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/hugetlbpage.o 2024-04-25T16:48:05.417 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/futex/built-in.a 2024-04-25T16:48:05.629 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/ntp.o 2024-04-25T16:48:05.729 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/cpu/microcode/built-in.a 2024-04-25T16:48:05.944 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/acpi/boot.o 2024-04-25T16:48:06.068 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:05 smithi173 ceph-mon[32556]: pgmap v843: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 195 KiB/s wr, 44 op/s 2024-04-25T16:48:06.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:05 smithi098 ceph-mon[29162]: pgmap v843: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 195 KiB/s wr, 44 op/s 2024-04-25T16:48:06.068 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/cacheinfo.o 2024-04-25T16:48:06.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:05 smithi203 ceph-mon[33115]: pgmap v843: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 195 KiB/s wr, 44 op/s 2024-04-25T16:48:06.560 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/acpi/sleep.o 2024-04-25T16:48:07.190 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/page_io.o 2024-04-25T16:48:08.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:07 smithi098 ceph-mon[29162]: pgmap v844: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 148 KiB/s wr, 35 op/s 2024-04-25T16:48:08.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:07 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:48:08.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:07 smithi173 ceph-mon[32556]: pgmap v844: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 148 KiB/s wr, 35 op/s 2024-04-25T16:48:08.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:07 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:48:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:07 smithi203 ceph-mon[33115]: pgmap v844: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 148 KiB/s wr, 35 op/s 2024-04-25T16:48:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:07 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:48:08.247 INFO:tasks.workunit.client.0.smithi098.stdout: AR security/built-in.a 2024-04-25T16:48:09.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:48:09.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:48:09.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:48:09.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:48:09.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:48:09.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:48:09.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:48:09.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:48:09.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:48:09.213 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/clocksource.o 2024-04-25T16:48:09.260 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/swap_state.o 2024-04-25T16:48:10.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:09 smithi098 ceph-mon[29162]: pgmap v845: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 155 KiB/s wr, 37 op/s 2024-04-25T16:48:10.014 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:09 smithi173 ceph-mon[32556]: pgmap v845: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 155 KiB/s wr, 37 op/s 2024-04-25T16:48:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:09 smithi203 ceph-mon[33115]: pgmap v845: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 155 KiB/s wr, 37 op/s 2024-04-25T16:48:11.615 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/dump_pagetables.o 2024-04-25T16:48:11.654 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/numa.o 2024-04-25T16:48:12.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:11 smithi098 ceph-mon[29162]: pgmap v846: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 190 KiB/s wr, 43 op/s 2024-04-25T16:48:12.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:11 smithi173 ceph-mon[32556]: pgmap v846: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 190 KiB/s wr, 43 op/s 2024-04-25T16:48:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:11 smithi203 ceph-mon[33115]: pgmap v846: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 190 KiB/s wr, 43 op/s 2024-04-25T16:48:12.389 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/scattered.o 2024-04-25T16:48:12.753 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/numa_64.o 2024-04-25T16:48:13.175 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/kernel/acpi/wakeup_64.o 2024-04-25T16:48:13.426 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/acpi/cppc.o 2024-04-25T16:48:13.705 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/swapfile.o 2024-04-25T16:48:14.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:13 smithi098 ceph-mon[29162]: pgmap v847: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 161 KiB/s wr, 39 op/s 2024-04-25T16:48:14.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:13 smithi173 ceph-mon[32556]: pgmap v847: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 161 KiB/s wr, 39 op/s 2024-04-25T16:48:14.078 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/jiffies.o 2024-04-25T16:48:14.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:13 smithi203 ceph-mon[33115]: pgmap v847: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 161 KiB/s wr, 39 op/s 2024-04-25T16:48:15.602 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/timer_list.o 2024-04-25T16:48:16.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:15 smithi098 ceph-mon[29162]: pgmap v848: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 229 KiB/s wr, 49 op/s 2024-04-25T16:48:16.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:15 smithi173 ceph-mon[32556]: pgmap v848: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 229 KiB/s wr, 49 op/s 2024-04-25T16:48:16.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:15 smithi203 ceph-mon[33115]: pgmap v848: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 229 KiB/s wr, 49 op/s 2024-04-25T16:48:16.650 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/topology.o 2024-04-25T16:48:17.479 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/common.o 2024-04-25T16:48:17.759 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/acpi/cstate.o 2024-04-25T16:48:17.939 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/amdtopology.o 2024-04-25T16:48:18.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:17 smithi098 ceph-mon[29162]: pgmap v849: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 164 KiB/s wr, 40 op/s 2024-04-25T16:48:18.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:17 smithi173 ceph-mon[32556]: pgmap v849: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 164 KiB/s wr, 40 op/s 2024-04-25T16:48:18.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:17 smithi203 ceph-mon[33115]: pgmap v849: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 164 KiB/s wr, 40 op/s 2024-04-25T16:48:18.426 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/srat.o 2024-04-25T16:48:18.474 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/rdrand.o 2024-04-25T16:48:18.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:18.538+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:48:18.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:18.538+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:48:18.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:18.538+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:48:18.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:18.538+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:48:18.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:18.538+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:48:20.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:19 smithi098 ceph-mon[29162]: pgmap v850: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 176 KiB/s wr, 43 op/s 2024-04-25T16:48:20.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:19 smithi173 ceph-mon[32556]: pgmap v850: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 176 KiB/s wr, 43 op/s 2024-04-25T16:48:20.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:19 smithi203 ceph-mon[33115]: pgmap v850: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 176 KiB/s wr, 43 op/s 2024-04-25T16:48:20.486 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/swap_slots.o 2024-04-25T16:48:21.091 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/timeconv.o 2024-04-25T16:48:21.210 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/match.o 2024-04-25T16:48:21.672 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/cgroup.o 2024-04-25T16:48:22.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:21 smithi098 ceph-mon[29162]: pgmap v851: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 207 KiB/s wr, 49 op/s 2024-04-25T16:48:22.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:21 smithi173 ceph-mon[32556]: pgmap v851: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 207 KiB/s wr, 49 op/s 2024-04-25T16:48:22.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:21 smithi203 ceph-mon[33115]: pgmap v851: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 207 KiB/s wr, 49 op/s 2024-04-25T16:48:22.825 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/acpi/built-in.a 2024-04-25T16:48:23.173 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/rstat.o 2024-04-25T16:48:24.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:23 smithi098 ceph-mon[29162]: pgmap v852: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 159 KiB/s wr, 40 op/s 2024-04-25T16:48:24.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:23 smithi173 ceph-mon[32556]: pgmap v852: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 159 KiB/s wr, 40 op/s 2024-04-25T16:48:24.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:23 smithi203 ceph-mon[33115]: pgmap v852: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 159 KiB/s wr, 40 op/s 2024-04-25T16:48:24.211 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/apic.o 2024-04-25T16:48:24.250 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/pkeys.o 2024-04-25T16:48:25.218 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/apic_common.o 2024-04-25T16:48:25.294 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/timecounter.o 2024-04-25T16:48:25.744 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/alarmtimer.o 2024-04-25T16:48:25.876 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/bugs.o 2024-04-25T16:48:26.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:25 smithi098 ceph-mon[29162]: pgmap v853: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 185 KiB/s wr, 46 op/s 2024-04-25T16:48:26.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:25 smithi173 ceph-mon[32556]: pgmap v853: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 185 KiB/s wr, 46 op/s 2024-04-25T16:48:26.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:25 smithi203 ceph-mon[33115]: pgmap v853: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 185 KiB/s wr, 46 op/s 2024-04-25T16:48:26.518 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/dmapool.o 2024-04-25T16:48:28.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:27 smithi098 ceph-mon[29162]: pgmap v854: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 131 KiB/s wr, 37 op/s 2024-04-25T16:48:28.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:27 smithi173 ceph-mon[32556]: pgmap v854: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 131 KiB/s wr, 37 op/s 2024-04-25T16:48:28.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:27 smithi203 ceph-mon[33115]: pgmap v854: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 131 KiB/s wr, 37 op/s 2024-04-25T16:48:29.487 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/aperfmperf.o 2024-04-25T16:48:29.499 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/cpuid-deps.o 2024-04-25T16:48:29.866 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/kaslr.o 2024-04-25T16:48:30.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:29 smithi098 ceph-mon[29162]: pgmap v855: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 153 KiB/s wr, 42 op/s 2024-04-25T16:48:30.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:29 smithi173 ceph-mon[32556]: pgmap v855: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 153 KiB/s wr, 42 op/s 2024-04-25T16:48:30.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:29 smithi203 ceph-mon[33115]: pgmap v855: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 153 KiB/s wr, 42 op/s 2024-04-25T16:48:31.876 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/hugetlb.o 2024-04-25T16:48:32.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:31 smithi098 ceph-mon[29162]: pgmap v856: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 171 KiB/s wr, 43 op/s 2024-04-25T16:48:32.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:31 smithi173 ceph-mon[32556]: pgmap v856: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 171 KiB/s wr, 43 op/s 2024-04-25T16:48:32.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:31 smithi203 ceph-mon[33115]: pgmap v856: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 171 KiB/s wr, 43 op/s 2024-04-25T16:48:32.193 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/namespace.o 2024-04-25T16:48:32.324 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/apic_noop.o 2024-04-25T16:48:32.611 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/ipi.o 2024-04-25T16:48:32.668 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/posix-timers.o 2024-04-25T16:48:33.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:33.553+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:48:33.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:33.553+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:48:33.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:33.553+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:48:33.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:33.553+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:48:33.551 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:33.553+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:48:33.709 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/posix-cpu-timers.o 2024-04-25T16:48:34.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:33 smithi173 ceph-mon[32556]: pgmap v857: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 129 KiB/s wr, 35 op/s 2024-04-25T16:48:34.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:33 smithi203 ceph-mon[33115]: pgmap v857: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 129 KiB/s wr, 35 op/s 2024-04-25T16:48:34.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:33 smithi098 ceph-mon[29162]: pgmap v857: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 129 KiB/s wr, 35 op/s 2024-04-25T16:48:34.367 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/umwait.o 2024-04-25T16:48:35.250 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/mm/pti.o 2024-04-25T16:48:35.580 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/posix-clock.o 2024-04-25T16:48:35.949 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/vector.o 2024-04-25T16:48:36.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:35 smithi173 ceph-mon[32556]: pgmap v858: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 214 KiB/s wr, 107 op/s 2024-04-25T16:48:36.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:35 smithi098 ceph-mon[29162]: pgmap v858: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 214 KiB/s wr, 107 op/s 2024-04-25T16:48:36.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:35 smithi203 ceph-mon[33115]: pgmap v858: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 214 KiB/s wr, 107 op/s 2024-04-25T16:48:37.912 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/cgroup-v1.o 2024-04-25T16:48:38.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:37 smithi173 ceph-mon[32556]: pgmap v859: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 190 KiB/s wr, 121 op/s 2024-04-25T16:48:38.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:37 smithi203 ceph-mon[33115]: pgmap v859: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 190 KiB/s wr, 121 op/s 2024-04-25T16:48:38.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:37 smithi098 ceph-mon[29162]: pgmap v859: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 190 KiB/s wr, 121 op/s 2024-04-25T16:48:39.808 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/hw_nmi.o 2024-04-25T16:48:39.832 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/io_apic.o 2024-04-25T16:48:40.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:39 smithi173 ceph-mon[32556]: pgmap v860: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 195 KiB/s wr, 168 op/s 2024-04-25T16:48:40.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:39 smithi203 ceph-mon[33115]: pgmap v860: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 195 KiB/s wr, 168 op/s 2024-04-25T16:48:40.226 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/proc.o 2024-04-25T16:48:40.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:39 smithi098 ceph-mon[29162]: pgmap v860: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 195 KiB/s wr, 168 op/s 2024-04-25T16:48:40.621 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/hugetlb_vmemmap.o 2024-04-25T16:48:40.908 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/mm/built-in.a 2024-04-25T16:48:41.648 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/itimer.o 2024-04-25T16:48:41.987 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mempolicy.o 2024-04-25T16:48:42.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:41 smithi173 ceph-mon[32556]: pgmap v861: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 197 KiB/s wr, 171 op/s 2024-04-25T16:48:42.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:41 smithi203 ceph-mon[33115]: pgmap v861: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 197 KiB/s wr, 171 op/s 2024-04-25T16:48:42.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:41 smithi098 ceph-mon[29162]: pgmap v861: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 197 KiB/s wr, 171 op/s 2024-04-25T16:48:42.957 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/crypto/built-in.a 2024-04-25T16:48:43.174 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_clock.o 2024-04-25T16:48:44.003 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/freezer.o 2024-04-25T16:48:44.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:43 smithi203 ceph-mon[33115]: pgmap v862: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 166 KiB/s wr, 167 op/s 2024-04-25T16:48:44.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:43 smithi098 ceph-mon[29162]: pgmap v862: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 166 KiB/s wr, 167 op/s 2024-04-25T16:48:44.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:43 smithi173 ceph-mon[32556]: pgmap v862: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 166 KiB/s wr, 167 op/s 2024-04-25T16:48:45.499 INFO:tasks.workunit.client.0.smithi098.stdout: MKCAP arch/x86/kernel/cpu/capflags.c 2024-04-25T16:48:45.686 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/bpf/core.o 2024-04-25T16:48:46.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:45 smithi203 ceph-mon[33115]: pgmap v863: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 230 KiB/s wr, 176 op/s 2024-04-25T16:48:46.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:45 smithi098 ceph-mon[29162]: pgmap v863: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 230 KiB/s wr, 176 op/s 2024-04-25T16:48:46.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:45 smithi173 ceph-mon[32556]: pgmap v863: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 230 KiB/s wr, 176 op/s 2024-04-25T16:48:46.785 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/events/core.o 2024-04-25T16:48:46.927 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/msi.o 2024-04-25T16:48:47.306 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/ring_buffer.o 2024-04-25T16:48:48.109 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/clockevents.o 2024-04-25T16:48:48.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:47 smithi203 ceph-mon[33115]: pgmap v864: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 149 KiB/s wr, 103 op/s 2024-04-25T16:48:48.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:47 smithi098 ceph-mon[29162]: pgmap v864: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 149 KiB/s wr, 103 op/s 2024-04-25T16:48:48.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:47 smithi173 ceph-mon[32556]: pgmap v864: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 149 KiB/s wr, 103 op/s 2024-04-25T16:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:48.568+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:48.568+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:48.568+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:48.568+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:48:48.566 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:48:48.568+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:48:49.694 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/sparse.o 2024-04-25T16:48:49.897 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/powerflags.o 2024-04-25T16:48:49.899 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/legacy_freezer.o 2024-04-25T16:48:50.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:49 smithi203 ceph-mon[33115]: pgmap v865: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 160 KiB/s wr, 87 op/s 2024-04-25T16:48:50.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:49 smithi098 ceph-mon[29162]: pgmap v865: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 160 KiB/s wr, 87 op/s 2024-04-25T16:48:50.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:49 smithi173 ceph-mon[32556]: pgmap v865: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 160 KiB/s wr, 87 op/s 2024-04-25T16:48:51.846 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/feat_ctl.o 2024-04-25T16:48:52.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:51 smithi203 ceph-mon[33115]: pgmap v866: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 186 KiB/s wr, 47 op/s 2024-04-25T16:48:52.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:51 smithi098 ceph-mon[29162]: pgmap v866: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 186 KiB/s wr, 47 op/s 2024-04-25T16:48:52.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:51 smithi173 ceph-mon[32556]: pgmap v866: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 186 KiB/s wr, 47 op/s 2024-04-25T16:48:52.699 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/tick-common.o 2024-04-25T16:48:52.975 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/apic_flat_64.o 2024-04-25T16:48:54.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:53 smithi203 ceph-mon[33115]: pgmap v867: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 163 KiB/s wr, 39 op/s 2024-04-25T16:48:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:53 smithi098 ceph-mon[29162]: pgmap v867: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 163 KiB/s wr, 39 op/s 2024-04-25T16:48:54.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:53 smithi173 ceph-mon[32556]: pgmap v867: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 163 KiB/s wr, 39 op/s 2024-04-25T16:48:54.389 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace.o 2024-04-25T16:48:54.698 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/bpf/built-in.a 2024-04-25T16:48:54.798 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/apic/probe_64.o 2024-04-25T16:48:55.174 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/pids.o 2024-04-25T16:48:55.554 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/sparse-vmemmap.o 2024-04-25T16:48:56.022 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/intel.o 2024-04-25T16:48:56.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:55 smithi203 ceph-mon[33115]: pgmap v868: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 187 KiB/s wr, 45 op/s 2024-04-25T16:48:56.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:55 smithi098 ceph-mon[29162]: pgmap v868: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 187 KiB/s wr, 45 op/s 2024-04-25T16:48:56.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:55 smithi173 ceph-mon[32556]: pgmap v868: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 187 KiB/s wr, 45 op/s 2024-04-25T16:48:57.675 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_output.o 2024-04-25T16:48:57.780 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/apic/built-in.a 2024-04-25T16:48:58.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:57 smithi203 ceph-mon[33115]: pgmap v869: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 130 KiB/s wr, 37 op/s 2024-04-25T16:48:58.238 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/events/ring_buffer.o 2024-04-25T16:48:58.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:57 smithi098 ceph-mon[29162]: pgmap v869: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 130 KiB/s wr, 37 op/s 2024-04-25T16:48:58.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:57 smithi173 ceph-mon[32556]: pgmap v869: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 130 KiB/s wr, 37 op/s 2024-04-25T16:48:58.391 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_seq.o 2024-04-25T16:48:59.127 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/tick-broadcast.o 2024-04-25T16:49:00.152 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/rdma.o 2024-04-25T16:49:00.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:48:59 smithi203 ceph-mon[33115]: pgmap v870: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 154 KiB/s wr, 41 op/s 2024-04-25T16:49:00.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:48:59 smithi098 ceph-mon[29162]: pgmap v870: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 154 KiB/s wr, 41 op/s 2024-04-25T16:49:00.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:48:59 smithi173 ceph-mon[32556]: pgmap v870: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 154 KiB/s wr, 41 op/s 2024-04-25T16:49:00.751 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/mmu_notifier.o 2024-04-25T16:49:02.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:01 smithi173 ceph-mon[32556]: pgmap v871: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 202 KiB/s wr, 44 op/s 2024-04-25T16:49:02.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:01 smithi203 ceph-mon[33115]: pgmap v871: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 202 KiB/s wr, 44 op/s 2024-04-25T16:49:02.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:01 smithi098 ceph-mon[29162]: pgmap v871: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 202 KiB/s wr, 44 op/s 2024-04-25T16:49:02.695 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/intel_pconfig.o 2024-04-25T16:49:03.027 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_stat.o 2024-04-25T16:49:03.066 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/tsx.o 2024-04-25T16:49:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:03.582+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:49:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:03.582+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:49:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:03.582+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:49:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:03.582+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:49:03.581 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:03.582+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:49:03.796 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/tick-broadcast-hrtimer.o 2024-04-25T16:49:04.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:03 smithi203 ceph-mon[33115]: pgmap v872: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 159 KiB/s wr, 35 op/s 2024-04-25T16:49:04.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:03 smithi098 ceph-mon[29162]: pgmap v872: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 159 KiB/s wr, 35 op/s 2024-04-25T16:49:04.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:03 smithi173 ceph-mon[32556]: pgmap v872: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 159 KiB/s wr, 35 op/s 2024-04-25T16:49:04.423 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/events/callchain.o 2024-04-25T16:49:04.430 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/cpuset.o 2024-04-25T16:49:04.813 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/kprobes/core.o 2024-04-25T16:49:05.248 INFO:tasks.workunit.client.0.smithi098.stdout: LDS arch/x86/kernel/vmlinux.lds 2024-04-25T16:49:05.759 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/misc.o 2024-04-25T16:49:05.907 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/slub.o 2024-04-25T16:49:06.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:05 smithi098 ceph-mon[29162]: pgmap v873: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 219 KiB/s wr, 43 op/s 2024-04-25T16:49:06.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:05 smithi203 ceph-mon[33115]: pgmap v873: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 219 KiB/s wr, 43 op/s 2024-04-25T16:49:06.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:05 smithi173 ceph-mon[32556]: pgmap v873: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 219 KiB/s wr, 43 op/s 2024-04-25T16:49:07.645 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/intel_epb.o 2024-04-25T16:49:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:07 smithi203 ceph-mon[33115]: pgmap v874: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 203 KiB/s wr, 37 op/s 2024-04-25T16:49:08.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:07 smithi098 ceph-mon[29162]: pgmap v874: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 203 KiB/s wr, 37 op/s 2024-04-25T16:49:08.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:07 smithi173 ceph-mon[32556]: pgmap v874: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 203 KiB/s wr, 37 op/s 2024-04-25T16:49:08.629 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/tick-oneshot.o 2024-04-25T16:49:09.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:08 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:49:09.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:08 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:49:09.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:08 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:49:09.832 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_printk.o 2024-04-25T16:49:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:09 smithi203 ceph-mon[33115]: pgmap v875: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 204 KiB/s wr, 40 op/s 2024-04-25T16:49:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:49:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:49:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:49:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:09 smithi098 ceph-mon[29162]: pgmap v875: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 204 KiB/s wr, 40 op/s 2024-04-25T16:49:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:49:10.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:49:10.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:49:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:09 smithi173 ceph-mon[32556]: pgmap v875: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 204 KiB/s wr, 40 op/s 2024-04-25T16:49:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:49:10.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:49:10.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:49:10.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:49:10.820 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/events/hw_breakpoint.o 2024-04-25T16:49:11.083 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/events/uprobes.o 2024-04-25T16:49:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:10 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T16:49:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:10 smithi203 ceph-mon[33115]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T16:49:11.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:10 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T16:49:11.206 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:11 smithi098 ceph-mon[29162]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T16:49:11.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:10 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi098 to 176.2M 2024-04-25T16:49:11.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:10 smithi173 ceph-mon[32556]: Unable to set osd_memory_target on smithi098 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-25T16:49:11.617 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/kprobes/opt.o 2024-04-25T16:49:11.891 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cgroup/debug.o 2024-04-25T16:49:12.139 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/amd.o 2024-04-25T16:49:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:11 smithi203 ceph-mon[33115]: pgmap v876: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 206 KiB/s wr, 42 op/s 2024-04-25T16:49:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:11 smithi098 ceph-mon[29162]: pgmap v876: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 206 KiB/s wr, 42 op/s 2024-04-25T16:49:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:11 smithi173 ceph-mon[32556]: pgmap v876: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 206 KiB/s wr, 42 op/s 2024-04-25T16:49:13.433 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/tick-sched.o 2024-04-25T16:49:13.982 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/migrate.o 2024-04-25T16:49:14.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:13 smithi203 ceph-mon[33115]: pgmap v877: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 152 KiB/s wr, 35 op/s 2024-04-25T16:49:14.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:13 smithi098 ceph-mon[29162]: pgmap v877: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 152 KiB/s wr, 35 op/s 2024-04-25T16:49:14.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:13 smithi173 ceph-mon[32556]: pgmap v877: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 152 KiB/s wr, 35 op/s 2024-04-25T16:49:16.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:15 smithi203 ceph-mon[33115]: pgmap v878: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 191 KiB/s wr, 43 op/s 2024-04-25T16:49:16.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:15 smithi098 ceph-mon[29162]: pgmap v878: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 191 KiB/s wr, 43 op/s 2024-04-25T16:49:16.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:15 smithi173 ceph-mon[32556]: pgmap v878: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 191 KiB/s wr, 43 op/s 2024-04-25T16:49:16.321 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/pid_list.o 2024-04-25T16:49:17.134 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/memory-tiers.o 2024-04-25T16:49:17.345 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/cgroup/built-in.a 2024-04-25T16:49:17.748 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/page_counter.o 2024-04-25T16:49:17.846 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/events/built-in.a 2024-04-25T16:49:17.899 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/hygon.o 2024-04-25T16:49:17.999 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/kprobes/built-in.a 2024-04-25T16:49:18.042 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/hugetlb_cgroup.o 2024-04-25T16:49:18.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:17 smithi203 ceph-mon[33115]: pgmap v879: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 145 KiB/s wr, 36 op/s 2024-04-25T16:49:18.214 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/fork.o 2024-04-25T16:49:18.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:17 smithi098 ceph-mon[29162]: pgmap v879: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 145 KiB/s wr, 36 op/s 2024-04-25T16:49:18.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:17 smithi173 ceph-mon[32556]: pgmap v879: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 145 KiB/s wr, 36 op/s 2024-04-25T16:49:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:18.598+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:49:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:18.598+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:49:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:18.598+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:49:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:18.598+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:49:18.596 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:18.598+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:49:19.414 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/vsyscall.o 2024-04-25T16:49:20.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:19 smithi203 ceph-mon[33115]: pgmap v880: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 159 KiB/s wr, 39 op/s 2024-04-25T16:49:20.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:19 smithi098 ceph-mon[29162]: pgmap v880: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 159 KiB/s wr, 39 op/s 2024-04-25T16:49:20.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:19 smithi173 ceph-mon[32556]: pgmap v880: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 159 KiB/s wr, 39 op/s 2024-04-25T16:49:21.116 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/kernel/head_64.o 2024-04-25T16:49:21.591 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/timekeeping_debug.o 2024-04-25T16:49:21.865 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/early_ioremap.o 2024-04-25T16:49:22.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:21 smithi203 ceph-mon[33115]: pgmap v881: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 197 KiB/s wr, 44 op/s 2024-04-25T16:49:22.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:21 smithi098 ceph-mon[29162]: pgmap v881: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 197 KiB/s wr, 44 op/s 2024-04-25T16:49:22.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:21 smithi173 ceph-mon[32556]: pgmap v881: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 197 KiB/s wr, 44 op/s 2024-04-25T16:49:22.484 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/centaur.o 2024-04-25T16:49:22.867 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_sched_switch.o 2024-04-25T16:49:23.108 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/zhaoxin.o 2024-04-25T16:49:23.790 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_nop.o 2024-04-25T16:49:24.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:23 smithi203 ceph-mon[33115]: pgmap v882: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 172 KiB/s wr, 39 op/s 2024-04-25T16:49:24.199 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/blktrace.o 2024-04-25T16:49:24.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:23 smithi098 ceph-mon[29162]: pgmap v882: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 172 KiB/s wr, 39 op/s 2024-04-25T16:49:24.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:23 smithi173 ceph-mon[32556]: pgmap v882: 129 pgs: 129 active+clean; 1.7 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 172 KiB/s wr, 39 op/s 2024-04-25T16:49:26.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:25 smithi203 ceph-mon[33115]: pgmap v883: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 205 KiB/s wr, 46 op/s 2024-04-25T16:49:26.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:25 smithi098 ceph-mon[29162]: pgmap v883: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 205 KiB/s wr, 46 op/s 2024-04-25T16:49:26.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:25 smithi173 ceph-mon[32556]: pgmap v883: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 205 KiB/s wr, 46 op/s 2024-04-25T16:49:27.390 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_events.o 2024-04-25T16:49:27.582 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/time/namespace.o 2024-04-25T16:49:27.812 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/secretmem.o 2024-04-25T16:49:27.909 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/perfctr-watchdog.o 2024-04-25T16:49:27.999 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/memfd.o 2024-04-25T16:49:28.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:27 smithi203 ceph-mon[33115]: pgmap v884: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 191 KiB/s wr, 40 op/s 2024-04-25T16:49:28.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:27 smithi098 ceph-mon[29162]: pgmap v884: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 191 KiB/s wr, 40 op/s 2024-04-25T16:49:28.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:27 smithi173 ceph-mon[32556]: pgmap v884: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 191 KiB/s wr, 40 op/s 2024-04-25T16:49:29.341 INFO:tasks.workunit.client.0.smithi098.stdout: CC mm/ptdump.o 2024-04-25T16:49:30.104 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/ia32/audit.o 2024-04-25T16:49:30.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:29 smithi203 ceph-mon[33115]: pgmap v885: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 194 KiB/s wr, 42 op/s 2024-04-25T16:49:30.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:29 smithi098 ceph-mon[29162]: pgmap v885: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 194 KiB/s wr, 42 op/s 2024-04-25T16:49:30.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:29 smithi173 ceph-mon[32556]: pgmap v885: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 194 KiB/s wr, 42 op/s 2024-04-25T16:49:30.434 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/ia32/built-in.a 2024-04-25T16:49:30.572 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_export.o 2024-04-25T16:49:31.716 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/atom/built-in.a 2024-04-25T16:49:31.837 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/ce4100/built-in.a 2024-04-25T16:49:32.009 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/platform/efi/memmap.o 2024-04-25T16:49:32.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:31 smithi203 ceph-mon[33115]: pgmap v886: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 204 KiB/s wr, 44 op/s 2024-04-25T16:49:32.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:31 smithi098 ceph-mon[29162]: pgmap v886: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 204 KiB/s wr, 44 op/s 2024-04-25T16:49:32.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:31 smithi173 ceph-mon[32556]: pgmap v886: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 204 KiB/s wr, 44 op/s 2024-04-25T16:49:32.988 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/time/built-in.a 2024-04-25T16:49:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:33.613+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:49:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:33.613+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:49:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:33.613+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:49:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:33.613+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:49:33.611 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:33.613+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:49:33.682 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/vmware.o 2024-04-25T16:49:33.776 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/platform/efi/quirks.o 2024-04-25T16:49:34.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:33 smithi203 ceph-mon[33115]: pgmap v887: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 161 KiB/s wr, 39 op/s 2024-04-25T16:49:34.240 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/platform/efi/efi.o 2024-04-25T16:49:34.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:33 smithi098 ceph-mon[29162]: pgmap v887: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 161 KiB/s wr, 39 op/s 2024-04-25T16:49:34.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:33 smithi173 ceph-mon[32556]: pgmap v887: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 161 KiB/s wr, 39 op/s 2024-04-25T16:49:34.359 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/head64.o 2024-04-25T16:49:34.788 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/platform/efi/efi_64.o 2024-04-25T16:49:34.848 INFO:tasks.workunit.client.0.smithi098.stdout: AR mm/built-in.a 2024-04-25T16:49:36.100 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/platform/efi/efi_stub_64.o 2024-04-25T16:49:36.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:35 smithi203 ceph-mon[33115]: pgmap v888: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 204 KiB/s wr, 47 op/s 2024-04-25T16:49:36.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:35 smithi098 ceph-mon[29162]: pgmap v888: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 204 KiB/s wr, 47 op/s 2024-04-25T16:49:36.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:35 smithi173 ceph-mon[32556]: pgmap v888: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 204 KiB/s wr, 47 op/s 2024-04-25T16:49:36.570 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/exec_domain.o 2024-04-25T16:49:36.895 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_event_perf.o 2024-04-25T16:49:38.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:37 smithi203 ceph-mon[33115]: pgmap v889: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 173 KiB/s wr, 40 op/s 2024-04-25T16:49:38.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:37 smithi098 ceph-mon[29162]: pgmap v889: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 173 KiB/s wr, 40 op/s 2024-04-25T16:49:38.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:37 smithi173 ceph-mon[32556]: pgmap v889: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 173 KiB/s wr, 40 op/s 2024-04-25T16:49:38.298 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/panic.o 2024-04-25T16:49:38.520 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/hypervisor.o 2024-04-25T16:49:40.023 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/mshyperv.o 2024-04-25T16:49:40.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:39 smithi203 ceph-mon[33115]: pgmap v890: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 163 KiB/s wr, 42 op/s 2024-04-25T16:49:40.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:39 smithi098 ceph-mon[29162]: pgmap v890: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 163 KiB/s wr, 42 op/s 2024-04-25T16:49:40.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:39 smithi173 ceph-mon[32556]: pgmap v890: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 163 KiB/s wr, 42 op/s 2024-04-25T16:49:40.694 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/ebda.o 2024-04-25T16:49:41.175 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/platform-quirks.o 2024-04-25T16:49:41.876 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/platform/efi/efi_thunk_64.o 2024-04-25T16:49:42.141 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/platform/efi/runtime-map.o 2024-04-25T16:49:42.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:41 smithi203 ceph-mon[33115]: pgmap v891: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 194 KiB/s wr, 45 op/s 2024-04-25T16:49:42.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:41 smithi098 ceph-mon[29162]: pgmap v891: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 194 KiB/s wr, 45 op/s 2024-04-25T16:49:42.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:41 smithi173 ceph-mon[32556]: pgmap v891: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 194 KiB/s wr, 45 op/s 2024-04-25T16:49:42.274 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/process_64.o 2024-04-25T16:49:42.827 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cpu.o 2024-04-25T16:49:43.642 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_events_filter.o 2024-04-25T16:49:44.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:43 smithi203 ceph-mon[33115]: pgmap v892: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 166 KiB/s wr, 40 op/s 2024-04-25T16:49:44.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:43 smithi098 ceph-mon[29162]: pgmap v892: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 166 KiB/s wr, 40 op/s 2024-04-25T16:49:44.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:43 smithi173 ceph-mon[32556]: pgmap v892: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 166 KiB/s wr, 40 op/s 2024-04-25T16:49:45.037 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpu/capflags.o 2024-04-25T16:49:45.227 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/signal.o 2024-04-25T16:49:45.647 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_events_trigger.o 2024-04-25T16:49:45.831 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/geode/built-in.a 2024-04-25T16:49:45.933 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/signal_64.o 2024-04-25T16:49:46.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:45 smithi203 ceph-mon[33115]: pgmap v893: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 199 KiB/s wr, 46 op/s 2024-04-25T16:49:46.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:45 smithi098 ceph-mon[29162]: pgmap v893: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 199 KiB/s wr, 46 op/s 2024-04-25T16:49:46.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:45 smithi173 ceph-mon[32556]: pgmap v893: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 199 KiB/s wr, 46 op/s 2024-04-25T16:49:47.135 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/cpu/built-in.a 2024-04-25T16:49:48.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:47 smithi203 ceph-mon[33115]: pgmap v894: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 154 KiB/s wr, 35 op/s 2024-04-25T16:49:48.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:47 smithi098 ceph-mon[29162]: pgmap v894: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 154 KiB/s wr, 35 op/s 2024-04-25T16:49:48.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:47 smithi173 ceph-mon[32556]: pgmap v894: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 154 KiB/s wr, 35 op/s 2024-04-25T16:49:48.431 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/efi/built-in.a 2024-04-25T16:49:48.625 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:48.628+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:49:48.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:48.628+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:49:48.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:48.628+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:49:48.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:48.628+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:49:48.626 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:49:48.628+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:49:48.892 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/iris/built-in.a 2024-04-25T16:49:49.021 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/platform/intel/iosf_mbi.o 2024-04-25T16:49:49.297 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/traps.o 2024-04-25T16:49:49.457 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_eprobe.o 2024-04-25T16:49:50.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:49 smithi203 ceph-mon[33115]: pgmap v895: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 177 KiB/s wr, 37 op/s 2024-04-25T16:49:50.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:49 smithi098 ceph-mon[29162]: pgmap v895: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 177 KiB/s wr, 37 op/s 2024-04-25T16:49:50.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:49 smithi173 ceph-mon[32556]: pgmap v895: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 177 KiB/s wr, 37 op/s 2024-04-25T16:49:50.602 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/idt.o 2024-04-25T16:49:50.784 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/irq.o 2024-04-25T16:49:51.932 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_kprobe.o 2024-04-25T16:49:52.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:51 smithi203 ceph-mon[33115]: pgmap v896: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 187 KiB/s wr, 43 op/s 2024-04-25T16:49:52.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:51 smithi098 ceph-mon[29162]: pgmap v896: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 187 KiB/s wr, 43 op/s 2024-04-25T16:49:52.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:51 smithi173 ceph-mon[32556]: pgmap v896: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 187 KiB/s wr, 43 op/s 2024-04-25T16:49:52.450 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/error_report-traces.o 2024-04-25T16:49:52.559 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/power-traces.o 2024-04-25T16:49:54.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:53 smithi203 ceph-mon[33115]: pgmap v897: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 148 KiB/s wr, 37 op/s 2024-04-25T16:49:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:53 smithi098 ceph-mon[29162]: pgmap v897: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 148 KiB/s wr, 37 op/s 2024-04-25T16:49:54.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:53 smithi173 ceph-mon[32556]: pgmap v897: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 148 KiB/s wr, 37 op/s 2024-04-25T16:49:55.032 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/intel/built-in.a 2024-04-25T16:49:55.198 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/intel-mid/built-in.a 2024-04-25T16:49:55.318 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/intel-quark/built-in.a 2024-04-25T16:49:55.444 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/olpc/built-in.a 2024-04-25T16:49:55.607 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/scx200/built-in.a 2024-04-25T16:49:55.762 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/ts5500/built-in.a 2024-04-25T16:49:55.895 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/uv/built-in.a 2024-04-25T16:49:55.998 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/platform/built-in.a 2024-04-25T16:49:56.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:55 smithi203 ceph-mon[33115]: pgmap v898: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 193 KiB/s wr, 43 op/s 2024-04-25T16:49:56.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:55 smithi098 ceph-mon[29162]: pgmap v898: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 193 KiB/s wr, 43 op/s 2024-04-25T16:49:56.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:55 smithi173 ceph-mon[32556]: pgmap v898: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 193 KiB/s wr, 43 op/s 2024-04-25T16:49:56.552 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/rpm-traces.o 2024-04-25T16:49:56.682 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/net/built-in.a 2024-04-25T16:49:56.748 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/exit.o 2024-04-25T16:49:56.955 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_dynevent.o 2024-04-25T16:49:57.147 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_probe.o 2024-04-25T16:49:58.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:57 smithi203 ceph-mon[33115]: pgmap v899: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 156 KiB/s wr, 35 op/s 2024-04-25T16:49:58.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:57 smithi098 ceph-mon[29162]: pgmap v899: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 156 KiB/s wr, 35 op/s 2024-04-25T16:49:58.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:57 smithi173 ceph-mon[32556]: pgmap v899: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 156 KiB/s wr, 35 op/s 2024-04-25T16:49:58.417 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/irq_64.o 2024-04-25T16:49:58.823 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/dumpstack_64.o 2024-04-25T16:49:59.542 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/time.o 2024-04-25T16:50:00.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:49:59 smithi203 ceph-mon[33115]: pgmap v900: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 181 KiB/s wr, 42 op/s 2024-04-25T16:50:00.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:49:59 smithi098 ceph-mon[29162]: pgmap v900: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 181 KiB/s wr, 42 op/s 2024-04-25T16:50:00.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:49:59 smithi173 ceph-mon[32556]: pgmap v900: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 181 KiB/s wr, 42 op/s 2024-04-25T16:50:00.861 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/ioport.o 2024-04-25T16:50:01.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:00 smithi203 ceph-mon[33115]: overall HEALTH_OK 2024-04-25T16:50:01.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:00 smithi098 ceph-mon[29162]: overall HEALTH_OK 2024-04-25T16:50:01.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:00 smithi173 ceph-mon[32556]: overall HEALTH_OK 2024-04-25T16:50:02.163 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:01 smithi173 ceph-mon[32556]: pgmap v901: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 200 KiB/s wr, 49 op/s 2024-04-25T16:50:02.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:01 smithi203 ceph-mon[33115]: pgmap v901: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 200 KiB/s wr, 49 op/s 2024-04-25T16:50:02.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:01 smithi098 ceph-mon[29162]: pgmap v901: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 200 KiB/s wr, 49 op/s 2024-04-25T16:50:03.094 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/dumpstack.o 2024-04-25T16:50:03.468 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/nmi.o 2024-04-25T16:50:03.640 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:03.643+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:50:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:03.643+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:50:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:03.643+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:50:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:03.643+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:50:03.641 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:03.643+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:50:04.061 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/trace_uprobe.o 2024-04-25T16:50:04.123 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/softirq.o 2024-04-25T16:50:04.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:03 smithi203 ceph-mon[33115]: pgmap v902: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 170 KiB/s wr, 41 op/s 2024-04-25T16:50:04.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:03 smithi098 ceph-mon[29162]: pgmap v902: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 170 KiB/s wr, 41 op/s 2024-04-25T16:50:04.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:03 smithi173 ceph-mon[32556]: pgmap v902: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 170 KiB/s wr, 41 op/s 2024-04-25T16:50:04.387 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/resource.o 2024-04-25T16:50:04.776 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/ldt.o 2024-04-25T16:50:05.538 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/trace/rethook.o 2024-04-25T16:50:06.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:05 smithi098 ceph-mon[29162]: pgmap v903: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 208 KiB/s wr, 47 op/s 2024-04-25T16:50:06.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:05 smithi203 ceph-mon[33115]: pgmap v903: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 208 KiB/s wr, 47 op/s 2024-04-25T16:50:06.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:05 smithi173 ceph-mon[32556]: pgmap v903: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 208 KiB/s wr, 47 op/s 2024-04-25T16:50:07.093 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/setup.o 2024-04-25T16:50:08.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:07 smithi203 ceph-mon[33115]: pgmap v904: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 168 KiB/s wr, 41 op/s 2024-04-25T16:50:08.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:07 smithi098 ceph-mon[29162]: pgmap v904: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 168 KiB/s wr, 41 op/s 2024-04-25T16:50:08.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:07 smithi173 ceph-mon[32556]: pgmap v904: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 168 KiB/s wr, 41 op/s 2024-04-25T16:50:09.197 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/x86_init.o 2024-04-25T16:50:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:09 smithi203 ceph-mon[33115]: pgmap v905: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 175 KiB/s wr, 43 op/s 2024-04-25T16:50:10.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:09 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:50:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:09 smithi098 ceph-mon[29162]: pgmap v905: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 175 KiB/s wr, 43 op/s 2024-04-25T16:50:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:09 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:50:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:09 smithi173 ceph-mon[32556]: pgmap v905: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 175 KiB/s wr, 43 op/s 2024-04-25T16:50:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:09 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:50:10.331 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/sysctl.o 2024-04-25T16:50:10.487 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/capability.o 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:50:11.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:50:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:50:11.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:50:11.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:50:11.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:50:11.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:50:11.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:50:11.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:50:11.324 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/ptrace.o 2024-04-25T16:50:11.326 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/user.o 2024-04-25T16:50:11.763 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/signal.o 2024-04-25T16:50:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:11 smithi203 ceph-mon[33115]: pgmap v906: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 171 KiB/s wr, 42 op/s 2024-04-25T16:50:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:11 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-25T16:50:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:11 smithi203 ceph-mon[33115]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T16:50:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:11 smithi203 ceph-mon[33115]: Unable to set osd_memory_target on smithi203 to 184758784: error parsing value: Value '184758784' is below minimum 939524096 2024-04-25T16:50:12.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:11 smithi203 ceph-mon[33115]: Unable to set osd_memory_target on smithi173 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T16:50:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:11 smithi098 ceph-mon[29162]: pgmap v906: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 171 KiB/s wr, 42 op/s 2024-04-25T16:50:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:11 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-25T16:50:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:11 smithi098 ceph-mon[29162]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T16:50:12.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:11 smithi098 ceph-mon[29162]: Unable to set osd_memory_target on smithi203 to 184758784: error parsing value: Value '184758784' is below minimum 939524096 2024-04-25T16:50:12.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:11 smithi098 ceph-mon[29162]: Unable to set osd_memory_target on smithi173 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T16:50:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:11 smithi173 ceph-mon[32556]: pgmap v906: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 171 KiB/s wr, 42 op/s 2024-04-25T16:50:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:11 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi203 to 176.1M 2024-04-25T16:50:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:11 smithi173 ceph-mon[32556]: Adjusting osd_memory_target on smithi173 to 176.2M 2024-04-25T16:50:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:11 smithi173 ceph-mon[32556]: Unable to set osd_memory_target on smithi203 to 184758784: error parsing value: Value '184758784' is below minimum 939524096 2024-04-25T16:50:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:11 smithi173 ceph-mon[32556]: Unable to set osd_memory_target on smithi173 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-25T16:50:13.118 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/trace/built-in.a 2024-04-25T16:50:13.887 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/sys.o 2024-04-25T16:50:14.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:13 smithi203 ceph-mon[33115]: pgmap v907: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 135 KiB/s wr, 34 op/s 2024-04-25T16:50:14.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:13 smithi098 ceph-mon[29162]: pgmap v907: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 135 KiB/s wr, 34 op/s 2024-04-25T16:50:14.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:13 smithi173 ceph-mon[32556]: pgmap v907: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 135 KiB/s wr, 34 op/s 2024-04-25T16:50:15.379 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/umh.o 2024-04-25T16:50:15.426 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/workqueue.o 2024-04-25T16:50:15.673 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/i8259.o 2024-04-25T16:50:16.187 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:15 smithi203 ceph-mon[33115]: pgmap v908: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 179 KiB/s wr, 43 op/s 2024-04-25T16:50:16.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:15 smithi098 ceph-mon[29162]: pgmap v908: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 179 KiB/s wr, 43 op/s 2024-04-25T16:50:16.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:15 smithi173 ceph-mon[32556]: pgmap v908: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 179 KiB/s wr, 43 op/s 2024-04-25T16:50:16.782 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/pid.o 2024-04-25T16:50:18.145 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/task_work.o 2024-04-25T16:50:18.185 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:17 smithi203 ceph-mon[33115]: pgmap v909: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 138 KiB/s wr, 36 op/s 2024-04-25T16:50:18.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:17 smithi098 ceph-mon[29162]: pgmap v909: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 138 KiB/s wr, 36 op/s 2024-04-25T16:50:18.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:17 smithi173 ceph-mon[32556]: pgmap v909: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 138 KiB/s wr, 36 op/s 2024-04-25T16:50:18.656 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:18.658+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:50:18.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:18.658+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T16:50:18.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:18.658+0000 7fdecc87b640 1 -- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 msgr2=0x55e32682ed40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T16:50:18.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:18.658+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T16:50:18.657 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:50:18.658+0000 7fdecc87b640 1 --2- 172.21.15.98:0/2691354310 >> [v2:172.21.15.203:6834/2230353021,v1:172.21.15.203:6835/2230353021] conn(0x55e326434f60 0x55e32682ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T16:50:20.007 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/extable.o 2024-04-25T16:50:20.119 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/irqinit.o 2024-04-25T16:50:20.185 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:19 smithi203 ceph-mon[33115]: pgmap v910: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 141 KiB/s wr, 42 op/s 2024-04-25T16:50:20.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:19 smithi098 ceph-mon[29162]: pgmap v910: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 141 KiB/s wr, 42 op/s 2024-04-25T16:50:20.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:19 smithi173 ceph-mon[32556]: pgmap v910: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 141 KiB/s wr, 42 op/s 2024-04-25T16:50:20.695 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/params.o 2024-04-25T16:50:21.717 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/jump_label.o 2024-04-25T16:50:21.932 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/irq_work.o 2024-04-25T16:50:22.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:21 smithi203 ceph-mon[33115]: pgmap v911: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 162 KiB/s wr, 45 op/s 2024-04-25T16:50:22.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:21 smithi098 ceph-mon[29162]: pgmap v911: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 162 KiB/s wr, 45 op/s 2024-04-25T16:50:22.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:21 smithi173 ceph-mon[32556]: pgmap v911: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 162 KiB/s wr, 45 op/s 2024-04-25T16:50:22.273 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-04-25T16:50:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-25T16:50:22.311 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 76, in _scrub done = self.fs.wait_until_scrub_complete(tag=tag, sleep=30, timeout=self.scrub_timeout) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/cephfs/filesystem.py", line 1806, in wait_until_scrub_complete out_json = self.rank_tell(["scrub", "status"], rank=rank, timeout=timeout) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/cephfs/filesystem.py", line 1372, in rank_tell return self.mds_tell(*args, mds_id=f"{self.id}:{rank}", **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/cephfs/filesystem.py", line 1356, in mds_tell out = self.get_ceph_cmd_stdout("tell", f"mds.{mds_id}", *args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-04-25T16:50:23.396 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/probe_roms.o 2024-04-25T16:50:23.829 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/sys_ia32.o 2024-04-25T16:50:24.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:23 smithi203 ceph-mon[33115]: pgmap v912: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 138 KiB/s wr, 40 op/s 2024-04-25T16:50:24.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:23 smithi098 ceph-mon[29162]: pgmap v912: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 138 KiB/s wr, 40 op/s 2024-04-25T16:50:24.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:23 smithi173 ceph-mon[32556]: pgmap v912: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 138 KiB/s wr, 40 op/s 2024-04-25T16:50:24.683 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/kthread.o 2024-04-25T16:50:26.169 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/signal_32.o 2024-04-25T16:50:26.185 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:25 smithi203 ceph-mon[33115]: pgmap v913: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 191 KiB/s wr, 45 op/s 2024-04-25T16:50:26.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:25 smithi098 ceph-mon[29162]: pgmap v913: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 191 KiB/s wr, 45 op/s 2024-04-25T16:50:26.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:25 smithi173 ceph-mon[32556]: pgmap v913: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 191 KiB/s wr, 45 op/s 2024-04-25T16:50:27.050 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/sys_ni.o 2024-04-25T16:50:27.107 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/nsproxy.o 2024-04-25T16:50:27.631 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/sys_x86_64.o 2024-04-25T16:50:28.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:27 smithi203 ceph-mon[33115]: pgmap v914: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 150 KiB/s wr, 35 op/s 2024-04-25T16:50:28.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:27 smithi098 ceph-mon[29162]: pgmap v914: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 150 KiB/s wr, 35 op/s 2024-04-25T16:50:28.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:27 smithi173 ceph-mon[32556]: pgmap v914: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 150 KiB/s wr, 35 op/s 2024-04-25T16:50:28.309 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/notifier.o 2024-04-25T16:50:29.060 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/espfix_64.o 2024-04-25T16:50:29.139 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/ksysfs.o 2024-04-25T16:50:30.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:29 smithi203 ceph-mon[33115]: pgmap v915: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 175 KiB/s wr, 37 op/s 2024-04-25T16:50:30.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:29 smithi098 ceph-mon[29162]: pgmap v915: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 175 KiB/s wr, 37 op/s 2024-04-25T16:50:30.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:29 smithi173 ceph-mon[32556]: pgmap v915: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 175 KiB/s wr, 37 op/s 2024-04-25T16:50:30.652 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/ksysfs.o 2024-04-25T16:50:31.142 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/bootflag.o 2024-04-25T16:50:32.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:31 smithi203 ceph-mon[33115]: pgmap v916: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 189 KiB/s wr, 38 op/s 2024-04-25T16:50:32.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:31 smithi098 ceph-mon[29162]: pgmap v916: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 189 KiB/s wr, 38 op/s 2024-04-25T16:50:32.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:31 smithi173 ceph-mon[32556]: pgmap v916: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 189 KiB/s wr, 38 op/s 2024-04-25T16:50:32.634 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/cred.o 2024-04-25T16:50:34.184 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/reboot.o 2024-04-25T16:50:34.185 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:33 smithi203 ceph-mon[33115]: pgmap v917: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 157 KiB/s wr, 32 op/s 2024-04-25T16:50:34.245 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/async.o 2024-04-25T16:50:34.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:33 smithi098 ceph-mon[29162]: pgmap v917: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 157 KiB/s wr, 32 op/s 2024-04-25T16:50:34.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:33 smithi173 ceph-mon[32556]: pgmap v917: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 157 KiB/s wr, 32 op/s 2024-04-25T16:50:34.315 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/e820.o 2024-04-25T16:50:34.789 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/pci-dma.o 2024-04-25T16:50:35.695 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/range.o 2024-04-25T16:50:36.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:35 smithi203 ceph-mon[33115]: pgmap v918: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 197 KiB/s wr, 39 op/s 2024-04-25T16:50:36.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:35 smithi098 ceph-mon[29162]: pgmap v918: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 197 KiB/s wr, 39 op/s 2024-04-25T16:50:36.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:35 smithi173 ceph-mon[32556]: pgmap v918: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 197 KiB/s wr, 39 op/s 2024-04-25T16:50:36.274 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/smpboot.o 2024-04-25T16:50:36.550 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/quirks.o 2024-04-25T16:50:36.961 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/ucount.o 2024-04-25T16:50:38.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:37 smithi203 ceph-mon[33115]: pgmap v919: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 132 KiB/s wr, 31 op/s 2024-04-25T16:50:38.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:37 smithi098 ceph-mon[29162]: pgmap v919: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 132 KiB/s wr, 31 op/s 2024-04-25T16:50:38.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:37 smithi173 ceph-mon[32556]: pgmap v919: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 132 KiB/s wr, 31 op/s 2024-04-25T16:50:38.576 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/topology.o 2024-04-25T16:50:39.355 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/kdebugfs.o 2024-04-25T16:50:40.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:39 smithi098 ceph-mon[29162]: pgmap v920: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 147 KiB/s wr, 35 op/s 2024-04-25T16:50:40.263 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:39 smithi173 ceph-mon[32556]: pgmap v920: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 147 KiB/s wr, 35 op/s 2024-04-25T16:50:40.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:39 smithi203 ceph-mon[33115]: pgmap v920: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 147 KiB/s wr, 35 op/s 2024-04-25T16:50:40.970 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/alternative.o 2024-04-25T16:50:41.043 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/regset.o 2024-04-25T16:50:41.070 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/ksyms_common.o 2024-04-25T16:50:41.208 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/i8253.o 2024-04-25T16:50:41.451 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/groups.o 2024-04-25T16:50:42.179 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/kcmp.o 2024-04-25T16:50:42.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:41 smithi203 ceph-mon[33115]: pgmap v921: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 154 KiB/s wr, 38 op/s 2024-04-25T16:50:42.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:41 smithi098 ceph-mon[29162]: pgmap v921: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 154 KiB/s wr, 38 op/s 2024-04-25T16:50:42.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:41 smithi173 ceph-mon[32556]: pgmap v921: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 154 KiB/s wr, 38 op/s 2024-04-25T16:50:43.352 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/freezer.o 2024-04-25T16:50:44.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:43 smithi098 ceph-mon[29162]: pgmap v922: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 128 KiB/s wr, 30 op/s 2024-04-25T16:50:44.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:43 smithi173 ceph-mon[32556]: pgmap v922: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 128 KiB/s wr, 30 op/s 2024-04-25T16:50:44.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:43 smithi203 ceph-mon[33115]: pgmap v922: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 128 KiB/s wr, 30 op/s 2024-04-25T16:50:44.608 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/profile.o 2024-04-25T16:50:45.268 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/hw_breakpoint.o 2024-04-25T16:50:45.831 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/stacktrace.o 2024-04-25T16:50:46.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:45 smithi098 ceph-mon[29162]: pgmap v923: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 180 KiB/s wr, 39 op/s 2024-04-25T16:50:46.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:45 smithi173 ceph-mon[32556]: pgmap v923: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 180 KiB/s wr, 39 op/s 2024-04-25T16:50:46.429 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/tsc.o 2024-04-25T16:50:46.435 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:45 smithi203 ceph-mon[33115]: pgmap v923: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 180 KiB/s wr, 39 op/s 2024-04-25T16:50:47.710 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/dma.o 2024-04-25T16:50:48.161 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/smp.o 2024-04-25T16:50:48.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:47 smithi098 ceph-mon[29162]: pgmap v924: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 151 KiB/s wr, 33 op/s 2024-04-25T16:50:48.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:47 smithi173 ceph-mon[32556]: pgmap v924: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 151 KiB/s wr, 33 op/s 2024-04-25T16:50:48.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:47 smithi203 ceph-mon[33115]: pgmap v924: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 151 KiB/s wr, 33 op/s 2024-04-25T16:50:48.549 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/uid16.o 2024-04-25T16:50:49.980 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/tsc_msr.o 2024-04-25T16:50:50.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:49 smithi098 ceph-mon[29162]: pgmap v925: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 161 KiB/s wr, 36 op/s 2024-04-25T16:50:50.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:49 smithi173 ceph-mon[32556]: pgmap v925: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 161 KiB/s wr, 36 op/s 2024-04-25T16:50:50.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:49 smithi203 ceph-mon[33115]: pgmap v925: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 161 KiB/s wr, 36 op/s 2024-04-25T16:50:50.611 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/io_delay.o 2024-04-25T16:50:51.281 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/rtc.o 2024-04-25T16:50:51.669 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/kallsyms.o 2024-04-25T16:50:52.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:51 smithi098 ceph-mon[29162]: pgmap v926: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 170 KiB/s wr, 39 op/s 2024-04-25T16:50:52.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:51 smithi173 ceph-mon[32556]: pgmap v926: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 170 KiB/s wr, 39 op/s 2024-04-25T16:50:52.364 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/acct.o 2024-04-25T16:50:52.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:51 smithi203 ceph-mon[33115]: pgmap v926: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 170 KiB/s wr, 39 op/s 2024-04-25T16:50:52.798 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/crash_core.o 2024-04-25T16:50:53.022 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/kexec_core.o 2024-04-25T16:50:54.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:53 smithi098 ceph-mon[29162]: pgmap v927: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 136 KiB/s wr, 34 op/s 2024-04-25T16:50:54.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:53 smithi173 ceph-mon[32556]: pgmap v927: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 136 KiB/s wr, 34 op/s 2024-04-25T16:50:54.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:53 smithi203 ceph-mon[33115]: pgmap v927: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 136 KiB/s wr, 34 op/s 2024-04-25T16:50:54.766 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/kexec.o 2024-04-25T16:50:54.955 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/resource.o 2024-04-25T16:50:55.001 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/kernel/irqflags.o 2024-04-25T16:50:55.296 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/compat.o 2024-04-25T16:50:56.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:55 smithi098 ceph-mon[29162]: pgmap v928: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 176 KiB/s wr, 43 op/s 2024-04-25T16:50:56.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:55 smithi173 ceph-mon[32556]: pgmap v928: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 176 KiB/s wr, 43 op/s 2024-04-25T16:50:56.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:55 smithi203 ceph-mon[33115]: pgmap v928: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 176 KiB/s wr, 43 op/s 2024-04-25T16:50:57.222 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/utsname.o 2024-04-25T16:50:57.267 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/static_call.o 2024-04-25T16:50:58.005 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/process.o 2024-04-25T16:50:58.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:57 smithi098 ceph-mon[29162]: pgmap v929: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 134 KiB/s wr, 35 op/s 2024-04-25T16:50:58.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:57 smithi173 ceph-mon[32556]: pgmap v929: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 134 KiB/s wr, 35 op/s 2024-04-25T16:50:58.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:57 smithi203 ceph-mon[33115]: pgmap v929: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 134 KiB/s wr, 35 op/s 2024-04-25T16:50:59.110 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/ptrace.o 2024-04-25T16:50:59.624 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/tls.o 2024-04-25T16:51:00.238 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:50:59 smithi203 ceph-mon[33115]: pgmap v930: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 144 KiB/s wr, 35 op/s 2024-04-25T16:51:00.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:50:59 smithi098 ceph-mon[29162]: pgmap v930: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 144 KiB/s wr, 35 op/s 2024-04-25T16:51:00.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:50:59 smithi173 ceph-mon[32556]: pgmap v930: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 144 KiB/s wr, 35 op/s 2024-04-25T16:51:00.302 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/pid_namespace.o 2024-04-25T16:51:01.415 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/step.o 2024-04-25T16:51:01.506 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/i8237.o 2024-04-25T16:51:01.851 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/stop_machine.o 2024-04-25T16:51:02.121 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/stacktrace.o 2024-04-25T16:51:02.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:01 smithi098 ceph-mon[29162]: pgmap v931: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 165 KiB/s wr, 41 op/s 2024-04-25T16:51:02.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:01 smithi173 ceph-mon[32556]: pgmap v931: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 165 KiB/s wr, 41 op/s 2024-04-25T16:51:02.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:01 smithi203 ceph-mon[33115]: pgmap v931: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 165 KiB/s wr, 41 op/s 2024-04-25T16:51:04.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:03 smithi098 ceph-mon[29162]: pgmap v932: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 139 KiB/s wr, 35 op/s 2024-04-25T16:51:04.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:03 smithi173 ceph-mon[32556]: pgmap v932: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 139 KiB/s wr, 35 op/s 2024-04-25T16:51:04.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:03 smithi203 ceph-mon[33115]: pgmap v932: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 139 KiB/s wr, 35 op/s 2024-04-25T16:51:05.513 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/audit.o 2024-04-25T16:51:05.589 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/auditfilter.o 2024-04-25T16:51:05.993 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/auditsc.o 2024-04-25T16:51:06.129 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/audit_watch.o 2024-04-25T16:51:06.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:05 smithi098 ceph-mon[29162]: pgmap v933: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 183 KiB/s wr, 40 op/s 2024-04-25T16:51:06.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:05 smithi173 ceph-mon[32556]: pgmap v933: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 183 KiB/s wr, 40 op/s 2024-04-25T16:51:06.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:05 smithi203 ceph-mon[33115]: pgmap v933: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 183 KiB/s wr, 40 op/s 2024-04-25T16:51:06.655 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/reboot.o 2024-04-25T16:51:06.984 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/audit_fsnotify.o 2024-04-25T16:51:07.526 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/audit_tree.o 2024-04-25T16:51:07.563 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/kprobes.o 2024-04-25T16:51:08.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:07 smithi098 ceph-mon[29162]: pgmap v934: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 138 KiB/s wr, 32 op/s 2024-04-25T16:51:08.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:07 smithi173 ceph-mon[32556]: pgmap v934: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 138 KiB/s wr, 32 op/s 2024-04-25T16:51:08.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:07 smithi203 ceph-mon[33115]: pgmap v934: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 138 KiB/s wr, 32 op/s 2024-04-25T16:51:10.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:09 smithi098 ceph-mon[29162]: pgmap v935: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 148 KiB/s wr, 34 op/s 2024-04-25T16:51:10.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:09 smithi173 ceph-mon[32556]: pgmap v935: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 148 KiB/s wr, 34 op/s 2024-04-25T16:51:10.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:09 smithi203 ceph-mon[33115]: pgmap v935: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 148 KiB/s wr, 34 op/s 2024-04-25T16:51:11.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:10 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:51:11.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:10 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:51:11.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:10 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:51:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:11 smithi098 ceph-mon[29162]: pgmap v936: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 167 KiB/s wr, 39 op/s 2024-04-25T16:51:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:51:12.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:51:12.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:11 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:51:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:11 smithi173 ceph-mon[32556]: pgmap v936: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 167 KiB/s wr, 39 op/s 2024-04-25T16:51:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:51:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:51:12.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:11 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:51:12.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:11 smithi203 ceph-mon[33115]: pgmap v936: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 167 KiB/s wr, 39 op/s 2024-04-25T16:51:12.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:51:12.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:51:12.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:11 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:51:12.770 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/msr.o 2024-04-25T16:51:13.570 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/seccomp.o 2024-04-25T16:51:13.630 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/cpuid.o 2024-04-25T16:51:13.746 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/relay.o 2024-04-25T16:51:13.997 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/early-quirks.o 2024-04-25T16:51:14.197 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/smp.o 2024-04-25T16:51:14.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:13 smithi098 ceph-mon[29162]: pgmap v937: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 136 KiB/s wr, 30 op/s 2024-04-25T16:51:14.253 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/utsname_sysctl.o 2024-04-25T16:51:14.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:13 smithi173 ceph-mon[32556]: pgmap v937: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 136 KiB/s wr, 30 op/s 2024-04-25T16:51:14.278 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/smpboot.o 2024-04-25T16:51:14.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:13 smithi203 ceph-mon[33115]: pgmap v937: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 136 KiB/s wr, 30 op/s 2024-04-25T16:51:16.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:16 smithi098 ceph-mon[29162]: pgmap v938: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 183 KiB/s wr, 39 op/s 2024-04-25T16:51:16.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:16 smithi173 ceph-mon[32556]: pgmap v938: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 183 KiB/s wr, 39 op/s 2024-04-25T16:51:16.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:16 smithi203 ceph-mon[33115]: pgmap v938: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 183 KiB/s wr, 39 op/s 2024-04-25T16:51:17.594 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/tsc_sync.o 2024-04-25T16:51:18.232 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/delayacct.o 2024-04-25T16:51:18.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:18 smithi098 ceph-mon[29162]: pgmap v939: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 141 KiB/s wr, 35 op/s 2024-04-25T16:51:18.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:18 smithi173 ceph-mon[32556]: pgmap v939: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 141 KiB/s wr, 35 op/s 2024-04-25T16:51:18.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:18 smithi203 ceph-mon[33115]: pgmap v939: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 141 KiB/s wr, 35 op/s 2024-04-25T16:51:18.852 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/taskstats.o 2024-04-25T16:51:19.387 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/setup_percpu.o 2024-04-25T16:51:20.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:20 smithi173 ceph-mon[32556]: pgmap v940: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 149 KiB/s wr, 34 op/s 2024-04-25T16:51:20.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:20 smithi203 ceph-mon[33115]: pgmap v940: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 149 KiB/s wr, 34 op/s 2024-04-25T16:51:20.476 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/tsacct.o 2024-04-25T16:51:20.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:20 smithi098 ceph-mon[29162]: pgmap v940: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 149 KiB/s wr, 34 op/s 2024-04-25T16:51:21.138 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/tracepoint.o 2024-04-25T16:51:21.234 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/irq_work.o 2024-04-25T16:51:21.936 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/static_call.o 2024-04-25T16:51:21.949 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/mpparse.o 2024-04-25T16:51:22.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:22 smithi173 ceph-mon[32556]: pgmap v941: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 162 KiB/s wr, 38 op/s 2024-04-25T16:51:22.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:22 smithi203 ceph-mon[33115]: pgmap v941: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 162 KiB/s wr, 38 op/s 2024-04-25T16:51:22.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:22 smithi098 ceph-mon[29162]: pgmap v941: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 162 KiB/s wr, 38 op/s 2024-04-25T16:51:22.769 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/trace_clock.o 2024-04-25T16:51:24.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:24 smithi203 ceph-mon[33115]: pgmap v942: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 125 KiB/s wr, 31 op/s 2024-04-25T16:51:24.186 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4352cc2700 time 2024-04-25T16:51:23.889218+0000 2024-04-25T16:51:24.186 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f435d4f6ba6] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f435d4f6dd1] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55bef7f8b26b] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bef7f2a428] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bef7f2b8a9] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bef7f2d9c3] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bef7df985d] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bef7dfbe8e] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bef7dfc894] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bef7de705b] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f435d673ac6] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 12: (DispatchQueue::entry()+0xb27) [0x7f435d6710b9] 2024-04-25T16:51:24.187 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f435d75bccb] 2024-04-25T16:51:24.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 14: (Thread::entry_wrapper()+0x43) [0x7f435d4c572d] 2024-04-25T16:51:24.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 15: (Thread::_entry_func(void*)+0xd) [0x7f435d4c5749] 2024-04-25T16:51:24.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f435c0691ca] 2024-04-25T16:51:24.188 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 17: clone() 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:23.890+0000 7f4352cc2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4352cc2700 time 2024-04-25T16:51:23.889218+0000 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f435d4f6ba6] 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f435d4f6dd1] 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55bef7f8b26b] 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bef7f2a428] 2024-04-25T16:51:24.189 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bef7f2b8a9] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bef7f2d9c3] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bef7df985d] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bef7dfbe8e] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bef7dfc894] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bef7de705b] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f435d673ac6] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 12: (DispatchQueue::entry()+0xb27) [0x7f435d6710b9] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f435d75bccb] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 14: (Thread::entry_wrapper()+0x43) [0x7f435d4c572d] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 15: (Thread::_entry_func(void*)+0xd) [0x7f435d4c5749] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f435c0691ca] 2024-04-25T16:51:24.190 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 17: clone() 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: *** Caught signal (Aborted) ** 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: in thread 7f4352cc2700 thread_name:ms_dispatch 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f435c073d20] 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2: gsignal() 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 3: abort() 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f435d4f6ced] 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f435d4f6dd1] 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55bef7f8b26b] 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bef7f2a428] 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bef7f2b8a9] 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bef7f2d9c3] 2024-04-25T16:51:24.191 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bef7df985d] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bef7dfbe8e] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bef7dfc894] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bef7de705b] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f435d673ac6] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 15: (DispatchQueue::entry()+0xb27) [0x7f435d6710b9] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f435d75bccb] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 17: (Thread::entry_wrapper()+0x43) [0x7f435d4c572d] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 18: (Thread::_entry_func(void*)+0xd) [0x7f435d4c5749] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f435c0691ca] 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 20: clone() 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:23.893+0000 7f4352cc2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: in thread 7f4352cc2700 thread_name:ms_dispatch 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:51:24.192 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f435c073d20] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2: gsignal() 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 3: abort() 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f435d4f6ced] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f435d4f6dd1] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55bef7f8b26b] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bef7f2a428] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bef7f2b8a9] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bef7f2d9c3] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bef7df985d] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bef7dfbe8e] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bef7dfc894] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bef7de705b] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f435d673ac6] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 15: (DispatchQueue::entry()+0xb27) [0x7f435d6710b9] 2024-04-25T16:51:24.193 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f435d75bccb] 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 17: (Thread::entry_wrapper()+0x43) [0x7f435d4c572d] 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 18: (Thread::_entry_func(void*)+0xd) [0x7f435d4c5749] 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f435c0691ca] 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 20: clone() 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: -1> 2024-04-25T16:51:23.890+0000 7f4352cc2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4352cc2700 time 2024-04-25T16:51:23.889218+0000 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f435d4f6ba6] 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f435d4f6dd1] 2024-04-25T16:51:24.194 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55bef7f8b26b] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bef7f2a428] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bef7f2b8a9] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bef7f2d9c3] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bef7df985d] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bef7dfbe8e] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bef7dfc894] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bef7de705b] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f435d673ac6] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 12: (DispatchQueue::entry()+0xb27) [0x7f435d6710b9] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f435d75bccb] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 14: (Thread::entry_wrapper()+0x43) [0x7f435d4c572d] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 15: (Thread::_entry_func(void*)+0xd) [0x7f435d4c5749] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f435c0691ca] 2024-04-25T16:51:24.195 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 17: clone() 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 0> 2024-04-25T16:51:23.893+0000 7f4352cc2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: in thread 7f4352cc2700 thread_name:ms_dispatch 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f435c073d20] 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2: gsignal() 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 3: abort() 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f435d4f6ced] 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f435d4f6dd1] 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55bef7f8b26b] 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bef7f2a428] 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bef7f2b8a9] 2024-04-25T16:51:24.196 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bef7f2d9c3] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bef7df985d] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bef7dfbe8e] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bef7dfc894] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bef7de705b] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f435d673ac6] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 15: (DispatchQueue::entry()+0xb27) [0x7f435d6710b9] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f435d75bccb] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 17: (Thread::entry_wrapper()+0x43) [0x7f435d4c572d] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 18: (Thread::_entry_func(void*)+0xd) [0x7f435d4c5749] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f435c0691ca] 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 20: clone() 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: -1> 2024-04-25T16:51:23.890+0000 7f4352cc2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: In function 'MDRequestRef MDCache::request_get(metareqid_t)' thread 7f4352cc2700 time 2024-04-25T16:51:23.889218+0000 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-3291-g994e2f20/rpm/el8/BUILD/ceph-19.0.0-3291-g994e2f20/src/mds/MDCache.cc: 9733: FAILED ceph_assert(p != active_requests.end()) 2024-04-25T16:51:24.197 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.198 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:51:24.198 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x127) [0x7f435d4f6ba6] 2024-04-25T16:51:24.198 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f435d4f6dd1] 2024-04-25T16:51:24.198 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 3: (MDCache::request_get(metareqid_t)+0x97) [0x55bef7f8b26b] 2024-04-25T16:51:24.198 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 4: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bef7f2a428] 2024-04-25T16:51:24.198 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 5: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bef7f2b8a9] 2024-04-25T16:51:24.198 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 6: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bef7f2d9c3] 2024-04-25T16:51:24.198 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bef7df985d] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bef7dfbe8e] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bef7dfc894] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bef7de705b] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f435d673ac6] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 12: (DispatchQueue::entry()+0xb27) [0x7f435d6710b9] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f435d75bccb] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 14: (Thread::entry_wrapper()+0x43) [0x7f435d4c572d] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 15: (Thread::_entry_func(void*)+0xd) [0x7f435d4c5749] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 16: /lib64/libpthread.so.0(+0x81ca) [0x7f435c0691ca] 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 17: clone() 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 0> 2024-04-25T16:51:23.893+0000 7f4352cc2700 -1 *** Caught signal (Aborted) ** 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: in thread 7f4352cc2700 thread_name:ms_dispatch 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T16:51:24.199 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f435c073d20] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2: gsignal() 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 3: abort() 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26e) [0x7f435d4f6ced] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 5: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f435d4f6dd1] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 6: (MDCache::request_get(metareqid_t)+0x97) [0x55bef7f8b26b] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0xae) [0x55bef7f2a428] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 8: (Server::handle_peer_request(boost::intrusive_ptr const&)+0x5df) [0x55bef7f2b8a9] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 9: (Server::dispatch(boost::intrusive_ptr const&)+0xdf3) [0x55bef7f2d9c3] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 10: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x7a7) [0x55bef7df985d] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 11: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x2a6) [0x55bef7dfbe8e] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 12: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x96) [0x55bef7dfc894] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 13: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x2df) [0x55bef7de705b] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 14: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xb8) [0x7f435d673ac6] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 15: (DispatchQueue::entry()+0xb27) [0x7f435d6710b9] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f435d75bccb] 2024-04-25T16:51:24.200 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 17: (Thread::entry_wrapper()+0x43) [0x7f435d4c572d] 2024-04-25T16:51:24.201 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 18: (Thread::_entry_func(void*)+0xd) [0x7f435d4c5749] 2024-04-25T16:51:24.201 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 19: /lib64/libpthread.so.0(+0x81ca) [0x7f435c0691ca] 2024-04-25T16:51:24.201 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 20: clone() 2024-04-25T16:51:24.201 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-04-25T16:51:24.201 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[69629]: 2024-04-25T16:51:24.201 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:23 smithi203 podman[71222]: 2024-04-25 16:51:23.95444395 +0000 UTC m=+0.019605929 container died 144f2147d5cdc92ec3a28abd120fe8636ab27ad2a75275887211e227d48d6f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T16:51:24.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:24 smithi173 ceph-mon[32556]: pgmap v942: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 125 KiB/s wr, 31 op/s 2024-04-25T16:51:24.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:24 smithi098 ceph-mon[29162]: pgmap v942: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 125 KiB/s wr, 31 op/s 2024-04-25T16:51:25.913 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:25 smithi203 podman[71222]: 2024-04-25 16:51:25.657019895 +0000 UTC m=+1.722181875 container remove 144f2147d5cdc92ec3a28abd120fe8636ab27ad2a75275887211e227d48d6f96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:51:25.914 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:25 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-04-25T16:51:26.186 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:26 smithi203 ceph-mon[33115]: pgmap v943: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 181 KiB/s wr, 41 op/s 2024-04-25T16:51:26.186 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:25 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Failed with result 'exit-code'. 2024-04-25T16:51:26.186 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:25 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 5min 12.502s CPU time. 2024-04-25T16:51:26.262 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:26 smithi173 ceph-mon[32556]: pgmap v943: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 181 KiB/s wr, 41 op/s 2024-04-25T16:51:26.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:26 smithi098 ceph-mon[29162]: pgmap v943: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 181 KiB/s wr, 41 op/s 2024-04-25T16:51:28.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:28 smithi203 ceph-mon[33115]: pgmap v944: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 138 KiB/s wr, 35 op/s 2024-04-25T16:51:28.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:28 smithi098 ceph-mon[29162]: pgmap v944: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 138 KiB/s wr, 35 op/s 2024-04-25T16:51:28.514 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:28 smithi173 ceph-mon[32556]: pgmap v944: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 138 KiB/s wr, 35 op/s 2024-04-25T16:51:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:30 smithi203 ceph-mon[33115]: pgmap v945: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 133 KiB/s wr, 32 op/s 2024-04-25T16:51:30.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:30 smithi098 ceph-mon[29162]: pgmap v945: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 133 KiB/s wr, 32 op/s 2024-04-25T16:51:30.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:30 smithi173 ceph-mon[32556]: pgmap v945: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 133 KiB/s wr, 32 op/s 2024-04-25T16:51:32.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:32 smithi173 ceph-mon[32556]: pgmap v946: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 126 KiB/s wr, 33 op/s 2024-04-25T16:51:32.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:32 smithi203 ceph-mon[33115]: pgmap v946: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 126 KiB/s wr, 33 op/s 2024-04-25T16:51:32.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:32 smithi098 ceph-mon[29162]: pgmap v946: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 126 KiB/s wr, 33 op/s 2024-04-25T16:51:34.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:34 smithi203 ceph-mon[33115]: pgmap v947: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 87 KiB/s wr, 26 op/s 2024-04-25T16:51:34.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:34 smithi098 ceph-mon[29162]: pgmap v947: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 87 KiB/s wr, 26 op/s 2024-04-25T16:51:34.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:34 smithi173 ceph-mon[32556]: pgmap v947: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 87 KiB/s wr, 26 op/s 2024-04-25T16:51:36.230 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:36 smithi203 ceph-mon[33115]: pgmap v948: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 83 KiB/s wr, 26 op/s 2024-04-25T16:51:36.230 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:35 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-04-25T16:51:36.230 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:35 smithi203 systemd[1]: Stopped Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:51:36.230 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:35 smithi203 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service: Consumed 5min 12.502s CPU time. 2024-04-25T16:51:36.230 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:35 smithi203 systemd[1]: Starting Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:51:36.230 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:36 smithi203 podman[71322]: 2024-04-25 16:51:36.082725173 +0000 UTC m=+0.018562443 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:51:36.230 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:36 smithi203 podman[71322]: 2024-04-25 16:51:36.233179819 +0000 UTC m=+0.169017076 container create cddb1fa31d5bda8fcdf312279303a01b1f7e67b5217768e8afcab1776dfd2629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T16:51:36.302 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:36 smithi098 ceph-mon[29162]: pgmap v948: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 83 KiB/s wr, 26 op/s 2024-04-25T16:51:36.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:36 smithi173 ceph-mon[32556]: pgmap v948: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 83 KiB/s wr, 26 op/s 2024-04-25T16:51:36.936 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:36 smithi203 podman[71322]: 2024-04-25 16:51:36.503180664 +0000 UTC m=+0.439017921 container init cddb1fa31d5bda8fcdf312279303a01b1f7e67b5217768e8afcab1776dfd2629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T16:51:36.936 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:36 smithi203 podman[71322]: 2024-04-25 16:51:36.505924721 +0000 UTC m=+0.441761986 container start cddb1fa31d5bda8fcdf312279303a01b1f7e67b5217768e8afcab1776dfd2629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:51:36.936 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:36 smithi203 ceph-mds[71341]: -- 172.21.15.203:0/1422563711 <== mon.1 v2:172.21.15.203:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c52827fa40 con 0x55c5282c7600 2024-04-25T16:51:36.936 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:36 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[71337]: starting mds.f at 2024-04-25T16:51:36.936 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:36 smithi203 bash[71322]: cddb1fa31d5bda8fcdf312279303a01b1f7e67b5217768e8afcab1776dfd2629 2024-04-25T16:51:36.936 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:51:36 smithi203 systemd[1]: Started Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:51:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:37 smithi203 ceph-mon[33115]: daemon mds.f restarted 2024-04-25T16:51:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:37 smithi203 ceph-mon[33115]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:51:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:37 smithi203 ceph-mon[33115]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T16:51:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:37 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.203:6834/3038883759,v1:172.21.15.203:6835/3038883759] up:boot 2024-04-25T16:51:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:37 smithi203 ceph-mon[33115]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-25T16:51:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:37 smithi203 ceph-mon[33115]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,3=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:51:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:37 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:51:37.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:37 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:51:37.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:37 smithi098 ceph-mon[29162]: daemon mds.f restarted 2024-04-25T16:51:37.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:37 smithi098 ceph-mon[29162]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:51:37.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:37 smithi098 ceph-mon[29162]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T16:51:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:37 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.203:6834/3038883759,v1:172.21.15.203:6835/3038883759] up:boot 2024-04-25T16:51:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:37 smithi098 ceph-mon[29162]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-25T16:51:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:37 smithi098 ceph-mon[29162]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,3=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:51:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:37 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:51:37.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:37 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:51:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:37 smithi173 ceph-mon[32556]: daemon mds.f restarted 2024-04-25T16:51:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:37 smithi173 ceph-mon[32556]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:51:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:37 smithi173 ceph-mon[32556]: osdmap e106: 12 total, 12 up, 12 in 2024-04-25T16:51:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:37 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.203:6834/3038883759,v1:172.21.15.203:6835/3038883759] up:boot 2024-04-25T16:51:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:37 smithi173 ceph-mon[32556]: Standby daemon mds.g assigned to filesystem cephfs as rank 4 2024-04-25T16:51:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:37 smithi173 ceph-mon[32556]: fsmap cephfs:4/5 {0=b=up:active,1=c=up:active,2=i=up:active,3=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-04-25T16:51:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:37 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:51:37.513 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:37 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-04-25T16:51:38.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:38 smithi203 ceph-mon[33115]: pgmap v949: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-25T16:51:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:38 smithi098 ceph-mon[29162]: pgmap v949: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-25T16:51:38.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:38 smithi173 ceph-mon[32556]: pgmap v949: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 15 op/s 2024-04-25T16:51:39.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:39 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:resolve 2024-04-25T16:51:39.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:39 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:51:39.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:39 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:resolve 2024-04-25T16:51:39.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:39 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:51:39.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:39 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:resolve 2024-04-25T16:51:39.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:39 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-04-25T16:51:40.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:40 smithi203 ceph-mon[33115]: pgmap v951: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.0 KiB/s wr, 15 op/s 2024-04-25T16:51:40.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:40 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:reconnect 2024-04-25T16:51:40.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:40 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:51:40.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:40 smithi203 ceph-mon[33115]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0449994 2024-04-25T16:51:40.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:40 smithi098 ceph-mon[29162]: pgmap v951: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.0 KiB/s wr, 15 op/s 2024-04-25T16:51:40.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:40 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:reconnect 2024-04-25T16:51:40.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:40 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:51:40.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:40 smithi098 ceph-mon[29162]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0449994 2024-04-25T16:51:40.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:40 smithi173 ceph-mon[32556]: pgmap v951: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 5.0 KiB/s wr, 15 op/s 2024-04-25T16:51:40.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:40 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:reconnect 2024-04-25T16:51:40.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:40 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-04-25T16:51:40.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:40 smithi173 ceph-mon[32556]: reconnect by client.15534 192.168.0.1:0/4080322749 after 0.0449994 2024-04-25T16:51:41.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:41 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:rejoin 2024-04-25T16:51:41.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:41 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:51:41.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:41 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:rejoin 2024-04-25T16:51:41.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:41 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:51:41.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:41 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:rejoin 2024-04-25T16:51:41.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:41 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-04-25T16:51:42.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:42 smithi203 ceph-mon[33115]: pgmap v952: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-25T16:51:42.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:42 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:clientreplay 2024-04-25T16:51:42.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:42 smithi203 ceph-mon[33115]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:51:42.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:42 smithi203 ceph-mon[33115]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:51:42.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:42 smithi098 ceph-mon[29162]: pgmap v952: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-25T16:51:42.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:42 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:clientreplay 2024-04-25T16:51:42.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:42 smithi098 ceph-mon[29162]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:51:42.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:42 smithi098 ceph-mon[29162]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:51:42.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:42 smithi173 ceph-mon[32556]: pgmap v952: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-25T16:51:42.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:42 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:clientreplay 2024-04-25T16:51:42.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:42 smithi173 ceph-mon[32556]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:clientreplay 4 up:active 2024-04-25T16:51:42.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:42 smithi173 ceph-mon[32556]: daemon mds.g is now active in filesystem cephfs as rank 4 2024-04-25T16:51:43.001 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/static_call_inline.o 2024-04-25T16:51:43.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:43 smithi203 ceph-mon[33115]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:51:43.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:43 smithi203 ceph-mon[33115]: Cluster is now healthy 2024-04-25T16:51:43.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:43 smithi203 ceph-mon[33115]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:active 2024-04-25T16:51:43.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:43 smithi203 ceph-mon[33115]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:51:43.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:43 smithi203 ceph-mon[33115]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:51:43.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:43 smithi098 ceph-mon[29162]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:51:43.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:43 smithi098 ceph-mon[29162]: Cluster is now healthy 2024-04-25T16:51:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:43 smithi098 ceph-mon[29162]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:active 2024-04-25T16:51:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:43 smithi098 ceph-mon[29162]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:51:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:43 smithi098 ceph-mon[29162]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:51:43.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:43 smithi173 ceph-mon[32556]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:51:43.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:43 smithi173 ceph-mon[32556]: Cluster is now healthy 2024-04-25T16:51:43.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:43 smithi173 ceph-mon[32556]: mds.? [v2:172.21.15.98:6838/879610341,v1:172.21.15.98:6839/879610341] up:active 2024-04-25T16:51:43.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:43 smithi173 ceph-mon[32556]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T16:51:43.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:43 smithi173 ceph-mon[32556]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:51:44.271 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/trace.o 2024-04-25T16:51:44.278 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/crash_dump.o 2024-04-25T16:51:44.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:44 smithi203 ceph-mon[33115]: pgmap v953: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 18 op/s 2024-04-25T16:51:44.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:44 smithi098 ceph-mon[29162]: pgmap v953: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 18 op/s 2024-04-25T16:51:44.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:44 smithi173 ceph-mon[32556]: pgmap v953: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.3 KiB/s wr, 18 op/s 2024-04-25T16:51:44.661 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/rethook.o 2024-04-25T16:51:44.894 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/crash_core_64.o 2024-04-25T16:51:45.162 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/jump_label.o 2024-04-25T16:51:45.340 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/machine_kexec_64.o 2024-04-25T16:51:46.075 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:51:45 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: 2024-04-25T16:51:45.771+0000 7f2e273c4700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T16:51:46.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:46 smithi203 ceph-mon[33115]: pgmap v954: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 120 KiB/s wr, 180 op/s 2024-04-25T16:51:46.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:46 smithi098 ceph-mon[29162]: pgmap v954: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 120 KiB/s wr, 180 op/s 2024-04-25T16:51:46.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:46 smithi173 ceph-mon[32556]: pgmap v954: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 120 KiB/s wr, 180 op/s 2024-04-25T16:51:47.848 INFO:tasks.workunit.client.0.smithi098.stdout: AS arch/x86/kernel/relocate_kernel_64.o 2024-04-25T16:51:48.162 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/crash.o 2024-04-25T16:51:48.427 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/context_tracking.o 2024-04-25T16:51:48.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:48 smithi203 ceph-mon[33115]: pgmap v955: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 131 KiB/s wr, 216 op/s 2024-04-25T16:51:48.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:48 smithi098 ceph-mon[29162]: pgmap v955: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 131 KiB/s wr, 216 op/s 2024-04-25T16:51:48.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:48 smithi173 ceph-mon[32556]: pgmap v955: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 131 KiB/s wr, 216 op/s 2024-04-25T16:51:49.399 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/crash_dump_64.o 2024-04-25T16:51:50.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:50 smithi203 ceph-mon[33115]: pgmap v956: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 127 KiB/s wr, 204 op/s 2024-04-25T16:51:50.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:50 smithi098 ceph-mon[29162]: pgmap v956: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 127 KiB/s wr, 204 op/s 2024-04-25T16:51:50.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:50 smithi173 ceph-mon[32556]: pgmap v956: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 127 KiB/s wr, 204 op/s 2024-04-25T16:51:50.654 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/module.o 2024-04-25T16:51:50.673 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/early_printk.o 2024-04-25T16:51:50.858 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/hpet.o 2024-04-25T16:51:50.930 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/iomem.o 2024-04-25T16:51:52.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:52 smithi203 ceph-mon[33115]: pgmap v957: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 166 KiB/s wr, 207 op/s 2024-04-25T16:51:52.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:52 smithi098 ceph-mon[29162]: pgmap v957: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 166 KiB/s wr, 207 op/s 2024-04-25T16:51:52.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:52 smithi173 ceph-mon[32556]: pgmap v957: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 166 KiB/s wr, 207 op/s 2024-04-25T16:51:53.134 INFO:tasks.workunit.client.0.smithi098.stdout: CC kernel/rseq.o 2024-04-25T16:51:54.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:54 smithi203 ceph-mon[33115]: pgmap v958: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 182 KiB/s wr, 210 op/s 2024-04-25T16:51:54.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:54 smithi098 ceph-mon[29162]: pgmap v958: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 182 KiB/s wr, 210 op/s 2024-04-25T16:51:54.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:54 smithi173 ceph-mon[32556]: pgmap v958: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 182 KiB/s wr, 210 op/s 2024-04-25T16:51:54.733 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/amd_nb.o 2024-04-25T16:51:54.945 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/kvm.o 2024-04-25T16:51:55.277 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/kvmclock.o 2024-04-25T16:51:55.850 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/paravirt.o 2024-04-25T16:51:56.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:56 smithi203 ceph-mon[33115]: pgmap v959: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 212 KiB/s wr, 215 op/s 2024-04-25T16:51:56.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:56 smithi098 ceph-mon[29162]: pgmap v959: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 212 KiB/s wr, 215 op/s 2024-04-25T16:51:56.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:56 smithi173 ceph-mon[32556]: pgmap v959: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 212 KiB/s wr, 215 op/s 2024-04-25T16:51:56.991 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/pvclock.o 2024-04-25T16:51:57.159 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/pcspeaker.o 2024-04-25T16:51:57.773 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/check.o 2024-04-25T16:51:58.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:51:58 smithi203 ceph-mon[33115]: pgmap v960: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 125 KiB/s wr, 81 op/s 2024-04-25T16:51:58.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:51:58 smithi098 ceph-mon[29162]: pgmap v960: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 125 KiB/s wr, 81 op/s 2024-04-25T16:51:58.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:51:58 smithi173 ceph-mon[32556]: pgmap v960: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 125 KiB/s wr, 81 op/s 2024-04-25T16:52:00.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:00 smithi203 ceph-mon[33115]: pgmap v961: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 145 KiB/s wr, 55 op/s 2024-04-25T16:52:00.478 INFO:tasks.workunit.client.0.smithi098.stdout: AR kernel/built-in.a 2024-04-25T16:52:00.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:00 smithi098 ceph-mon[29162]: pgmap v961: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 145 KiB/s wr, 55 op/s 2024-04-25T16:52:00.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:00 smithi173 ceph-mon[32556]: pgmap v961: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 145 KiB/s wr, 55 op/s 2024-04-25T16:52:01.280 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/uprobes.o 2024-04-25T16:52:01.446 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/perf_regs.o 2024-04-25T16:52:01.978 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/tracepoint.o 2024-04-25T16:52:02.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:02 smithi203 ceph-mon[33115]: pgmap v962: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 165 KiB/s wr, 45 op/s 2024-04-25T16:52:02.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:02 smithi098 ceph-mon[29162]: pgmap v962: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 165 KiB/s wr, 45 op/s 2024-04-25T16:52:02.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:02 smithi173 ceph-mon[32556]: pgmap v962: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 165 KiB/s wr, 45 op/s 2024-04-25T16:52:03.022 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/itmt.o 2024-04-25T16:52:03.271 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/umip.o 2024-04-25T16:52:03.275 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/unwind_orc.o 2024-04-25T16:52:03.538 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/callthunks.o 2024-04-25T16:52:04.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:04 smithi203 ceph-mon[33115]: pgmap v963: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 155 KiB/s wr, 39 op/s 2024-04-25T16:52:04.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:04 smithi098 ceph-mon[29162]: pgmap v963: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 155 KiB/s wr, 39 op/s 2024-04-25T16:52:04.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:04 smithi173 ceph-mon[32556]: pgmap v963: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 155 KiB/s wr, 39 op/s 2024-04-25T16:52:05.553 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/audit_64.o 2024-04-25T16:52:06.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:06 smithi203 ceph-mon[33115]: pgmap v964: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 179 KiB/s wr, 45 op/s 2024-04-25T16:52:06.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:06 smithi098 ceph-mon[29162]: pgmap v964: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 179 KiB/s wr, 45 op/s 2024-04-25T16:52:06.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:06 smithi173 ceph-mon[32556]: pgmap v964: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 179 KiB/s wr, 45 op/s 2024-04-25T16:52:06.817 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/mmconf-fam10h_64.o 2024-04-25T16:52:07.670 INFO:tasks.workunit.client.0.smithi098.stdout: CC arch/x86/kernel/vsmp_64.o 2024-04-25T16:52:08.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:08 smithi203 ceph-mon[33115]: pgmap v965: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 148 KiB/s wr, 39 op/s 2024-04-25T16:52:08.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:08 smithi098 ceph-mon[29162]: pgmap v965: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 148 KiB/s wr, 39 op/s 2024-04-25T16:52:08.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:08 smithi173 ceph-mon[32556]: pgmap v965: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 148 KiB/s wr, 39 op/s 2024-04-25T16:52:10.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:10 smithi203 ceph-mon[33115]: pgmap v966: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 168 KiB/s wr, 40 op/s 2024-04-25T16:52:10.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:10 smithi098 ceph-mon[29162]: pgmap v966: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 168 KiB/s wr, 40 op/s 2024-04-25T16:52:10.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:10 smithi173 ceph-mon[32556]: pgmap v966: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 168 KiB/s wr, 40 op/s 2024-04-25T16:52:10.943 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/kernel/built-in.a 2024-04-25T16:52:12.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:12 smithi203 ceph-mon[33115]: pgmap v967: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 164 KiB/s wr, 42 op/s 2024-04-25T16:52:12.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:12 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:52:12.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:12 smithi098 ceph-mon[29162]: pgmap v967: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 164 KiB/s wr, 42 op/s 2024-04-25T16:52:12.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:12 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:52:12.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:12 smithi173 ceph-mon[32556]: pgmap v967: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 164 KiB/s wr, 42 op/s 2024-04-25T16:52:12.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:12 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:52:12.598 INFO:tasks.workunit.client.0.smithi098.stdout: AR arch/x86/built-in.a 2024-04-25T16:52:13.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:13 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:52:13.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:13 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:52:13.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:13 smithi203 ceph-mon[33115]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:52:13.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:13 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:52:13.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:13 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:52:13.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:13 smithi098 ceph-mon[29162]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:52:13.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:13 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:52:13.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:13 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:52:13.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:13 smithi173 ceph-mon[32556]: from='mgr.14150 172.21.15.98:0/1873260408' entity='mgr.x' 2024-04-25T16:52:14.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:14 smithi203 ceph-mon[33115]: pgmap v968: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 133 KiB/s wr, 36 op/s 2024-04-25T16:52:14.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:14 smithi098 ceph-mon[29162]: pgmap v968: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 133 KiB/s wr, 36 op/s 2024-04-25T16:52:14.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:14 smithi173 ceph-mon[32556]: pgmap v968: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 133 KiB/s wr, 36 op/s 2024-04-25T16:52:14.675 INFO:tasks.workunit.client.0.smithi098.stderr:make[1]: *** [/home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/Makefile:2034: .] Error 2 2024-04-25T16:52:14.676 INFO:tasks.workunit.client.0.smithi098.stderr:make: *** [Makefile:234: __sub-make] Error 2 2024-04-25T16:52:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-25T16:52:14.677 INFO:tasks.workunit:Stopping ['kernel_untar_build.sh'] on client.0... 2024-04-25T16:52:14.677 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T16:52:16.225 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test kernel_untar_build.sh) on smithi098 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2024-04-25T16:52:16.458 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=25fc6b198dc14be39cf1e4a724142f81 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test kernel_untar_build.sh) on smithi098 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2024-04-25T16:52:16.461 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T16:52:16.471 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T16:52:16.471 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: 2024-04-25T16:52:16.477 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T16:52:16.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:52:16.511 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:16 smithi203 ceph-mon[33115]: pgmap v969: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 147 KiB/s wr, 56 op/s 2024-04-25T16:52:16.511 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:16 smithi098 ceph-mon[29162]: pgmap v969: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 147 KiB/s wr, 56 op/s 2024-04-25T16:52:16.520 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:16 smithi173 ceph-mon[32556]: pgmap v969: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 147 KiB/s wr, 56 op/s 2024-04-25T16:52:17.741 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.743+0000 7f1448b00640 1 -- 172.21.15.98:0/3388351712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444104d40 msgr2=0x7f1444105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.743+0000 7f1448b00640 1 --2- 172.21.15.98:0/3388351712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444104d40 0x7f1444105150 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1438009900 tx=0x7f143802a410 comp rx=0 tx=0).stop 2024-04-25T16:52:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.744+0000 7f1448b00640 1 -- 172.21.15.98:0/3388351712 shutdown_connections 2024-04-25T16:52:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.744+0000 7f1448b00640 1 --2- 172.21.15.98:0/3388351712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1444109a40 0x7f144410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.744+0000 7f1448b00640 1 --2- 172.21.15.98:0/3388351712 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444104d40 0x7f1444105150 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.744+0000 7f1448b00640 1 --2- 172.21.15.98:0/3388351712 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1444103b00 0x7f1444104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.744+0000 7f1448b00640 1 -- 172.21.15.98:0/3388351712 >> 172.21.15.98:0/3388351712 conn(0x7f14440fed10 msgr2=0x7f14441011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:17.742 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.744+0000 7f1448b00640 1 -- 172.21.15.98:0/3388351712 shutdown_connections 2024-04-25T16:52:17.743 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.744+0000 7f1448b00640 1 -- 172.21.15.98:0/3388351712 wait complete. 2024-04-25T16:52:17.744 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1448b00640 1 Processor -- start 2024-04-25T16:52:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1448b00640 1 -- start start 2024-04-25T16:52:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1448b00640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444103b00 0x7f144419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1448b00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1444104d40 0x7f144419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1448b00640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1444109a40 0x7f14441a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1448b00640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f144410f4d0 con 0x7f1444103b00 2024-04-25T16:52:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1448b00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f144410f350 con 0x7f1444104d40 2024-04-25T16:52:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1448b00640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f144410f650 con 0x7f1444109a40 2024-04-25T16:52:17.745 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f144259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444103b00 0x7f144419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f144259c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444103b00 0x7f144419a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44644/0 (socket says 172.21.15.98:44644) 2024-04-25T16:52:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f144259c640 1 -- 172.21.15.98:0/1073310635 learned_addr learned my addr 172.21.15.98:0/1073310635 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1441d9b640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1444104d40 0x7f144419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.745+0000 7f1442d9d640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1444109a40 0x7f14441a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f144259c640 1 -- 172.21.15.98:0/1073310635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1444109a40 msgr2=0x7f14441a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f144259c640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1444109a40 0x7f14441a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f144259c640 1 -- 172.21.15.98:0/1073310635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1444104d40 msgr2=0x7f144419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:17.746 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f144259c640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1444104d40 0x7f144419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f144259c640 1 -- 172.21.15.98:0/1073310635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14441a1c80 con 0x7f1444103b00 2024-04-25T16:52:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f144259c640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444103b00 0x7f144419a190 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f143400eb50 tx=0x7f143400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f14337fe640 1 -- 172.21.15.98:0/1073310635 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1434003b70 con 0x7f1444103b00 2024-04-25T16:52:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14441a1f70 con 0x7f1444103b00 2024-04-25T16:52:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f14337fe640 1 -- 172.21.15.98:0/1073310635 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f143400ee50 con 0x7f1444103b00 2024-04-25T16:52:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f14337fe640 1 -- 172.21.15.98:0/1073310635 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1434010720 con 0x7f1444103b00 2024-04-25T16:52:17.747 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.746+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14441a23d0 con 0x7f1444103b00 2024-04-25T16:52:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.748+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1444108850 con 0x7f1444103b00 2024-04-25T16:52:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.749+0000 7f14337fe640 1 -- 172.21.15.98:0/1073310635 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f1434004a40 con 0x7f1444103b00 2024-04-25T16:52:17.748 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.750+0000 7f14337fe640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f142c0afb70 0x7f142c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.750+0000 7f1441d9b640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f142c0afb70 0x7f142c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.750+0000 7f14337fe640 1 -- 172.21.15.98:0/1073310635 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f14340dff00 con 0x7f1444103b00 2024-04-25T16:52:17.749 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.750+0000 7f1441d9b640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f142c0afb70 0x7f142c0b2080 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f144419b1e0 tx=0x7f1438000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:17.751 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.754+0000 7f14337fe640 1 -- 172.21.15.98:0/1073310635 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14340a7fc0 con 0x7f1444103b00 2024-04-25T16:52:17.933 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.936+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f144402dd20 con 0x7f1444103b00 2024-04-25T16:52:17.935 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.938+0000 7f14337fe640 1 -- 172.21.15.98:0/1073310635 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+27569 (secure 0 0 0) 0x7f14340a78c0 con 0x7f1444103b00 2024-04-25T16:52:17.936 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:52:17.936 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":106,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:51:36.553341+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6839/2689051587":"2024-04-26T16:44:26.579258+0000","172.21.15.98:6838/2689051587":"2024-04-26T16:44:26.579258+0000","172.21.15.173:6833/2439698545":"2024-04-26T16:44:03.604569+0000","172.21.15.173:6832/2439698545":"2024-04-26T16:44:03.604569+0000","172.21.15.203:6832/2590142748":"2024-04-26T16:44:03.513876+0000","172.21.15.203:6837/3458796538":"2024-04-26T16:43:39.806154+0000","172.21.15.203:6839/3267653510":"2024-04-26T16:43:37.634911+0000","172.21.15.203:6835/4038856286":"2024-04-26T16:43:37.105127+0000","172.21.15.173:6838/3538054402":"2024-04-26T16:35:43.083983+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.173:6838/2873375163":"2024-04-26T16:40:54.470086+0000","172.21.15.173:6836/3031685940":"2024-04-26T16:44:26.346994+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6832/1945729206":"2024-04-26T16:43:15.031214+0000","172.21.15.203:6834/4038856286":"2024-04-26T16:43:37.105127+0000","172.21.15.173:6838/2050448663":"2024-04-26T16:35:09.397460+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.173:6839/3538054402":"2024-04-26T16:35:43.083983+0000","172.21.15.203:6834/3629133920":"2024-04-26T16:51:36.553281+0000","172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6836/3458796538":"2024-04-26T16:43:39.806154+0000","172.21.15.203:6838/3267653510":"2024-04-26T16:43:37.634911+0000","172.21.15.203:6834/1151278804":"2024-04-26T16:40:28.525704+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6836/998442662":"2024-04-26T16:37:09.422158+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.173:6837/3031685940":"2024-04-26T16:44:26.346994+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6835/3629133920":"2024-04-26T16:51:36.553281+0000","172.21.15.173:6836/311453164":"2024-04-26T16:40:25.599313+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6839/2982027693":"2024-04-26T16:39:30.428647+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6834/890593273":"2024-04-26T16:35:58.723078+0000","172.21.15.173:6839/2050448663":"2024-04-26T16:35:09.397460+0000","172.21.15.203:6835/890593273":"2024-04-26T16:35:58.723078+0000","172.21.15.173:6838/3664329633":"2024-04-26T16:36:17.370724+0000","172.21.15.173:6839/3664329633":"2024-04-26T16:36:17.370724+0000","172.21.15.203:6833/2590142748":"2024-04-26T16:44:03.513876+0000","172.21.15.203:6837/998442662":"2024-04-26T16:37:09.422158+0000","172.21.15.203:6838/2982027693":"2024-04-26T16:39:30.428647+0000","172.21.15.203:6835/2230353021":"2024-04-26T16:35:26.239567+0000","172.21.15.173:6837/311453164":"2024-04-26T16:40:25.599313+0000","172.21.15.203:6833/1945729206":"2024-04-26T16:43:15.031214+0000","172.21.15.203:6834/2230353021":"2024-04-26T16:35:26.239567+0000","172.21.15.203:6835/1151278804":"2024-04-26T16:40:28.525704+0000","172.21.15.173:6832/3037734321":"2024-04-26T16:43:15.040785+0000","172.21.15.173:6839/2873375163":"2024-04-26T16:40:54.470086+0000","172.21.15.98:6838/3308601909":"2024-04-26T16:41:27.323250+0000","172.21.15.98:6839/3308601909":"2024-04-26T16:41:27.323250+0000","172.21.15.173:6833/3037734321":"2024-04-26T16:43:15.040785+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:52:17.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f142c0afb70 msgr2=0x7f142c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:17.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f142c0afb70 0x7f142c0b2080 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f144419b1e0 tx=0x7f1438000bc0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444103b00 msgr2=0x7f144419a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:17.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444103b00 0x7f144419a190 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f143400eb50 tx=0x7f143400c6a0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 shutdown_connections 2024-04-25T16:52:17.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f142c0afb70 0x7f142c0b2080 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1444109a40 0x7f14441a1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.938 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1444104d40 0x7f144419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 --2- 172.21.15.98:0/1073310635 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f1444103b00 0x7f144419a190 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:17.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 >> 172.21.15.98:0/1073310635 conn(0x7f14440fed10 msgr2=0x7f1444105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:17.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 shutdown_connections 2024-04-25T16:52:17.939 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:17.940+0000 7f1448b00640 1 -- 172.21.15.98:0/1073310635 wait complete. 2024-04-25T16:52:17.949 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:52:17.950 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:52:18.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 -- 172.21.15.98:0/838008858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4103c40 msgr2=0x7f01d4104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 --2- 172.21.15.98:0/838008858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4103c40 0x7f01d4104050 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f01c8009900 tx=0x7f01c802a410 comp rx=0 tx=0).stop 2024-04-25T16:52:18.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 -- 172.21.15.98:0/838008858 shutdown_connections 2024-04-25T16:52:18.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 --2- 172.21.15.98:0/838008858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d4108940 0x7f01d410ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.064 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 --2- 172.21.15.98:0/838008858 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4103c40 0x7f01d4104050 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 --2- 172.21.15.98:0/838008858 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01d4102a00 0x7f01d4103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 -- 172.21.15.98:0/838008858 >> 172.21.15.98:0/838008858 conn(0x7f01d40fdbf0 msgr2=0x7f01d41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:18.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 -- 172.21.15.98:0/838008858 shutdown_connections 2024-04-25T16:52:18.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.066+0000 7f01d9c51640 1 -- 172.21.15.98:0/838008858 wait complete. 2024-04-25T16:52:18.065 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d9c51640 1 Processor -- start 2024-04-25T16:52:18.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d9c51640 1 -- start start 2024-04-25T16:52:18.067 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d9c51640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4102a00 0x7f01d4075830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d9c51640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01d4103c40 0x7f01d419ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d9c51640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d4108940 0x7f01d41a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d9c51640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f01d410e3b0 con 0x7f01d4102a00 2024-04-25T16:52:18.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d9c51640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f01d410e230 con 0x7f01d4103c40 2024-04-25T16:52:18.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d9c51640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f01d410e530 con 0x7f01d4108940 2024-04-25T16:52:18.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4102a00 0x7f01d4075830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d37fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4102a00 0x7f01d4075830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44656/0 (socket says 172.21.15.98:44656) 2024-04-25T16:52:18.068 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d37fe640 1 -- 172.21.15.98:0/14996695 learned_addr learned my addr 172.21.15.98:0/14996695 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d2ffd640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01d4103c40 0x7f01d419ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d3fff640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d4108940 0x7f01d41a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d37fe640 1 -- 172.21.15.98:0/14996695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d4108940 msgr2=0x7f01d41a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d37fe640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d4108940 0x7f01d41a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d37fe640 1 -- 172.21.15.98:0/14996695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01d4103c40 msgr2=0x7f01d419ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d37fe640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01d4103c40 0x7f01d419ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.067+0000 7f01d37fe640 1 -- 172.21.15.98:0/14996695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d41a1ca0 con 0x7f01d4102a00 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.068+0000 7f01d37fe640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4102a00 0x7f01d4075830 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f01c400e9f0 tx=0x7f01c400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:18.069 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.068+0000 7f01d0ff9640 1 -- 172.21.15.98:0/14996695 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c400ccc0 con 0x7f01d4102a00 2024-04-25T16:52:18.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.068+0000 7f01d0ff9640 1 -- 172.21.15.98:0/14996695 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f01c40044b0 con 0x7f01d4102a00 2024-04-25T16:52:18.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.068+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d41a1f90 con 0x7f01d4102a00 2024-04-25T16:52:18.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.068+0000 7f01d0ff9640 1 -- 172.21.15.98:0/14996695 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4010430 con 0x7f01d4102a00 2024-04-25T16:52:18.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.068+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d41a23f0 con 0x7f01d4102a00 2024-04-25T16:52:18.070 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.069+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d406a190 con 0x7f01d4102a00 2024-04-25T16:52:18.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.070+0000 7f01d0ff9640 1 -- 172.21.15.98:0/14996695 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f01c4003f90 con 0x7f01d4102a00 2024-04-25T16:52:18.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.071+0000 7f01d0ff9640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f01bc0afaf0 0x7f01bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.071+0000 7f01d2ffd640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f01bc0afaf0 0x7f01bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.072+0000 7f01d0ff9640 1 -- 172.21.15.98:0/14996695 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f01c40dfa40 con 0x7f01d4102a00 2024-04-25T16:52:18.071 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.072+0000 7f01d2ffd640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f01bc0afaf0 0x7f01bc0b2000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01c80057c0 tx=0x7f01c8031400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:18.074 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.076+0000 7f01d0ff9640 1 -- 172.21.15.98:0/14996695 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01d406a190 con 0x7f01d4102a00 2024-04-25T16:52:18.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:18 smithi098 ceph-mon[29162]: pgmap v970: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 116 KiB/s wr, 53 op/s 2024-04-25T16:52:18.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:18 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1073310635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:52:18.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.260+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01d4063840 con 0x7f01d4102a00 2024-04-25T16:52:18.258 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.261+0000 7f01d0ff9640 1 -- 172.21.15.98:0/14996695 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f01c40a7b60 con 0x7f01d4102a00 2024-04-25T16:52:18.259 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:52:18.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f01bc0afaf0 msgr2=0x7f01bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.260 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f01bc0afaf0 0x7f01bc0b2000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f01c80057c0 tx=0x7f01c8031400 comp rx=0 tx=0).stop 2024-04-25T16:52:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4102a00 msgr2=0x7f01d4075830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4102a00 0x7f01d4075830 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f01c400e9f0 tx=0x7f01c400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 shutdown_connections 2024-04-25T16:52:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f01bc0afaf0 0x7f01bc0b2000 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01d4108940 0x7f01d41a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01d4103c40 0x7f01d419ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 --2- 172.21.15.98:0/14996695 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f01d4102a00 0x7f01d4075830 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.261 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 >> 172.21.15.98:0/14996695 conn(0x7f01d40fdbf0 msgr2=0x7f01d4104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:18.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 shutdown_connections 2024-04-25T16:52:18.262 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.263+0000 7f01d9c51640 1 -- 172.21.15.98:0/14996695 wait complete. 2024-04-25T16:52:18.272 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:52:18.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/2086633470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 msgr2=0x7f7a78103c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/2086633470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 0x7f7a78103c10 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009900 tx=0x7f7a6802a410 comp rx=0 tx=0).stop 2024-04-25T16:52:18.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/2086633470 shutdown_connections 2024-04-25T16:52:18.386 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/2086633470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a781042a0 0x7f7a7810a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/2086633470 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 0x7f7a78103c10 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/2086633470 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a78102d80 0x7f7a78103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/2086633470 >> 172.21.15.98:0/2086633470 conn(0x7f7a780fd1a0 msgr2=0x7f7a780ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/2086633470 shutdown_connections 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/2086633470 wait complete. 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 Processor -- start 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- start start 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a78102d80 0x7f7a78199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 0x7f7a7819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.387 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a781042a0 0x7f7a781a1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7810f330 con 0x7f7a78103760 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7810f1b0 con 0x7f7a78102d80 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.386+0000 7f7a7dd9d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f7a7810f4b0 con 0x7f7a781042a0 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 0x7f7a7819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 0x7f7a7819a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44680/0 (socket says 172.21.15.98:44680) 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 -- 172.21.15.98:0/1913578787 learned_addr learned my addr 172.21.15.98:0/1913578787 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a7cd9b640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a78102d80 0x7f7a78199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a7d59c640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a781042a0 0x7f7a781a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.388 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 -- 172.21.15.98:0/1913578787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a781042a0 msgr2=0x7f7a781a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a781042a0 0x7f7a781a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 -- 172.21.15.98:0/1913578787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a78102d80 msgr2=0x7f7a78199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a78102d80 0x7f7a78199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 -- 172.21.15.98:0/1913578787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a781a1ab0 con 0x7f7a78103760 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a7d59c640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a781042a0 0x7f7a781a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6ffff640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 0x7f7a7819a4b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009230 tx=0x7f7a680040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6dffb640 1 -- 172.21.15.98:0/1913578787 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a6803f030 con 0x7f7a78103760 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6dffb640 1 -- 172.21.15.98:0/1913578787 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7a680044c0 con 0x7f7a78103760 2024-04-25T16:52:18.389 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a6dffb640 1 -- 172.21.15.98:0/1913578787 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a68035070 con 0x7f7a78103760 2024-04-25T16:52:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a781a1d40 con 0x7f7a78103760 2024-04-25T16:52:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.387+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a781a21d0 con 0x7f7a78103760 2024-04-25T16:52:18.390 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.388+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a78107720 con 0x7f7a78103760 2024-04-25T16:52:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.390+0000 7f7a6dffb640 1 -- 172.21.15.98:0/1913578787 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f7a68042030 con 0x7f7a78103760 2024-04-25T16:52:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.390+0000 7f7a6dffb640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a600afb60 0x7f7a600b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.391+0000 7f7a7cd9b640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a600afb60 0x7f7a600b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.391+0000 7f7a6dffb640 1 -- 172.21.15.98:0/1913578787 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f7a681000a0 con 0x7f7a78103760 2024-04-25T16:52:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.391+0000 7f7a7cd9b640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a600afb60 0x7f7a600b2070 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7a74005fd0 tx=0x7f7a74005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:18.391 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.394+0000 7f7a6dffb640 1 -- 172.21.15.98:0/1913578787 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7a680c81c0 con 0x7f7a78103760 2024-04-25T16:52:18.435 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:18 smithi203 ceph-mon[33115]: pgmap v970: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 116 KiB/s wr, 53 op/s 2024-04-25T16:52:18.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:18 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1073310635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:52:18.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:18 smithi173 ceph-mon[32556]: pgmap v970: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 116 KiB/s wr, 53 op/s 2024-04-25T16:52:18.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:18 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1073310635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:52:18.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.571+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a780637a0 con 0x7f7a78103760 2024-04-25T16:52:18.569 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.572+0000 7f7a6dffb640 1 -- 172.21.15.98:0/1913578787 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f7a680c7ac0 con 0x7f7a78103760 2024-04-25T16:52:18.569 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:52:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a600afb60 msgr2=0x7f7a600b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a600afb60 0x7f7a600b2070 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7a74005fd0 tx=0x7f7a74005850 comp rx=0 tx=0).stop 2024-04-25T16:52:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 msgr2=0x7f7a7819a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 0x7f7a7819a4b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f7a68009230 tx=0x7f7a680040c0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.572 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 shutdown_connections 2024-04-25T16:52:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f7a600afb60 0x7f7a600b2070 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7a781042a0 0x7f7a781a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f7a78103760 0x7f7a7819a4b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 --2- 172.21.15.98:0/1913578787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7a78102d80 0x7f7a78199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 >> 172.21.15.98:0/1913578787 conn(0x7f7a780fd1a0 msgr2=0x7f7a781048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 shutdown_connections 2024-04-25T16:52:18.573 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.574+0000 7f7a7dd9d640 1 -- 172.21.15.98:0/1913578787 wait complete. 2024-04-25T16:52:18.583 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:52:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 -- 172.21.15.98:0/346424349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334103c20 msgr2=0x7f5334104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 --2- 172.21.15.98:0/346424349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334103c20 0x7f5334104030 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5330009900 tx=0x7f533002a410 comp rx=0 tx=0).stop 2024-04-25T16:52:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 -- 172.21.15.98:0/346424349 shutdown_connections 2024-04-25T16:52:18.697 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 --2- 172.21.15.98:0/346424349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5334108920 0x7f533410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 --2- 172.21.15.98:0/346424349 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f5334103c20 0x7f5334104030 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 --2- 172.21.15.98:0/346424349 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f53341029e0 0x7f53341036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 -- 172.21.15.98:0/346424349 >> 172.21.15.98:0/346424349 conn(0x7f53340fdbd0 msgr2=0x7f5334100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 -- 172.21.15.98:0/346424349 shutdown_connections 2024-04-25T16:52:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 -- 172.21.15.98:0/346424349 wait complete. 2024-04-25T16:52:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 Processor -- start 2024-04-25T16:52:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.697+0000 7f533c2f9640 1 -- start start 2024-04-25T16:52:18.698 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533c2f9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341029e0 0x7f533410d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533c2f9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5334103c20 0x7f533410b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533c2f9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5334108920 0x7f533410be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533c2f9640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f533410f5a0 con 0x7f53341029e0 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533c2f9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f533410f420 con 0x7f5334108920 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533c2f9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f533410f720 con 0x7f5334103c20 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341029e0 0x7f533410d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341029e0 0x7f533410d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44702/0 (socket says 172.21.15.98:44702) 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 -- 172.21.15.98:0/138132659 learned_addr learned my addr 172.21.15.98:0/138132659 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a896640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5334108920 0x7f533410be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.699 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f5339894640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5334103c20 0x7f533410b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 -- 172.21.15.98:0/138132659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5334103c20 msgr2=0x7f533410b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5334103c20 0x7f533410b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 -- 172.21.15.98:0/138132659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5334108920 msgr2=0x7f533410be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5334108920 0x7f533410be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 -- 172.21.15.98:0/138132659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533410c5c0 con 0x7f53341029e0 2024-04-25T16:52:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a896640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5334108920 0x7f533410be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:52:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f533a095640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341029e0 0x7f533410d2b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5328006fd0 tx=0x7f532800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:18.700 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f53277fe640 1 -- 172.21.15.98:0/138132659 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5328011830 con 0x7f53341029e0 2024-04-25T16:52:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.698+0000 7f53277fe640 1 -- 172.21.15.98:0/138132659 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f5328016070 con 0x7f53341029e0 2024-04-25T16:52:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.699+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53341a4670 con 0x7f53341029e0 2024-04-25T16:52:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.699+0000 7f53277fe640 1 -- 172.21.15.98:0/138132659 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5328010d30 con 0x7f53341029e0 2024-04-25T16:52:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.699+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53341a4b60 con 0x7f53341029e0 2024-04-25T16:52:18.701 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.700+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5334107730 con 0x7f53341029e0 2024-04-25T16:52:18.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.701+0000 7f53277fe640 1 -- 172.21.15.98:0/138132659 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f53280119d0 con 0x7f53341029e0 2024-04-25T16:52:18.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.702+0000 7f53277fe640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f53200afaf0 0x7f53200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:18.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.702+0000 7f5339894640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f53200afaf0 0x7f53200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:18.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.702+0000 7f53277fe640 1 -- 172.21.15.98:0/138132659 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f53280df480 con 0x7f53341029e0 2024-04-25T16:52:18.702 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.702+0000 7f5339894640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f53200afaf0 0x7f53200b2000 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5330031460 tx=0x7f5330000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:18.704 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.705+0000 7f53277fe640 1 -- 172.21.15.98:0/138132659 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f53280a75a0 con 0x7f53341029e0 2024-04-25T16:52:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.883+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f533402dd30 con 0x7f53341029e0 2024-04-25T16:52:18.881 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.883+0000 7f53277fe640 1 -- 172.21.15.98:0/138132659 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f53280a6ea0 con 0x7f53341029e0 2024-04-25T16:52:18.881 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:52:18.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.885+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f53200afaf0 msgr2=0x7f53200b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.883 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.885+0000 7f533c2f9640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f53200afaf0 0x7f53200b2000 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5330031460 tx=0x7f5330000bc0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.885+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341029e0 msgr2=0x7f533410d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:18.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.885+0000 7f533c2f9640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341029e0 0x7f533410d2b0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5328006fd0 tx=0x7f532800b8e0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.885+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 shutdown_connections 2024-04-25T16:52:18.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.885+0000 7f533c2f9640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f53200afaf0 0x7f53200b2000 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.885+0000 7f533c2f9640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5334108920 0x7f533410be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.886+0000 7f533c2f9640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5334103c20 0x7f533410b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.886+0000 7f533c2f9640 1 --2- 172.21.15.98:0/138132659 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f53341029e0 0x7f533410d2b0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:18.884 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.886+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 >> 172.21.15.98:0/138132659 conn(0x7f53340fdbd0 msgr2=0x7f5334104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:18.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.886+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 shutdown_connections 2024-04-25T16:52:18.885 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:18.886+0000 7f533c2f9640 1 -- 172.21.15.98:0/138132659 wait complete. 2024-04-25T16:52:18.895 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:52:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 -- 172.21.15.98:0/698106981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6400103c20 msgr2=0x7f6400104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 --2- 172.21.15.98:0/698106981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6400103c20 0x7f6400104030 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f63f4009900 tx=0x7f63f402a410 comp rx=0 tx=0).stop 2024-04-25T16:52:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 -- 172.21.15.98:0/698106981 shutdown_connections 2024-04-25T16:52:19.008 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 --2- 172.21.15.98:0/698106981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6400108920 0x7f640010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 --2- 172.21.15.98:0/698106981 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f6400103c20 0x7f6400104030 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 --2- 172.21.15.98:0/698106981 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64001029e0 0x7f64001036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 -- 172.21.15.98:0/698106981 >> 172.21.15.98:0/698106981 conn(0x7f64000fdbd0 msgr2=0x7f6400100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 -- 172.21.15.98:0/698106981 shutdown_connections 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.008+0000 7f64055d7640 1 -- 172.21.15.98:0/698106981 wait complete. 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f64055d7640 1 Processor -- start 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f64055d7640 1 -- start start 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f64055d7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64001029e0 0x7f640019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:19.009 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f64055d7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6400103c20 0x7f640019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f64055d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6400108920 0x7f64001a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f64055d7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f640010f4c0 con 0x7f64001029e0 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f64055d7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f640010f340 con 0x7f6400103c20 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f64055d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f640010f640 con 0x7f6400108920 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f63feffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64001029e0 0x7f640019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f63feffd640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64001029e0 0x7f640019a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44714/0 (socket says 172.21.15.98:44714) 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f63feffd640 1 -- 172.21.15.98:0/2229712887 learned_addr learned my addr 172.21.15.98:0/2229712887 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f63fe7fc640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6400103c20 0x7f640019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:19.010 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.009+0000 7f63ff7fe640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6400108920 0x7f64001a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63feffd640 1 -- 172.21.15.98:0/2229712887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6400108920 msgr2=0x7f64001a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63feffd640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6400108920 0x7f64001a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63feffd640 1 -- 172.21.15.98:0/2229712887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6400103c20 msgr2=0x7f640019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63feffd640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6400103c20 0x7f640019aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63feffd640 1 -- 172.21.15.98:0/2229712887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64001a1fa0 con 0x7f64001029e0 2024-04-25T16:52:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63fe7fc640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6400103c20 0x7f640019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:52:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63feffd640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64001029e0 0x7f640019a4d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f63f000c970 tx=0x7f63f000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:19.011 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63e7fff640 1 -- 172.21.15.98:0/2229712887 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f0004220 con 0x7f64001029e0 2024-04-25T16:52:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63e7fff640 1 -- 172.21.15.98:0/2229712887 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f63f0007e40 con 0x7f64001029e0 2024-04-25T16:52:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64001a2290 con 0x7f64001029e0 2024-04-25T16:52:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f63e7fff640 1 -- 172.21.15.98:0/2229712887 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63f000f450 con 0x7f64001029e0 2024-04-25T16:52:19.012 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.010+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64001a27e0 con 0x7f64001029e0 2024-04-25T16:52:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.011+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6400107730 con 0x7f64001029e0 2024-04-25T16:52:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.013+0000 7f63e7fff640 1 -- 172.21.15.98:0/2229712887 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f63f00079b0 con 0x7f64001029e0 2024-04-25T16:52:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.013+0000 7f63e7fff640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63e80afaf0 0x7f63e80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.013+0000 7f63fe7fc640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63e80afaf0 0x7f63e80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:19.013 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.013+0000 7f63e7fff640 1 -- 172.21.15.98:0/2229712887 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f63f00df830 con 0x7f64001029e0 2024-04-25T16:52:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.014+0000 7f63fe7fc640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63e80afaf0 0x7f63e80b2000 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f63f4006010 tx=0x7f63f4031420 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:19.014 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.017+0000 7f63e7fff640 1 -- 172.21.15.98:0/2229712887 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63f0014030 con 0x7f64001029e0 2024-04-25T16:52:19.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.197+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f640002dd50 con 0x7f64001029e0 2024-04-25T16:52:19.195 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.198+0000 7f63e7fff640 1 -- 172.21.15.98:0/2229712887 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v106) ==== 103+0+11 (secure 0 0 0) 0x7f63f00a7950 con 0x7f64001029e0 2024-04-25T16:52:19.196 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:52:19.197 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63e80afaf0 msgr2=0x7f63e80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:19.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63e80afaf0 0x7f63e80b2000 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f63f4006010 tx=0x7f63f4031420 comp rx=0 tx=0).stop 2024-04-25T16:52:19.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64001029e0 msgr2=0x7f640019a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:19.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64001029e0 0x7f640019a4d0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f63f000c970 tx=0x7f63f000ce30 comp rx=0 tx=0).stop 2024-04-25T16:52:19.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 shutdown_connections 2024-04-25T16:52:19.198 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f63e80afaf0 0x7f63e80b2000 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6400108920 0x7f64001a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6400103c20 0x7f640019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 --2- 172.21.15.98:0/2229712887 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f64001029e0 0x7f640019a4d0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 >> 172.21.15.98:0/2229712887 conn(0x7f64000fdbd0 msgr2=0x7f6400104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 shutdown_connections 2024-04-25T16:52:19.199 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.200+0000 7f64055d7640 1 -- 172.21.15.98:0/2229712887 wait complete. 2024-04-25T16:52:19.207 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:19 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/14996695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.208 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:19 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1913578787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.208 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:19 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/138132659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.209 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T16:52:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 -- 172.21.15.98:0/440016251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 msgr2=0x7ffab8103390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 --2- 172.21.15.98:0/440016251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 0x7ffab8103390 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009900 tx=0x7ffab402a410 comp rx=0 tx=0).stop 2024-04-25T16:52:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 -- 172.21.15.98:0/440016251 shutdown_connections 2024-04-25T16:52:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 --2- 172.21.15.98:0/440016251 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffab8104480 0x7ffab810abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.320 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 --2- 172.21.15.98:0/440016251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffab8103960 0x7ffab8103df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 --2- 172.21.15.98:0/440016251 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 0x7ffab8103390 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 -- 172.21.15.98:0/440016251 >> 172.21.15.98:0/440016251 conn(0x7ffab80fd360 msgr2=0x7ffab80ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 -- 172.21.15.98:0/440016251 shutdown_connections 2024-04-25T16:52:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 -- 172.21.15.98:0/440016251 wait complete. 2024-04-25T16:52:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 Processor -- start 2024-04-25T16:52:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 -- start start 2024-04-25T16:52:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 0x7ffab8198fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:19.321 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffab8103960 0x7ffab8199520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.320+0000 7ffabff0d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffab8104480 0x7ffab81a04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabff0d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7ffab810e3d0 con 0x7ffab8102f80 2024-04-25T16:52:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabff0d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7ffab810e250 con 0x7ffab8103960 2024-04-25T16:52:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabff0d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7ffab810e550 con 0x7ffab8104480 2024-04-25T16:52:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 0x7ffab8198fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 0x7ffab8198fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:44734/0 (socket says 172.21.15.98:44734) 2024-04-25T16:52:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 -- 172.21.15.98:0/3428876341 learned_addr learned my addr 172.21.15.98:0/3428876341 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:19.322 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabd4a8640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffab8103960 0x7ffab8199520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabe4aa640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffab8104480 0x7ffab81a04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 -- 172.21.15.98:0/3428876341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffab8104480 msgr2=0x7ffab81a04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffab8104480 0x7ffab81a04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 -- 172.21.15.98:0/3428876341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffab8103960 msgr2=0x7ffab8199520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:19.323 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffab8103960 0x7ffab8199520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 -- 172.21.15.98:0/3428876341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab8101830 con 0x7ffab8102f80 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabd4a8640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffab8103960 0x7ffab8199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabe4aa640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffab8104480 0x7ffab81a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabdca9640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 0x7ffab8198fe0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ffab4031460 tx=0x7ffab4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffaaeffd640 1 -- 172.21.15.98:0/3428876341 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab403f030 con 0x7ffab8102f80 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffaaeffd640 1 -- 172.21.15.98:0/3428876341 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ffab4003a80 con 0x7ffab8102f80 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffaaeffd640 1 -- 172.21.15.98:0/3428876341 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4035070 con 0x7ffab8102f80 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab8101ac0 con 0x7ffab8102f80 2024-04-25T16:52:19.324 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.321+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab8102000 con 0x7ffab8102f80 2024-04-25T16:52:19.325 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.322+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab8107920 con 0x7ffab8102f80 2024-04-25T16:52:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.324+0000 7ffaaeffd640 1 -- 172.21.15.98:0/3428876341 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7ffab4042030 con 0x7ffab8102f80 2024-04-25T16:52:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.324+0000 7ffaaeffd640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffaa40afb70 0x7ffaa40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.325+0000 7ffabd4a8640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffaa40afb70 0x7ffaa40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.325+0000 7ffaaeffd640 1 -- 172.21.15.98:0/3428876341 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7ffab4100270 con 0x7ffab8102f80 2024-04-25T16:52:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.325+0000 7ffabd4a8640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffaa40afb70 0x7ffaa40b2080 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8005fd0 tx=0x7ffaa8005f40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:19.326 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.328+0000 7ffaaeffd640 1 -- 172.21.15.98:0/3428876341 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffab40c8390 con 0x7ffab8102f80 2024-04-25T16:52:19.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:19 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/14996695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:19 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1913578787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:19 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/138132659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:19 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/14996695' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:19 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1913578787' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:19 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/138132659' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:52:19.555 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.558+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7ffab802dd20 con 0x7ffab8102f80 2024-04-25T16:52:19.564 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.567+0000 7ffaaeffd640 1 -- 172.21.15.98:0/3428876341 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+803125 (secure 0 0 0) 0x7ffab40c7c90 con 0x7ffab8102f80 2024-04-25T16:52:19.576 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:52:19.576 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 20, 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: "active_gid": 14150, 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: "active_name": "x", 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addrs": { 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T16:52:19.577 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6800", 2024-04-25T16:52:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4283487583 2024-04-25T16:52:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v1", 2024-04-25T16:52:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:6801", 2024-04-25T16:52:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 4283487583 2024-04-25T16:52:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.578 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: "active_addr": "172.21.15.98:6801/4283487583", 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: "active_change": "2024-04-25T16:23:07.830395+0000", 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: "available": true, 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: "standbys": [ 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24106, 2024-04-25T16:52:19.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "z", 2024-04-25T16:52:19.580 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-04-25T16:52:19.580 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-04-25T16:52:19.580 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-04-25T16:52:19.580 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.580 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.580 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.580 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:19.581 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:19.581 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:19.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.582 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.582 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T16:52:19.582 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.582 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.583 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.585 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.585 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.586 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.587 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.589 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.590 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.590 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.590 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-04-25T16:52:19.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-04-25T16:52:19.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.591 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.592 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.594 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-04-25T16:52:19.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-04-25T16:52:19.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.594 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-04-25T16:52:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.595 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.597 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-04-25T16:52:19.598 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.598 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.598 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-04-25T16:52:19.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.599 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-04-25T16:52:19.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.600 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.600 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-04-25T16:52:19.600 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-04-25T16:52:19.600 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.600 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.601 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.601 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.601 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.601 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T16:52:19.602 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.602 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-04-25T16:52:19.602 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-04-25T16:52:19.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.603 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.603 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.603 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.603 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-04-25T16:52:19.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.606 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-04-25T16:52:19.606 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.606 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.606 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.606 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-04-25T16:52:19.606 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-04-25T16:52:19.606 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.608 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-04-25T16:52:19.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-04-25T16:52:19.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.608 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-04-25T16:52:19.609 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T16:52:19.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:52:19.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.610 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-04-25T16:52:19.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-04-25T16:52:19.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:19.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.611 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.611 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T16:52:19.611 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T16:52:19.611 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.611 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T16:52:19.611 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:52:19.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.612 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.612 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.612 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-04-25T16:52:19.612 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-04-25T16:52:19.612 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:19.612 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.612 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-04-25T16:52:19.613 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:19.613 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-04-25T16:52:19.613 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.613 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T16:52:19.613 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.614 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-04-25T16:52:19.614 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-04-25T16:52:19.614 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.614 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.614 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T16:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T16:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T16:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.615 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-04-25T16:52:19.616 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-04-25T16:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-04-25T16:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T16:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-04-25T16:52:19.617 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-04-25T16:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-04-25T16:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.618 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T16:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-04-25T16:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-04-25T16:52:19.619 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:19.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T16:52:19.621 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T16:52:19.621 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T16:52:19.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:52:19.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.621 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.622 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.622 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.622 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.622 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.622 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.623 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.623 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.623 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.623 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.623 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.623 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.623 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.624 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.624 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.624 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.624 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.624 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.624 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.624 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.625 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.625 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.625 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.625 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.625 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.626 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.626 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.626 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.626 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.626 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.626 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.626 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.627 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.627 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.627 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.628 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.629 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.629 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.629 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.629 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.630 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.630 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.630 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.630 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:19.631 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.632 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.632 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.632 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.632 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.632 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T16:52:19.633 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.633 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.633 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.633 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T16:52:19.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T16:52:19.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.633 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.634 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-04-25T16:52:19.634 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.634 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.634 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.635 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.636 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.636 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-04-25T16:52:19.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-04-25T16:52:19.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.636 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.636 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.636 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.637 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T16:52:19.637 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.637 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.637 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.638 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T16:52:19.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.639 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.639 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.639 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.639 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.640 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.640 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.640 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.640 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.640 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.640 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.641 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-04-25T16:52:19.641 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-04-25T16:52:19.641 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.641 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.641 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T16:52:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T16:52:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.642 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.643 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.644 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.644 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T16:52:19.644 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.644 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.644 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.644 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.645 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-04-25T16:52:19.645 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.645 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.645 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.645 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-04-25T16:52:19.646 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-04-25T16:52:19.646 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:19.646 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.646 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.646 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T16:52:19.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.646 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.646 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.647 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T16:52:19.647 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.647 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.647 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.647 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-04-25T16:52:19.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-04-25T16:52:19.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.648 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.648 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.648 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-04-25T16:52:19.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.648 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.650 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-04-25T16:52:19.650 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.650 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.650 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T16:52:19.650 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.652 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T16:52:19.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T16:52:19.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.652 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-04-25T16:52:19.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-04-25T16:52:19.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:19.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.654 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T16:52:19.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.658 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-04-25T16:52:19.658 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T16:52:19.658 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:19.658 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.658 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.659 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-04-25T16:52:19.659 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.659 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.659 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.659 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T16:52:19.659 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.659 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.659 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.660 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-04-25T16:52:19.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-04-25T16:52:19.660 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.660 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.660 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.661 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-04-25T16:52:19.661 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-04-25T16:52:19.661 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-04-25T16:52:19.661 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.662 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T16:52:19.662 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.662 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.662 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.662 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.662 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-04-25T16:52:19.662 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-04-25T16:52:19.662 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.663 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.663 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.663 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.663 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.663 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.663 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.663 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T16:52:19.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.664 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.665 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.665 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.665 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T16:52:19.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.666 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.666 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.666 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-04-25T16:52:19.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-04-25T16:52:19.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.666 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.667 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.667 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.667 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.667 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.667 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.667 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T16:52:19.667 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.668 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T16:52:19.669 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.669 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.669 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.669 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:19.669 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.669 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.670 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-04-25T16:52:19.670 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-04-25T16:52:19.670 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T16:52:19.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-04-25T16:52:19.671 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.672 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.672 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T16:52:19.672 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.672 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.672 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.672 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-04-25T16:52:19.673 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.673 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.673 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-04-25T16:52:19.673 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-04-25T16:52:19.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.674 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.674 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T16:52:19.674 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.674 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.674 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.674 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:19.674 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.674 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.675 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-04-25T16:52:19.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-04-25T16:52:19.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-04-25T16:52:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-04-25T16:52:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T16:52:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.677 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-04-25T16:52:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-04-25T16:52:19.678 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-04-25T16:52:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T16:52:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.679 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-04-25T16:52:19.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.680 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.680 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.680 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.680 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-04-25T16:52:19.680 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-04-25T16:52:19.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.680 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.681 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.682 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.682 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-04-25T16:52:19.682 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-04-25T16:52:19.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.682 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.682 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T16:52:19.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-04-25T16:52:19.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-04-25T16:52:19.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.684 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T16:52:19.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.684 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.684 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.684 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-04-25T16:52:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-04-25T16:52:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-04-25T16:52:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.685 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.686 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T16:52:19.686 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.686 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.686 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.686 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:19.686 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.687 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.687 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.687 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-04-25T16:52:19.687 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-04-25T16:52:19.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.689 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-04-25T16:52:19.689 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-04-25T16:52:19.689 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.689 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T16:52:19.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.690 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.690 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:19.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.691 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.691 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-04-25T16:52:19.691 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-04-25T16:52:19.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.692 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.692 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.692 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T16:52:19.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.693 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.693 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-04-25T16:52:19.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.694 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.694 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.694 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-04-25T16:52:19.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-04-25T16:52:19.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.695 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.695 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.695 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T16:52:19.695 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.696 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-04-25T16:52:19.696 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.697 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.697 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-04-25T16:52:19.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-04-25T16:52:19.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.698 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.698 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T16:52:19.698 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.699 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.699 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.699 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-04-25T16:52:19.699 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.700 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.700 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.700 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-04-25T16:52:19.700 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-04-25T16:52:19.700 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.701 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.701 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.701 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.701 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.702 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.702 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T16:52:19.702 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.702 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.703 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-04-25T16:52:19.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-04-25T16:52:19.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.703 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.704 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.704 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:19.704 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.704 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.704 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.704 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T16:52:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-04-25T16:52:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T16:52:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.706 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-04-25T16:52:19.706 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.706 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.706 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.706 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T16:52:19.706 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.706 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-04-25T16:52:19.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.708 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.708 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.708 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T16:52:19.708 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.708 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.708 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.708 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.709 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-04-25T16:52:19.710 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.710 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.710 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.710 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.710 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-04-25T16:52:19.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-04-25T16:52:19.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.710 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.711 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.711 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.711 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.711 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.711 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.711 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T16:52:19.711 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.712 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.712 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.712 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-04-25T16:52:19.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-04-25T16:52:19.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.712 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.712 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.712 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:19.713 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.713 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.713 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-04-25T16:52:19.713 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.713 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.714 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-04-25T16:52:19.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-04-25T16:52:19.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.714 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.714 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.714 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T16:52:19.714 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.714 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.715 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T16:52:19.715 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.715 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.715 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.715 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-04-25T16:52:19.715 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-04-25T16:52:19.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.716 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.716 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:19.716 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.716 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.716 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.716 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.717 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.718 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.718 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.718 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.718 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.718 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T16:52:19.718 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.718 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.718 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.719 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-04-25T16:52:19.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-04-25T16:52:19.719 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.719 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.719 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.719 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.719 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.720 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.720 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.720 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T16:52:19.720 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.720 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.720 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.721 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.722 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.722 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.722 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.722 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.722 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.722 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.722 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.722 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.723 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.723 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.723 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.723 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-04-25T16:52:19.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-04-25T16:52:19.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.724 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.724 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.724 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.724 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.724 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.724 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T16:52:19.724 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.726 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.726 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T16:52:19.726 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.726 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.727 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.727 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.728 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.728 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.728 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.728 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.728 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.728 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.728 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.728 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.729 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.729 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.729 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.729 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.729 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.729 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.729 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.730 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.730 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.730 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.730 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.730 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.730 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.731 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.731 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.731 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.731 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.731 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T16:52:19.731 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T16:52:19.731 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.731 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.732 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.732 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.732 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.732 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.732 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.732 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T16:52:19.732 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.732 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.733 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.733 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.733 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T16:52:19.733 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T16:52:19.733 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.733 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.733 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.733 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-04-25T16:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T16:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-04-25T16:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-04-25T16:52:19.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-04-25T16:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.737 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-04-25T16:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T16:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T16:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.740 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-04-25T16:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-04-25T16:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-04-25T16:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.741 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T16:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-04-25T16:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-04-25T16:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.742 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-04-25T16:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T16:52:19.743 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-04-25T16:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-04-25T16:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.744 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T16:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T16:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.745 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.746 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T16:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.747 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T16:52:19.748 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-04-25T16:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-04-25T16:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.749 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.750 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-04-25T16:52:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-04-25T16:52:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.752 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.753 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.754 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-04-25T16:52:19.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-04-25T16:52:19.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.754 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.754 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.754 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-04-25T16:52:19.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.756 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.757 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.757 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.757 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-04-25T16:52:19.757 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-04-25T16:52:19.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.757 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.758 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.758 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.758 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.758 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.758 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T16:52:19.758 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.759 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-04-25T16:52:19.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T16:52:19.761 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.762 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.762 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-04-25T16:52:19.762 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-04-25T16:52:19.762 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.762 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.763 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.763 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.763 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.763 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.763 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.763 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T16:52:19.763 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.763 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.764 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.764 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-04-25T16:52:19.764 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-04-25T16:52:19.764 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.764 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.764 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.764 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.765 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.765 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.765 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.765 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T16:52:19.765 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.765 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.765 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.766 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-04-25T16:52:19.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-04-25T16:52:19.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.766 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.766 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.766 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.767 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.767 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.767 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T16:52:19.767 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.767 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.767 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-04-25T16:52:19.768 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.768 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.768 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.768 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.769 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.769 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.769 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.769 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.769 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.769 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.770 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.770 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.770 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.770 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.770 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.770 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.770 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.771 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.771 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.771 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.771 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.772 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.772 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.772 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.772 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.772 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.772 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.772 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.773 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.774 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.774 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.774 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.774 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.774 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.774 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.774 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.775 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.776 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.777 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.777 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.777 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-04-25T16:52:19.777 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-04-25T16:52:19.777 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.777 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.778 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-04-25T16:52:19.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.778 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.778 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.778 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T16:52:19.778 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.779 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.779 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.779 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.779 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.779 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.779 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.779 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-04-25T16:52:19.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-04-25T16:52:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-04-25T16:52:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.781 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T16:52:19.782 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.782 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.782 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.782 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.783 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.784 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.785 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.786 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.786 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.786 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.786 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T16:52:19.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T16:52:19.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.787 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.788 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.788 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-04-25T16:52:19.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T16:52:19.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.788 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.788 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T16:52:19.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T16:52:19.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.790 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.790 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.790 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.790 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.790 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.790 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.792 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.793 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.794 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.794 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.794 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T16:52:19.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T16:52:19.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.794 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.796 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.797 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.797 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.797 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.797 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.797 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T16:52:19.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T16:52:19.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.799 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.799 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.799 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T16:52:19.799 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T16:52:19.799 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.799 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.799 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.799 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.800 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.801 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.802 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.802 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.802 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T16:52:19.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T16:52:19.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.802 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.803 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.804 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.805 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.806 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:19.806 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.806 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.806 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.806 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.806 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.807 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.807 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T16:52:19.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T16:52:19.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.807 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.807 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.808 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.809 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T16:52:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T16:52:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.810 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.811 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.812 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.814 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.814 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.814 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.814 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.814 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.814 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.814 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.814 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.815 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T16:52:19.816 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "[]", 2024-04-25T16:52:19.818 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.819 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.819 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.819 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.819 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.819 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.819 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.819 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.820 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.821 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.821 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.821 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.822 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T16:52:19.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T16:52:19.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.822 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.822 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.822 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.822 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T16:52:19.823 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.824 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.824 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.824 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.824 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.824 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.824 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.824 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.825 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T16:52:19.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T16:52:19.825 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.825 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.825 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.826 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.826 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.826 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.826 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.826 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.826 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.826 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.826 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.827 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.828 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.828 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.828 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.828 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.828 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.829 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T16:52:19.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T16:52:19.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.830 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.830 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.830 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.831 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.831 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.831 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.831 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.832 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.833 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.833 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.833 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.833 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.833 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.833 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.833 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T16:52:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T16:52:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.834 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T16:52:19.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.836 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.837 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.837 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T16:52:19.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T16:52:19.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.837 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.837 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.838 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.838 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.838 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.838 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.838 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.838 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.839 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.839 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T16:52:19.839 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T16:52:19.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.840 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.840 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.840 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.840 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.840 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.840 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.840 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.841 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.841 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.841 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T16:52:19.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T16:52:19.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.841 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.842 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.842 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T16:52:19.842 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.842 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.842 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.842 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.842 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.843 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.843 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.843 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T16:52:19.843 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T16:52:19.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.843 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.844 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:19.844 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.844 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.844 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.844 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.844 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.845 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.845 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.845 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T16:52:19.846 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T16:52:19.846 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.846 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.846 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-04-25T16:52:19.846 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T16:52:19.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T16:52:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-04-25T16:52:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.848 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.849 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.849 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.849 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.849 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T16:52:19.849 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T16:52:19.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.850 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.850 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.850 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.850 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.850 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.850 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.850 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T16:52:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T16:52:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.851 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.854 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T16:52:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T16:52:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.855 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T16:52:19.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.857 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T16:52:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T16:52:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.858 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.860 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T16:52:19.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T16:52:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.862 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.863 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-04-25T16:52:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-04-25T16:52:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.866 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-04-25T16:52:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-04-25T16:52:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.868 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-04-25T16:52:19.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-04-25T16:52:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.870 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-04-25T16:52:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.871 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-04-25T16:52:19.872 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-04-25T16:52:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.873 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-04-25T16:52:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.874 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T16:52:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T16:52:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.876 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.877 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.878 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.879 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.880 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.881 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.882 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.883 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.884 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.885 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.887 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-04-25T16:52:19.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-04-25T16:52:19.889 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-04-25T16:52:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T16:52:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.890 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.891 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T16:52:19.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.893 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-04-25T16:52:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.894 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.896 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.897 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.898 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-04-25T16:52:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-04-25T16:52:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-04-25T16:52:19.899 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T16:52:19.900 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T16:52:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-04-25T16:52:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.901 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-04-25T16:52:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T16:52:19.903 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-04-25T16:52:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-04-25T16:52:19.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.905 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-04-25T16:52:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.906 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-04-25T16:52:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-04-25T16:52:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.909 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.910 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.911 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.911 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.911 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.911 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.911 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.911 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.911 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.912 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.913 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.914 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.915 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.916 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-04-25T16:52:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-04-25T16:52:19.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-04-25T16:52:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T16:52:19.918 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-04-25T16:52:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-04-25T16:52:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.919 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-04-25T16:52:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T16:52:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.920 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-04-25T16:52:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-04-25T16:52:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.921 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-04-25T16:52:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-04-25T16:52:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-04-25T16:52:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-04-25T16:52:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T16:52:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.923 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-04-25T16:52:19.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T16:52:19.925 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.926 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T16:52:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.927 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-04-25T16:52:19.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-04-25T16:52:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-04-25T16:52:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.930 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T16:52:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-04-25T16:52:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.933 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.934 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.935 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.937 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.938 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.939 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.940 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-04-25T16:52:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-04-25T16:52:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T16:52:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.941 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-04-25T16:52:19.942 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.943 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.944 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.947 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-04-25T16:52:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-04-25T16:52:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-04-25T16:52:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-04-25T16:52:19.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-04-25T16:52:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-04-25T16:52:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.949 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T16:52:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-04-25T16:52:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-04-25T16:52:19.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-04-25T16:52:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T16:52:19.951 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-04-25T16:52:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-04-25T16:52:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.952 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-04-25T16:52:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.953 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.954 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.955 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.956 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.957 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.958 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.960 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.963 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.964 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-04-25T16:52:19.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-04-25T16:52:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-04-25T16:52:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.967 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.968 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T16:52:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T16:52:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-04-25T16:52:19.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:52:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.970 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.971 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.972 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-04-25T16:52:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-04-25T16:52:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-04-25T16:52:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.974 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.975 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.977 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.978 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.979 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.980 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.981 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.982 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.983 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.984 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.985 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-04-25T16:52:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:19.987 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.988 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.989 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.991 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.992 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:19.993 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:19.994 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.995 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:19.996 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:19.997 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-04-25T16:52:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:19.998 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-04-25T16:52:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-04-25T16:52:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-04-25T16:52:19.999 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T16:52:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.000 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-04-25T16:52:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-04-25T16:52:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-04-25T16:52:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.001 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T16:52:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.002 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.003 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.004 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.005 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.006 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.007 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.007 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.007 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.007 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.007 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.008 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.008 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.008 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.008 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.008 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.008 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.008 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.008 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.009 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.009 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.009 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.009 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.009 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.011 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.012 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.012 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.012 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.012 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-04-25T16:52:20.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.014 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.014 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.014 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-04-25T16:52:20.014 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.014 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-04-25T16:52:20.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T16:52:20.015 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.015 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.015 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.015 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.016 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.017 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.017 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.017 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.017 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.017 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.017 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.018 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.019 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.019 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.019 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.019 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.019 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.019 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.019 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.020 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.021 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.021 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.021 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.021 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.021 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.021 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.021 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.022 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.022 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.022 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.022 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.022 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.022 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-04-25T16:52:20.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.024 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.024 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.024 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.024 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.024 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.024 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-04-25T16:52:20.024 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T16:52:20.025 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.025 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.025 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-04-25T16:52:20.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-04-25T16:52:20.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.025 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.026 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.027 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-04-25T16:52:20.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-04-25T16:52:20.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.027 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.027 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.027 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-04-25T16:52:20.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.028 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.028 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.028 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T16:52:20.028 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.028 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.028 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.028 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-04-25T16:52:20.030 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.030 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.030 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.030 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.031 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.031 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.031 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.031 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.031 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-04-25T16:52:20.032 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.033 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.033 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T16:52:20.033 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T16:52:20.033 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.033 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.035 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.035 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.035 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.035 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.035 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.035 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.035 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.036 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.036 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.036 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.036 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.036 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.036 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.036 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.037 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.037 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.037 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.037 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.037 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.037 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.038 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.039 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.039 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.039 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.039 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.039 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.039 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.040 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.041 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.041 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.042 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.042 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.042 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.042 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.042 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.042 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.042 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.042 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.043 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.043 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.043 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.043 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.043 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.044 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.044 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.044 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.044 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.044 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.044 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.044 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.044 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.046 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.046 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.046 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.046 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.046 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.046 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.046 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.047 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.048 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.048 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.048 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.050 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.050 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.050 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.050 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.050 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.050 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.050 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.051 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.052 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.052 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.052 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.053 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.053 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.053 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.053 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.054 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.054 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.054 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.054 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.054 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.054 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.055 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.055 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.055 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.055 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.055 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.055 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.055 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.055 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.056 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.056 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.056 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.056 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.056 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.057 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-04-25T16:52:20.057 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.057 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.057 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.057 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.058 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.059 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.059 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.059 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.059 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.059 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.059 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.059 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.059 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.060 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.060 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.060 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.060 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.060 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.061 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.061 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.061 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.061 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.061 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.062 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.062 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.062 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.063 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.063 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.063 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.063 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.063 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.064 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.064 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.064 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.064 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.064 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.064 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.064 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.064 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.065 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.065 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.065 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.065 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.065 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.066 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.066 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.066 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.066 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.066 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.066 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.066 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.066 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.067 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.067 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.067 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.067 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.067 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.069 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T16:52:20.069 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.069 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.069 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.069 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-04-25T16:52:20.069 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-04-25T16:52:20.069 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.070 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.070 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.070 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.070 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.070 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.070 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.070 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T16:52:20.070 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.071 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.071 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.071 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.071 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.071 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.071 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.071 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.072 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.073 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.073 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.073 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.073 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.074 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.074 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.074 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.074 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.074 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.075 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.075 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.075 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.075 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.075 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.076 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.076 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.076 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.076 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.076 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.076 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.077 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.077 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.077 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.077 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.077 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.077 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.077 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.077 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.078 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.078 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.078 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.078 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.078 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.078 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.078 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.080 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.080 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.080 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-04-25T16:52:20.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T16:52:20.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.080 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.080 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.080 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.081 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-04-25T16:52:20.081 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-04-25T16:52:20.081 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-04-25T16:52:20.081 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.081 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-04-25T16:52:20.082 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.082 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.082 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.082 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.083 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.083 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.083 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.083 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.083 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.083 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.084 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.084 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.084 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.084 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-04-25T16:52:20.085 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.085 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.085 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.085 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.086 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.086 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.086 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.086 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.086 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.086 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.086 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.087 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.087 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.087 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.087 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.087 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.087 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.087 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.087 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.088 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.088 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.088 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.088 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.088 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.089 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.089 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.089 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.089 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.089 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.089 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.090 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.090 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.090 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.090 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.090 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.090 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.091 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.091 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.091 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.091 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.092 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.092 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.092 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.092 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.092 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.092 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.092 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.092 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.093 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.093 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.093 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.093 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.093 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.094 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.094 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.094 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.094 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.094 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.094 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.095 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.095 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.095 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.095 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.095 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.096 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.096 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.096 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.096 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.096 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.096 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.096 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.096 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.097 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.097 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-04-25T16:52:20.097 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.097 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.097 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.098 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.098 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.098 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.098 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.098 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.098 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.099 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.099 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.099 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.099 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.099 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.099 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.100 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.100 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.100 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.100 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.101 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.101 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.102 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.102 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.103 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.103 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.103 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.104 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.104 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.104 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.104 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.104 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.104 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.104 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.104 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.105 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.106 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.106 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.106 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.106 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.107 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.107 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.107 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.107 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.108 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.108 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.108 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.108 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.109 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-04-25T16:52:20.109 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.109 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.111 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.111 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.111 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.111 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.111 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.111 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.111 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.111 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.112 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.112 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.112 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.112 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.112 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.112 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.113 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.114 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.114 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.114 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.114 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.114 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.114 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.114 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.115 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.115 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.115 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.115 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.115 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.115 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.115 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.116 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.116 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.116 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.116 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.116 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.116 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.116 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.116 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.117 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.117 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.117 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.117 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.117 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.118 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.119 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.120 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.122 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.123 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-04-25T16:52:20.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-04-25T16:52:20.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:20.123 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.124 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T16:52:20.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-04-25T16:52:20.124 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.124 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.124 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-04-25T16:52:20.124 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T16:52:20.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.125 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.126 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-04-25T16:52:20.126 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-04-25T16:52:20.126 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.126 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.126 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.127 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.127 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.127 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T16:52:20.127 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T16:52:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T16:52:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.128 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.129 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.130 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.131 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.131 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.131 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.131 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.131 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.131 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.132 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.132 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.132 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.132 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.132 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.133 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.133 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.133 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.134 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.134 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.134 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.135 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.135 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.135 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.135 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.136 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.137 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.137 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.138 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.138 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.138 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.138 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.138 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.138 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.138 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.139 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.139 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.139 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.139 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.139 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.140 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.140 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.140 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.140 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.140 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.141 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.141 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.141 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.141 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.141 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.141 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-04-25T16:52:20.142 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-04-25T16:52:20.142 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.142 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.142 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.142 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-04-25T16:52:20.143 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.143 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.143 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-04-25T16:52:20.143 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.144 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.144 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.144 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:20.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:20.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.145 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:20.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.145 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-04-25T16:52:20.146 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.146 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.146 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.146 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.147 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.148 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.148 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.148 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.148 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.148 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-04-25T16:52:20.149 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.149 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.149 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.149 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-04-25T16:52:20.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-04-25T16:52:20.149 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.149 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.149 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.150 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.150 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.150 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.151 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-04-25T16:52:20.151 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-04-25T16:52:20.151 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.151 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.152 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.152 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.152 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.152 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.152 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T16:52:20.152 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T16:52:20.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.153 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.153 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.153 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.154 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.154 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.154 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.154 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.154 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.155 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.155 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.155 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.155 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.155 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.156 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.156 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.156 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.156 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.157 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.157 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.158 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.158 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.159 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.159 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.159 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.159 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.159 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.159 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.159 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.160 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.160 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.160 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.160 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.160 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.160 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.160 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.160 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.161 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.162 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.162 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.162 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.162 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.162 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-04-25T16:52:20.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-04-25T16:52:20.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.163 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.163 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.163 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.163 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.163 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.163 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.164 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.164 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.164 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.164 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T16:52:20.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T16:52:20.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.165 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-04-25T16:52:20.165 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.165 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.165 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.165 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.166 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.166 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.166 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-04-25T16:52:20.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-04-25T16:52:20.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:20.166 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.166 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-04-25T16:52:20.167 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.167 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.167 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.167 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.167 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.167 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.167 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.167 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.168 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:20.168 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:20.168 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.168 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.168 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.168 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T16:52:20.168 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.168 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.169 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.169 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T16:52:20.169 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.169 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.169 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.169 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:20.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:20.170 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.170 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.170 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.170 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-04-25T16:52:20.170 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.170 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T16:52:20.171 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.171 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.171 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.171 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.171 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.172 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.172 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.172 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.172 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.173 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.173 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-04-25T16:52:20.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-04-25T16:52:20.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-04-25T16:52:20.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.174 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.174 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.174 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.175 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T16:52:20.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T16:52:20.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-04-25T16:52:20.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.176 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.177 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-04-25T16:52:20.177 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-04-25T16:52:20.177 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.177 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.178 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.178 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.178 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T16:52:20.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T16:52:20.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.179 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.179 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.179 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T16:52:20.179 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T16:52:20.179 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T16:52:20.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.180 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.180 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.180 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.180 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.180 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.181 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.181 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-04-25T16:52:20.181 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.181 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.182 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.182 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.182 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.183 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.183 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.183 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.183 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.185 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.185 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.185 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.185 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.186 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.187 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.187 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.187 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.188 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.188 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.188 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.188 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.188 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.189 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.190 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-04-25T16:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-04-25T16:52:20.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.192 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.192 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.192 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.193 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-04-25T16:52:20.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T16:52:20.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.193 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.194 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.194 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.194 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.194 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.195 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.195 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.195 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.195 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.196 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.196 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.196 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.196 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.196 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.197 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-04-25T16:52:20.197 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-04-25T16:52:20.197 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.197 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.197 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.198 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.198 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.198 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.198 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.198 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.198 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.199 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-04-25T16:52:20.199 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.199 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.199 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-04-25T16:52:20.199 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-04-25T16:52:20.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.200 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.200 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.200 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T16:52:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T16:52:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.202 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.203 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.204 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.205 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.206 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.207 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.209 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:20.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.215 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.218 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.219 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-04-25T16:52:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.221 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.222 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.224 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.224 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.224 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.225 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.225 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.225 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.227 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.227 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.227 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.227 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.227 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.227 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.227 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.227 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.228 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.228 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.228 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.228 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.228 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.228 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.228 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.229 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.229 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.229 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.229 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.229 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.229 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.229 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.229 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.230 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.230 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.230 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.230 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.231 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.231 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.231 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.232 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.232 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.232 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-04-25T16:52:20.233 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.233 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.233 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.233 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.233 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.234 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.234 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.234 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.234 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.235 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.235 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.235 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.235 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.236 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.236 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.236 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.236 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.236 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.238 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.238 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.238 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.238 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.238 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.238 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.240 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.240 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.240 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.241 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.242 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-04-25T16:52:20.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-04-25T16:52:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-04-25T16:52:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T16:52:20.245 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.245 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.245 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.245 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.245 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.245 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.245 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-04-25T16:52:20.246 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.246 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.246 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.246 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.247 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.248 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.249 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.249 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.249 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.249 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.249 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.250 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.250 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.250 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.250 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.251 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.252 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.252 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.252 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.253 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.253 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.254 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.254 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.254 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.254 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.254 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.255 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.255 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.255 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.255 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.255 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.256 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.256 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.256 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.256 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.256 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.257 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.257 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.257 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.257 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.258 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-04-25T16:52:20.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-04-25T16:52:20.258 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.259 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.259 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.259 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.259 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.260 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-04-25T16:52:20.260 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-04-25T16:52:20.260 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.260 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.260 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.261 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-04-25T16:52:20.261 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.261 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.261 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.262 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.262 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-04-25T16:52:20.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-04-25T16:52:20.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.264 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-04-25T16:52:20.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-04-25T16:52:20.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.264 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.265 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.266 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-04-25T16:52:20.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-04-25T16:52:20.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:20.266 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.266 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.266 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.266 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.267 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.267 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.271 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-04-25T16:52:20.271 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-04-25T16:52:20.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.272 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.272 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-04-25T16:52:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-04-25T16:52:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.274 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-04-25T16:52:20.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-04-25T16:52:20.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.275 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.276 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-04-25T16:52:20.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-04-25T16:52:20.277 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:20.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-04-25T16:52:20.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.278 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.278 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.278 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.279 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.279 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.279 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.280 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.280 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-04-25T16:52:20.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-04-25T16:52:20.280 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.280 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.280 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.281 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.281 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.281 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.281 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.281 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.282 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-04-25T16:52:20.282 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-04-25T16:52:20.282 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.282 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.282 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.283 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.283 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.283 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.283 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.284 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-04-25T16:52:20.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-04-25T16:52:20.284 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.284 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.284 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.284 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.286 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.286 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.286 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-04-25T16:52:20.287 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.287 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.287 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.287 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-04-25T16:52:20.287 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-04-25T16:52:20.287 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.288 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.288 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.288 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.288 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T16:52:20.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.289 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-04-25T16:52:20.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-04-25T16:52:20.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.290 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.290 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.291 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.291 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.291 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-04-25T16:52:20.291 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.292 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.292 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.293 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.293 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.293 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.293 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.293 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.293 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.294 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.294 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.294 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.294 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.294 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.295 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.296 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.296 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.296 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.297 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.297 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.297 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.297 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.297 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.298 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.298 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.298 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.298 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.298 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.298 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.299 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.299 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.299 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.299 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.299 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.299 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.300 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.300 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.300 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.300 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.301 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.302 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.303 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.303 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.304 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-04-25T16:52:20.305 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.306 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.306 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.306 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.307 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.307 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.307 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.308 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.308 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.308 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.308 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.308 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.309 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.309 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.309 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.309 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.310 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.310 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.310 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.310 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.310 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.311 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.311 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.311 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.311 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.311 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.312 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.312 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.312 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.312 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.312 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.313 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.313 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.313 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.313 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.313 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.313 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.314 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.314 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.314 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.314 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.315 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.315 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.316 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.316 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.316 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.316 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.317 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.318 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-04-25T16:52:20.318 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.318 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.318 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.318 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.319 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.320 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.320 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.320 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.320 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.320 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.320 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.320 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.320 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.321 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.321 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.321 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.322 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.322 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.323 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.323 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.323 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.323 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.324 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.324 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.324 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.324 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.324 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.324 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.325 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.326 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.326 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.326 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.326 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.326 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.326 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.327 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.329 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.329 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.329 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.329 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.329 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.329 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.330 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.330 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-04-25T16:52:20.330 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.330 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.331 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.331 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-04-25T16:52:20.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-04-25T16:52:20.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.331 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.331 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.331 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T16:52:20.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.332 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.333 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:20.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:20.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.334 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.334 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-04-25T16:52:20.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.335 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.335 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.335 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.335 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.335 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.336 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.337 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.337 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.337 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.337 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.337 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.337 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.338 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.338 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.338 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.338 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.339 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.339 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.339 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.339 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.340 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.340 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.340 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.341 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.341 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.341 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.341 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.341 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.342 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.342 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.342 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.342 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.343 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.343 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.343 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.343 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.343 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.343 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.344 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.344 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.344 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.344 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.344 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.344 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.344 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.344 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.345 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.345 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.345 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.345 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.345 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.345 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.346 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.346 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.346 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.346 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.346 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.346 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.348 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.348 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.348 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.348 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.348 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.348 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.349 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-04-25T16:52:20.350 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-04-25T16:52:20.350 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.350 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.350 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.350 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.350 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.351 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.351 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.351 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T16:52:20.351 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.351 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.351 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.351 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.351 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-04-25T16:52:20.352 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-04-25T16:52:20.352 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.352 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.352 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.352 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.352 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.353 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T16:52:20.353 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.353 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.353 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.353 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.353 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-04-25T16:52:20.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-04-25T16:52:20.354 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.354 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.354 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.354 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.355 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T16:52:20.355 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.355 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.355 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.355 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-04-25T16:52:20.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-04-25T16:52:20.356 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.356 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T16:52:20.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.357 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-04-25T16:52:20.357 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-04-25T16:52:20.357 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.358 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.358 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.358 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.358 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.358 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.358 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.358 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T16:52:20.359 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.359 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.359 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.359 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.359 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-04-25T16:52:20.359 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-04-25T16:52:20.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.360 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.360 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.360 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.360 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.360 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-04-25T16:52:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-04-25T16:52:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.362 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.362 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.362 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.362 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.362 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.362 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.363 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.363 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.363 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-04-25T16:52:20.363 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-04-25T16:52:20.363 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.363 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.364 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T16:52:20.364 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.364 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.364 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.364 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.364 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.364 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.365 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.365 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T16:52:20.365 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T16:52:20.365 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.365 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.365 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.366 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.366 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.367 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.367 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:20.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:20.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-04-25T16:52:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-04-25T16:52:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.369 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.369 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.369 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-04-25T16:52:20.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-04-25T16:52:20.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.370 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.370 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-04-25T16:52:20.370 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.370 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.370 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.371 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-04-25T16:52:20.372 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-04-25T16:52:20.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.373 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.374 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.374 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-04-25T16:52:20.374 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-04-25T16:52:20.374 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.375 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.375 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.375 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.375 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.375 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.376 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.376 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.376 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.376 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.376 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.376 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.377 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.377 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.377 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.377 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.378 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.378 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.378 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.378 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.378 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.379 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.379 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.379 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.379 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.379 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.380 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.380 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.380 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.380 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.380 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.380 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.380 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.381 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.381 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.381 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.381 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.381 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.381 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.381 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.382 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.382 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.382 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.382 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.382 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.382 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.382 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.383 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.383 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.383 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.383 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.383 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.383 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.383 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.384 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.384 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.384 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.384 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.384 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.384 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.384 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.384 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.385 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.385 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.385 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.385 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.385 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.385 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.385 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.386 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.386 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.386 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.386 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.387 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-04-25T16:52:20.387 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-04-25T16:52:20.387 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.388 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.388 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.388 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.388 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.388 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.388 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.388 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-04-25T16:52:20.388 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-04-25T16:52:20.389 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.389 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.389 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.389 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.389 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.389 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.389 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.390 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.390 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.390 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.390 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.390 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.391 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.391 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.391 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.391 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.392 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-04-25T16:52:20.392 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-04-25T16:52:20.392 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.392 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.392 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T16:52:20.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.393 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.393 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.393 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.393 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.393 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.394 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.394 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.394 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.394 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.394 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-04-25T16:52:20.394 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.394 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.395 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.395 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.395 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.395 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.395 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.395 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.395 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.395 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.396 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.397 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.397 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.397 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.398 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.398 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.398 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.399 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.399 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.399 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.399 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.399 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.400 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.400 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.400 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.401 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.401 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.401 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.401 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.402 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.402 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.402 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.402 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.402 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.403 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.403 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.403 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.403 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.404 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.405 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.405 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.405 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.405 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.405 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.405 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.406 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.406 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.406 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.406 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.406 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.406 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.406 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.407 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-04-25T16:52:20.407 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.407 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.407 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.407 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.407 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.407 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.409 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.409 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.409 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.409 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.409 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.409 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.410 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.410 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.410 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.410 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.411 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.412 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.413 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.413 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.413 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.413 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.414 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.414 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.414 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.414 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.414 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.415 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.415 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.415 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.415 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.415 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.415 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.416 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.416 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.416 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.417 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.418 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.418 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.419 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.419 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.419 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.419 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-04-25T16:52:20.419 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-04-25T16:52:20.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-04-25T16:52:20.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.420 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T16:52:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-04-25T16:52:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-04-25T16:52:20.421 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.422 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-04-25T16:52:20.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.423 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-04-25T16:52:20.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-04-25T16:52:20.423 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.424 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.424 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T16:52:20.424 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.425 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-04-25T16:52:20.425 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T16:52:20.425 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.425 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.425 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.426 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.426 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.426 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T16:52:20.426 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.426 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.427 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.427 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.427 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.427 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.427 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.427 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.427 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.428 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.428 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.428 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.428 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.428 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.428 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.428 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.428 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.429 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.429 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.429 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-04-25T16:52:20.429 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.429 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.429 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.429 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-04-25T16:52:20.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-04-25T16:52:20.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:20.430 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.430 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.430 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-04-25T16:52:20.430 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.430 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.430 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.431 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.431 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.431 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.431 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.431 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-04-25T16:52:20.431 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-04-25T16:52:20.431 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.433 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.433 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:20.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:20.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:20.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.434 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.434 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.434 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.434 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.435 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.435 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.435 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.435 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.435 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.435 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.435 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.435 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.436 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.436 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.436 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.436 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.436 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.436 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.436 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.436 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.437 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.437 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.437 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.437 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.438 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.438 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.439 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.439 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.439 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.439 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.439 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.440 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.440 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.440 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.440 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.440 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.440 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.440 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.441 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.441 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.441 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.441 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.442 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.442 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.442 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.442 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.443 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.443 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.443 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.443 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.443 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.444 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.444 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.444 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.444 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.444 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.445 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.445 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.445 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.445 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.445 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.445 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.446 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.446 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.446 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.446 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.446 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.446 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.447 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.447 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.447 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.447 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.447 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.448 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-04-25T16:52:20.448 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-04-25T16:52:20.448 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.448 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.448 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.449 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.449 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.450 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-04-25T16:52:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-04-25T16:52:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-04-25T16:52:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-04-25T16:52:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-04-25T16:52:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T16:52:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.455 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "gid": 24119, 2024-04-25T16:52:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "y", 2024-04-25T16:52:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr_features": 4540701547738038271, 2024-04-25T16:52:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-04-25T16:52:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-04-25T16:52:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.456 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T16:52:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.460 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.464 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.467 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.468 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-04-25T16:52:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-04-25T16:52:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-04-25T16:52:20.469 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-04-25T16:52:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-04-25T16:52:20.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-04-25T16:52:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-04-25T16:52:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-04-25T16:52:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.472 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-04-25T16:52:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-04-25T16:52:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.473 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-04-25T16:52:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.474 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-04-25T16:52:20.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-04-25T16:52:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.476 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-04-25T16:52:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-04-25T16:52:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.477 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-04-25T16:52:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.478 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-04-25T16:52:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-04-25T16:52:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-04-25T16:52:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-04-25T16:52:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.481 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-04-25T16:52:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.482 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.483 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.484 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-04-25T16:52:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.485 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-04-25T16:52:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-04-25T16:52:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T16:52:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.487 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-04-25T16:52:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-04-25T16:52:20.488 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-04-25T16:52:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.489 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T16:52:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:52:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-04-25T16:52:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-04-25T16:52:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T16:52:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T16:52:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T16:52:20.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:52:20.493 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.493 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.493 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.494 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-04-25T16:52:20.494 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-04-25T16:52:20.494 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:20.494 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.494 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.494 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-04-25T16:52:20.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:20.495 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-04-25T16:52:20.495 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.495 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T16:52:20.495 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.495 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.495 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.496 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.496 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-04-25T16:52:20.496 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-04-25T16:52:20.496 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.496 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.496 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T16:52:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T16:52:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T16:52:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.497 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-04-25T16:52:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-04-25T16:52:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.498 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-04-25T16:52:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-04-25T16:52:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-04-25T16:52:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-04-25T16:52:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T16:52:20.499 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-04-25T16:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-04-25T16:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.500 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-04-25T16:52:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T16:52:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:52:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.501 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-04-25T16:52:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-04-25T16:52:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.502 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.504 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.506 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.509 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.510 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.511 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.511 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.511 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.511 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.511 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.511 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.511 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.512 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.512 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-04-25T16:52:20.512 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-04-25T16:52:20.512 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:20.512 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.512 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.513 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.513 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.513 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.513 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.513 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T16:52:20.513 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.513 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.514 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.514 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.514 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T16:52:20.514 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T16:52:20.514 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.514 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-04-25T16:52:20.515 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.515 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.515 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.515 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-04-25T16:52:20.515 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-04-25T16:52:20.515 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-04-25T16:52:20.515 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-04-25T16:52:20.515 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-04-25T16:52:20.516 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-04-25T16:52:20.516 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.516 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.516 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.516 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.516 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-04-25T16:52:20.516 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-04-25T16:52:20.517 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.517 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.517 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.517 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.517 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.517 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.517 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T16:52:20.518 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.518 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.518 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.519 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:20.519 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:20.519 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.519 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.519 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.519 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:20.519 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.520 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.520 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T16:52:20.520 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.520 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.521 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.521 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.521 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.521 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.521 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.521 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.522 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.522 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.522 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.522 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.522 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.522 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.522 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.523 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.523 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-04-25T16:52:20.523 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-04-25T16:52:20.523 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:20.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:20.524 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.524 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T16:52:20.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T16:52:20.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.526 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-04-25T16:52:20.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-04-25T16:52:20.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:20.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.527 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.527 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T16:52:20.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.528 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.528 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.528 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.528 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-04-25T16:52:20.529 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.529 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.529 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.529 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-04-25T16:52:20.530 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-04-25T16:52:20.530 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:20.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T16:52:20.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.531 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.531 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.531 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T16:52:20.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.532 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.532 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.532 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-04-25T16:52:20.532 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-04-25T16:52:20.532 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.532 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-04-25T16:52:20.533 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.533 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.533 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.533 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T16:52:20.533 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.534 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-04-25T16:52:20.534 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-04-25T16:52:20.534 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.534 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-04-25T16:52:20.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.535 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.535 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T16:52:20.535 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-04-25T16:52:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-04-25T16:52:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.536 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T16:52:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T16:52:20.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.538 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.538 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.538 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-04-25T16:52:20.538 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-04-25T16:52:20.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.539 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:20.539 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.539 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.539 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T16:52:20.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.540 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.540 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-04-25T16:52:20.540 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T16:52:20.540 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:20.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-04-25T16:52:20.541 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.541 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.541 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.541 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T16:52:20.541 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.542 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.542 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-04-25T16:52:20.542 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-04-25T16:52:20.542 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.542 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.543 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.543 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.543 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.543 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-04-25T16:52:20.543 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-04-25T16:52:20.543 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-04-25T16:52:20.543 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.544 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T16:52:20.544 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.544 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-04-25T16:52:20.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-04-25T16:52:20.545 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.545 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.545 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.545 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.545 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.545 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T16:52:20.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.546 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-04-25T16:52:20.546 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-04-25T16:52:20.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.547 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.547 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.547 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T16:52:20.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.548 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.548 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-04-25T16:52:20.548 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-04-25T16:52:20.548 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.548 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.549 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.549 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.549 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T16:52:20.549 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.549 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.550 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.550 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-04-25T16:52:20.550 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-04-25T16:52:20.550 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.550 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.550 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.551 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T16:52:20.551 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.551 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.551 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:20.551 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.551 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.551 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.552 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-04-25T16:52:20.552 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-04-25T16:52:20.552 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.552 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.552 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T16:52:20.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.553 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.553 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-04-25T16:52:20.553 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.554 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-04-25T16:52:20.554 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-04-25T16:52:20.554 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T16:52:20.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.555 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.555 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.555 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-04-25T16:52:20.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.556 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-04-25T16:52:20.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-04-25T16:52:20.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.557 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.557 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T16:52:20.557 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.557 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:20.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.558 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.558 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-04-25T16:52:20.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-04-25T16:52:20.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.559 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.559 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.559 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T16:52:20.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.559 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-04-25T16:52:20.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.560 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.560 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-04-25T16:52:20.560 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-04-25T16:52:20.560 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T16:52:20.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.562 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.562 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-04-25T16:52:20.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.562 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.563 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-04-25T16:52:20.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-04-25T16:52:20.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.564 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T16:52:20.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-04-25T16:52:20.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.565 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.565 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.565 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.565 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-04-25T16:52:20.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-04-25T16:52:20.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.566 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T16:52:20.566 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.566 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-04-25T16:52:20.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.567 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.567 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.567 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-04-25T16:52:20.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-04-25T16:52:20.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.567 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.567 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.568 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-04-25T16:52:20.569 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-04-25T16:52:20.569 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.570 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T16:52:20.570 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.570 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.570 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.570 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-04-25T16:52:20.571 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.571 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-04-25T16:52:20.571 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-04-25T16:52:20.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T16:52:20.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:20.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.573 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-04-25T16:52:20.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-04-25T16:52:20.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.574 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T16:52:20.574 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.574 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.574 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-04-25T16:52:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-04-25T16:52:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-04-25T16:52:20.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T16:52:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:20.576 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.577 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.577 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.577 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-04-25T16:52:20.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-04-25T16:52:20.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.577 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.578 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T16:52:20.578 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.578 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-04-25T16:52:20.578 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.579 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-04-25T16:52:20.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T16:52:20.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.581 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-04-25T16:52:20.581 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.582 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.582 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-04-25T16:52:20.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-04-25T16:52:20.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.582 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.582 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T16:52:20.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.584 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-04-25T16:52:20.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-04-25T16:52:20.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.584 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.584 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.584 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:20.584 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-04-25T16:52:20.585 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T16:52:20.586 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.586 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.586 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.586 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-04-25T16:52:20.586 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.586 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.586 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.587 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T16:52:20.587 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.587 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.587 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.587 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-04-25T16:52:20.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-04-25T16:52:20.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.588 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.588 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.588 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-04-25T16:52:20.588 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T16:52:20.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-04-25T16:52:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-04-25T16:52:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.589 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.590 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-04-25T16:52:20.590 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.590 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.590 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.590 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-04-25T16:52:20.590 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.590 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.590 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.592 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.592 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T16:52:20.592 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.592 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.592 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.592 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.592 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-04-25T16:52:20.593 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-04-25T16:52:20.593 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.593 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.593 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.593 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:20.593 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.593 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-04-25T16:52:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-04-25T16:52:20.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-04-25T16:52:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T16:52:20.595 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T16:52:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.596 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-04-25T16:52:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-04-25T16:52:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:20.597 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-04-25T16:52:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.598 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-04-25T16:52:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-04-25T16:52:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.599 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T16:52:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.600 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-04-25T16:52:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-04-25T16:52:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.601 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T16:52:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.602 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.603 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.604 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.605 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.606 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-04-25T16:52:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-04-25T16:52:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.607 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T16:52:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.608 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T16:52:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.611 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T16:52:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T16:52:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T16:52:20.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-04-25T16:52:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T16:52:20.617 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-04-25T16:52:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-04-25T16:52:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.618 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-04-25T16:52:20.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-04-25T16:52:20.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.621 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T16:52:20.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.621 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.621 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-04-25T16:52:20.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-04-25T16:52:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.622 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-04-25T16:52:20.623 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.623 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.623 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.623 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T16:52:20.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T16:52:20.623 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.624 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.624 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.624 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-04-25T16:52:20.624 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.624 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.624 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.624 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-04-25T16:52:20.624 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-04-25T16:52:20.625 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.625 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T16:52:20.625 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.625 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-04-25T16:52:20.625 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-04-25T16:52:20.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.626 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.626 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.626 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-04-25T16:52:20.626 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.626 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.626 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.626 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T16:52:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-04-25T16:52:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-04-25T16:52:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.627 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T16:52:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T16:52:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.628 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.629 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.629 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.629 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-04-25T16:52:20.629 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-04-25T16:52:20.629 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.629 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.630 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.630 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.630 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.630 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.630 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T16:52:20.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.631 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.631 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.631 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.631 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-04-25T16:52:20.631 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-04-25T16:52:20.631 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.631 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.632 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.632 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.632 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.632 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.632 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.633 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T16:52:20.633 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.633 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.633 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.633 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.633 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-04-25T16:52:20.633 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-04-25T16:52:20.634 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.634 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.634 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.634 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.634 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.634 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.635 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.635 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T16:52:20.635 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.635 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.635 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.635 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.635 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-04-25T16:52:20.636 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-04-25T16:52:20.636 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.636 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.636 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.636 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.636 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T16:52:20.637 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.637 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.637 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.638 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-04-25T16:52:20.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-04-25T16:52:20.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.638 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.638 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.638 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.639 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.639 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.639 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T16:52:20.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.640 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.640 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-04-25T16:52:20.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-04-25T16:52:20.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.642 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.642 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.642 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.642 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.643 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.643 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.643 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.643 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.644 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.644 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.644 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-04-25T16:52:20.644 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-04-25T16:52:20.644 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.645 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.645 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.645 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.645 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.645 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.645 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.646 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T16:52:20.646 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.646 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.646 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.646 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.647 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T16:52:20.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.648 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.648 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.648 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-04-25T16:52:20.648 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-04-25T16:52:20.648 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.649 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.649 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.649 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.649 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T16:52:20.649 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.650 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.650 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-04-25T16:52:20.650 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-04-25T16:52:20.650 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.650 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.650 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.651 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.652 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-04-25T16:52:20.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-04-25T16:52:20.652 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.652 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.652 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.652 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.652 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-04-25T16:52:20.653 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-04-25T16:52:20.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.654 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.654 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.654 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.654 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.654 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.654 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.655 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T16:52:20.655 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.655 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.655 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.655 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.655 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.655 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.656 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.657 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.657 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.657 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.657 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.657 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.658 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.658 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.658 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.658 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.659 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.660 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.660 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.660 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.660 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.661 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.661 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.661 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.661 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.661 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.661 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.662 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.662 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.662 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.662 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.662 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.662 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.662 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.662 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.663 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.663 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.664 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.665 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-04-25T16:52:20.665 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-04-25T16:52:20.665 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.665 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-04-25T16:52:20.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.666 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.666 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T16:52:20.666 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.666 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.667 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.667 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.667 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.668 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.668 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.668 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.668 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.668 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.668 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.668 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.668 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.669 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.669 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-04-25T16:52:20.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-04-25T16:52:20.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-04-25T16:52:20.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.670 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.670 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T16:52:20.670 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.671 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.671 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-04-25T16:52:20.672 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.672 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.672 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.672 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T16:52:20.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T16:52:20.673 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.673 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.673 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.673 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.673 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.673 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.674 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.674 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.674 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.674 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.674 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.674 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.674 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T16:52:20.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T16:52:20.675 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.675 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.675 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.675 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.675 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.676 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.676 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.676 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.676 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T16:52:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T16:52:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.677 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.678 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.679 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-04-25T16:52:20.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T16:52:20.680 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.680 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.680 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.680 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.680 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.681 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.681 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.681 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.681 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.681 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.681 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T16:52:20.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T16:52:20.682 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.682 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.682 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.682 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.683 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.683 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.683 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.683 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.683 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T16:52:20.684 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T16:52:20.684 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.684 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.684 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.684 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.684 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.684 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.685 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.685 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.685 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.685 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.685 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.685 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T16:52:20.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T16:52:20.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.686 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.686 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.686 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.686 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.686 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.687 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.687 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.687 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.687 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.687 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.687 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.687 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T16:52:20.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T16:52:20.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.688 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.688 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.688 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.688 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.688 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.689 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.689 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.689 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T16:52:20.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T16:52:20.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.690 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.690 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.691 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.691 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.691 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.691 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T16:52:20.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T16:52:20.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.692 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.692 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.692 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.692 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.693 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.693 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T16:52:20.694 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T16:52:20.694 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.694 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.695 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.695 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.695 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.695 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.695 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T16:52:20.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T16:52:20.696 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.696 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.696 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.696 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.696 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.696 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.697 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.697 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.697 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.697 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T16:52:20.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T16:52:20.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.698 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.698 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.698 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.698 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.698 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.698 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.699 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.699 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.699 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.699 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.699 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T16:52:20.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T16:52:20.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.699 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.700 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.700 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.700 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.700 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.700 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.700 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.700 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.701 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:20.702 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.702 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.702 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.702 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.702 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.702 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.702 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.704 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.704 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.704 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.704 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.704 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.704 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-04-25T16:52:20.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-04-25T16:52:20.705 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.705 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.705 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.705 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.705 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.705 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.705 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.705 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.706 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.706 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.706 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.706 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.706 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T16:52:20.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T16:52:20.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.706 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.707 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.708 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T16:52:20.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T16:52:20.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.708 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.708 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.708 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.708 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.709 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.709 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.709 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.709 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.709 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.709 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.709 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.709 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T16:52:20.710 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T16:52:20.710 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.710 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.710 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.710 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.710 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.710 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.710 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.711 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.711 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.711 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.711 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T16:52:20.712 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T16:52:20.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.712 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.712 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.712 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.712 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.712 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.713 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.713 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.713 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.714 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T16:52:20.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T16:52:20.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.714 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.714 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.714 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.715 INFO:teuthology.orchestra.run.smithi098.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T16:52:20.716 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T16:52:20.716 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.716 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.716 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.716 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "[]", 2024-04-25T16:52:20.716 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.717 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.717 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.717 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.717 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T16:52:20.718 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T16:52:20.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.718 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.718 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.718 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-04-25T16:52:20.718 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.719 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.719 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.719 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.720 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.720 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T16:52:20.720 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T16:52:20.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.720 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.721 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.721 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.721 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.721 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.721 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.721 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.721 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.721 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.722 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.722 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T16:52:20.722 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T16:52:20.722 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.722 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.723 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.723 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.723 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.723 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.723 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.723 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.724 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.724 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T16:52:20.724 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T16:52:20.724 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.725 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.725 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.725 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.726 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.726 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.726 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T16:52:20.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T16:52:20.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.727 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.727 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.728 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.728 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T16:52:20.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T16:52:20.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.728 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.729 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.729 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.729 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.729 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.729 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.730 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.730 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.730 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T16:52:20.730 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T16:52:20.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.731 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.731 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.731 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.731 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.731 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.731 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.731 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.731 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.732 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.732 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.732 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T16:52:20.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T16:52:20.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.733 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.733 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.733 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.733 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.733 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.733 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.734 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T16:52:20.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T16:52:20.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.735 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.735 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.735 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.735 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.735 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.736 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.736 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.736 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T16:52:20.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T16:52:20.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.736 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.736 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.737 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.737 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.737 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.737 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.737 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.737 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.737 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.738 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.738 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T16:52:20.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T16:52:20.738 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.738 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.738 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.739 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.739 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.739 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.739 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.739 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.739 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.739 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.740 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.740 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.740 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T16:52:20.740 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T16:52:20.740 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.740 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.740 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.741 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T16:52:20.741 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.741 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.741 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.741 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.741 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.742 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.742 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.742 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.742 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T16:52:20.742 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T16:52:20.742 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.742 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.743 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.743 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.743 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.743 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.743 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.743 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.744 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.744 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.744 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T16:52:20.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T16:52:20.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.744 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-04-25T16:52:20.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.745 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.745 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.745 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.745 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.746 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.746 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T16:52:20.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T16:52:20.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.746 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.746 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.747 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-04-25T16:52:20.747 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.747 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.747 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.747 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.747 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.747 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.747 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.748 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T16:52:20.748 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T16:52:20.748 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.748 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.748 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.748 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.748 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.749 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.749 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.749 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.749 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.749 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.750 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T16:52:20.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T16:52:20.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.750 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.750 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.750 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:20.750 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.751 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T16:52:20.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T16:52:20.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.752 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.752 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.752 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.752 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.752 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.753 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.753 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.753 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.753 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.753 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.753 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.753 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T16:52:20.754 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T16:52:20.754 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.754 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.754 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.754 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.754 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.754 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.755 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.755 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.755 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.755 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.755 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.755 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.755 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T16:52:20.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T16:52:20.756 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.756 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.756 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.756 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.757 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T16:52:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T16:52:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.758 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.759 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T16:52:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T16:52:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.760 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.761 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.762 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T16:52:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T16:52:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.763 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.764 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-04-25T16:52:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-04-25T16:52:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.765 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.766 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.767 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-04-25T16:52:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-04-25T16:52:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.768 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.769 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-04-25T16:52:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-04-25T16:52:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.770 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-04-25T16:52:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.771 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-04-25T16:52:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-04-25T16:52:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.772 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-04-25T16:52:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T16:52:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T16:52:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.774 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.775 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.776 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.777 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.778 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.781 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.781 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.781 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.781 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.781 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.781 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.781 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.782 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.782 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.782 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.782 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.782 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.782 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.782 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.783 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.783 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.783 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.783 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.783 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.783 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.783 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.784 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.785 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.785 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.785 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.785 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.785 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-04-25T16:52:20.785 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.785 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.786 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.786 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.786 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-04-25T16:52:20.786 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T16:52:20.786 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.786 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.786 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.786 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.787 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.787 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.787 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.787 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.787 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.787 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.787 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.788 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.789 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.790 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-04-25T16:52:20.790 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.790 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.790 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.790 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.790 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.792 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.792 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.792 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.792 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.792 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.793 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.793 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.793 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T16:52:20.793 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T16:52:20.793 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.793 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.793 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.793 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.794 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.794 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.794 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.794 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.794 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.794 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.795 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.795 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-04-25T16:52:20.795 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-04-25T16:52:20.795 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.795 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.796 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-04-25T16:52:20.796 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.796 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.796 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.796 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.796 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.796 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.797 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.797 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.797 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T16:52:20.797 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T16:52:20.797 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.798 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.798 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.798 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-04-25T16:52:20.798 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.798 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.798 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.799 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.799 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-04-25T16:52:20.799 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.799 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.799 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.799 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.800 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.800 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T16:52:20.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T16:52:20.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.800 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.800 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.801 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T16:52:20.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T16:52:20.801 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T16:52:20.801 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.801 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.801 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.801 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.802 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.802 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-04-25T16:52:20.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-04-25T16:52:20.802 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.803 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.803 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.803 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.804 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.805 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.805 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.805 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-04-25T16:52:20.805 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.805 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.806 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-04-25T16:52:20.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-04-25T16:52:20.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.806 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.806 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.807 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.807 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.807 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.807 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-04-25T16:52:20.807 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.807 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.808 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.808 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.808 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.808 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.808 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.808 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.808 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.808 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.809 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.809 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.809 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.809 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.809 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.809 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.809 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.810 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.810 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.810 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.810 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.810 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.811 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.811 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.812 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.812 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.812 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.812 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.812 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.813 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.813 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.813 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.813 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.813 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.813 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.813 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.814 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.814 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.814 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.814 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.814 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.814 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.814 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.814 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.815 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.816 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.816 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.816 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.816 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.816 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.816 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.816 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.816 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.817 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.817 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.817 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.817 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-04-25T16:52:20.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-04-25T16:52:20.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.817 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.818 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-04-25T16:52:20.818 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.818 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.818 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.818 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T16:52:20.818 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.818 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-04-25T16:52:20.819 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.820 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.820 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.820 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T16:52:20.820 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.820 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-04-25T16:52:20.821 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-04-25T16:52:20.821 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.821 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.821 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.821 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-04-25T16:52:20.821 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.821 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.821 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.822 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-04-25T16:52:20.822 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.822 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.822 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.822 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-04-25T16:52:20.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-04-25T16:52:20.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.823 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.823 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.823 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T16:52:20.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.823 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.823 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.823 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T16:52:20.823 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.824 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.824 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.824 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-04-25T16:52:20.824 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-04-25T16:52:20.824 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.824 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.824 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.825 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:20.825 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.825 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.825 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.825 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T16:52:20.825 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:20.826 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.827 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.827 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.827 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T16:52:20.827 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.827 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.827 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.827 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.828 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.828 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.828 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.828 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.828 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.829 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.829 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.829 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.829 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-04-25T16:52:20.829 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-04-25T16:52:20.829 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.830 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.830 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-04-25T16:52:20.830 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.830 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.830 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.830 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T16:52:20.830 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.831 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.831 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.831 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.831 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.831 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.831 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-04-25T16:52:20.831 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.832 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.832 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.832 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.832 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.832 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.832 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.832 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.833 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.833 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.833 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.833 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.833 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.833 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.834 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.834 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.834 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.834 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.835 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.835 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.835 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.835 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.835 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.835 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.835 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.835 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.836 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.836 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.836 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.836 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.836 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.836 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.837 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.837 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.837 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.837 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.838 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.838 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.838 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.838 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.838 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.839 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.839 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.839 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.839 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.839 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.840 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.840 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.840 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.840 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.840 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.840 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.841 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.841 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.841 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.841 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.841 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.841 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.842 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.842 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.843 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.843 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.843 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.843 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.843 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.844 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.844 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-04-25T16:52:20.845 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-04-25T16:52:20.845 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.845 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.846 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.846 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T16:52:20.846 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.846 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.847 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.847 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.847 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.848 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.848 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.848 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-04-25T16:52:20.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-04-25T16:52:20.849 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.849 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.849 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-04-25T16:52:20.849 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.849 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.849 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.850 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.850 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.850 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.850 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:20.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:20.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.851 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.851 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.851 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:20.851 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.851 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.852 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.852 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.852 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.852 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.852 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.853 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.853 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.854 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.854 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.855 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-04-25T16:52:20.855 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-04-25T16:52:20.855 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-04-25T16:52:20.855 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.855 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-04-25T16:52:20.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-04-25T16:52:20.856 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.856 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.856 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-04-25T16:52:20.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.856 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.856 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T16:52:20.857 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.857 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.857 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-04-25T16:52:20.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-04-25T16:52:20.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.858 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.858 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-04-25T16:52:20.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.858 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.858 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.858 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T16:52:20.858 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.859 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.860 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.860 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.860 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.860 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-04-25T16:52:20.860 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.860 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.860 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.861 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:20.861 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:20.861 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:20.861 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.861 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.861 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-04-25T16:52:20.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-04-25T16:52:20.861 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.862 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.862 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T16:52:20.862 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.862 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.862 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.862 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.863 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.863 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.863 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.863 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.863 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.863 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.863 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.863 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.864 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.865 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.865 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.865 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.865 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.865 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.865 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.865 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.866 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.866 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.866 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.866 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.866 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.866 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.866 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.866 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.867 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.868 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.868 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.868 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.868 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.868 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.868 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.868 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.868 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.869 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.869 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.869 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.869 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.869 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.870 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.870 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.870 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.870 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.871 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.871 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.871 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.871 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.871 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.871 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.871 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-04-25T16:52:20.872 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-04-25T16:52:20.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.872 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.872 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.872 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.872 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.873 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.873 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.873 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-04-25T16:52:20.873 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.873 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.873 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.874 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-04-25T16:52:20.874 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-04-25T16:52:20.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.874 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.874 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-04-25T16:52:20.874 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.875 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.875 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-04-25T16:52:20.875 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.876 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.876 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.876 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.876 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.876 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.877 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.877 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.877 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.877 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.877 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.877 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.878 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.878 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.878 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.878 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T16:52:20.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T16:52:20.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.879 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.879 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.879 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-04-25T16:52:20.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.879 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.879 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:52:20.880 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.880 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.880 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-04-25T16:52:20.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-04-25T16:52:20.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.881 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.881 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.881 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:20.881 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:20.882 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-04-25T16:52:20.882 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.882 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T16:52:20.882 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.882 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.883 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.883 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-04-25T16:52:20.883 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-04-25T16:52:20.883 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.883 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.884 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.884 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.884 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.884 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.884 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.884 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-04-25T16:52:20.884 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-04-25T16:52:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-04-25T16:52:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.886 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-04-25T16:52:20.886 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.886 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.886 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.886 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:52:20.886 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.887 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.887 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.887 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.887 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-04-25T16:52:20.887 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.887 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.888 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.888 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.888 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.888 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.888 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.888 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.888 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.889 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.889 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.889 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.889 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.889 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.889 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.889 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.889 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.890 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.890 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.890 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.890 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.890 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.890 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.890 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.893 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.894 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.895 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.895 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.895 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.895 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.895 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.895 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.895 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.895 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.896 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.896 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.896 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.896 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.896 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.897 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.897 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.897 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.897 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.897 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.898 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.898 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.898 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.898 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.898 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.899 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-04-25T16:52:20.899 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.899 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.899 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.899 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.900 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.900 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.900 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.900 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.900 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.900 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.900 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.900 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.901 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.901 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.901 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.901 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.901 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.901 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.901 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.901 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.902 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.902 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.902 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.902 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.902 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.903 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.903 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.903 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.903 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.903 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.903 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.903 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.903 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.904 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.905 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.905 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.905 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.905 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.905 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.905 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.905 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.905 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.906 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.906 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.906 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.906 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.906 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.907 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.907 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.907 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.907 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.908 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.908 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.908 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.908 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.909 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.909 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.909 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.909 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.909 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.909 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.909 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.910 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.910 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.910 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.911 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.911 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.911 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.911 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.911 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.912 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-04-25T16:52:20.912 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.912 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.912 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.912 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-04-25T16:52:20.913 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-04-25T16:52:20.913 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.913 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.913 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.913 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-04-25T16:52:20.913 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.914 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.914 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.914 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T16:52:20.914 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.914 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.914 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.914 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-04-25T16:52:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-04-25T16:52:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-04-25T16:52:20.915 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T16:52:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.917 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.918 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.919 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.921 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.922 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.923 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.924 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.925 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.926 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.927 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.928 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.929 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.930 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-04-25T16:52:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.931 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-04-25T16:52:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-04-25T16:52:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-04-25T16:52:20.932 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-04-25T16:52:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T16:52:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.935 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.936 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.937 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.938 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.939 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.940 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.942 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.943 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-04-25T16:52:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-04-25T16:52:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.944 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.945 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-04-25T16:52:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-04-25T16:52:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.946 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-04-25T16:52:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-04-25T16:52:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.947 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-04-25T16:52:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T16:52:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-04-25T16:52:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-04-25T16:52:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-04-25T16:52:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-04-25T16:52:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-04-25T16:52:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-04-25T16:52:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T16:52:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T16:52:20.954 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-04-25T16:52:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.956 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.957 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.958 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.959 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.960 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.963 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.964 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.968 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.969 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-04-25T16:52:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.970 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.971 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.972 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.973 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.974 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.975 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.976 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.977 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.978 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.979 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.981 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-04-25T16:52:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.983 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.984 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.985 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:20.986 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.987 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.988 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:20.990 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:20.991 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.992 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.993 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.995 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.996 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T16:52:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:20.997 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-04-25T16:52:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-04-25T16:52:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:20.998 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T16:52:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:20.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.000 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.001 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.003 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.004 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.006 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.007 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.008 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.009 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-04-25T16:52:21.010 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T16:52:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.011 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-04-25T16:52:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-04-25T16:52:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-04-25T16:52:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-04-25T16:52:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.013 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.014 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-04-25T16:52:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.015 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.016 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.017 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.018 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.019 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.020 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.021 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.022 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.023 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.024 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.025 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-04-25T16:52:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.027 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.028 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.031 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.032 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.033 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.034 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.035 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.036 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.037 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.037 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.037 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.037 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.037 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.037 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.037 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.038 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.038 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.038 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.039 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.039 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.039 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.039 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.040 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.040 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.040 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.040 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.040 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.041 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.042 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.042 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-04-25T16:52:21.042 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.042 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.042 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.043 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.043 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.043 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.043 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.043 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.043 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.043 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.043 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.044 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.044 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.044 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.044 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.044 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.044 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.045 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.045 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.045 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.045 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.045 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.045 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.045 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.046 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.046 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.046 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.046 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.046 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.046 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.047 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.048 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.049 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.049 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.049 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.049 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.049 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.049 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.049 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.049 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.050 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.050 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.050 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.050 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.051 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.051 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.051 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.051 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.051 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.051 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.052 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.053 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:21.053 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.053 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.053 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.053 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.053 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.054 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.054 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.054 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.054 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.054 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.054 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.054 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.055 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.055 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.055 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.055 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.055 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.055 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.055 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T16:52:21.056 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-04-25T16:52:21.057 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.057 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.057 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-04-25T16:52:21.057 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T16:52:21.057 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.057 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.057 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.058 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.058 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-04-25T16:52:21.058 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.058 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.058 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.058 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-04-25T16:52:21.058 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-04-25T16:52:21.059 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.059 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.059 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.059 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.059 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.059 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.059 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.059 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T16:52:21.060 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.060 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.060 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.060 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T16:52:21.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T16:52:21.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.060 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.061 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.061 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.061 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.061 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.061 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.061 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.061 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.061 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.062 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.062 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.062 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.062 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.062 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.063 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.063 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.063 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.063 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.063 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.063 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.064 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.064 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.064 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.064 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.064 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.064 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.064 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.064 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.065 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.065 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.065 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.065 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.065 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.066 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.066 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.066 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.066 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.066 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.066 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.066 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.067 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.067 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.067 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.067 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.067 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.068 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.068 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.068 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.068 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.069 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.069 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.069 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.069 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.069 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.069 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.070 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.070 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.070 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.070 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.071 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.071 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.071 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.071 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.071 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.072 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.072 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.072 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.072 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.072 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.072 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-04-25T16:52:21.073 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-04-25T16:52:21.073 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.073 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.073 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.073 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-04-25T16:52:21.074 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.074 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.074 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.074 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-04-25T16:52:21.074 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.074 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.074 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.075 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.075 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:21.075 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:21.075 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.075 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.075 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.075 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:21.075 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-04-25T16:52:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.076 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.077 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.077 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.077 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.077 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.077 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.078 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.078 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.078 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.078 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.078 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.079 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.079 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.079 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-04-25T16:52:21.080 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.080 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.080 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.080 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-04-25T16:52:21.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-04-25T16:52:21.081 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.081 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.081 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.081 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.081 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.081 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.082 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.082 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.082 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.082 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.082 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.082 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-04-25T16:52:21.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-04-25T16:52:21.083 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.083 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.083 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.083 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.083 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.083 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.083 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.084 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T16:52:21.084 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T16:52:21.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.084 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.084 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.085 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.085 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.085 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.086 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.086 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.086 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.086 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.086 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.086 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.086 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.087 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.087 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.087 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.087 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.087 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.087 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.087 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.087 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.088 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.088 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.088 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.088 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.088 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.088 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.088 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.088 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.089 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.089 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.089 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.089 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.089 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.089 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.090 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.090 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.090 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.090 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.090 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.090 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.090 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.091 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.091 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.091 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.091 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.091 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.091 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.091 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.091 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.092 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.092 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.092 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.092 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.092 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.092 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.092 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.093 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.093 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.093 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.093 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.093 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.093 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.093 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.094 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-04-25T16:52:21.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-04-25T16:52:21.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.094 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.094 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.094 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.094 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.094 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.095 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.095 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.095 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.095 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.095 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.095 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T16:52:21.095 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T16:52:21.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.096 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.096 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-04-25T16:52:21.096 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.096 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.096 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.096 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.097 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.097 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.097 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-04-25T16:52:21.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-04-25T16:52:21.097 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:21.097 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.098 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.098 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-04-25T16:52:21.098 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.098 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.098 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.098 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.098 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.099 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.099 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:21.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:21.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.099 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T16:52:21.100 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.100 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.100 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.100 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T16:52:21.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.101 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:21.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:21.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.101 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-04-25T16:52:21.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.101 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.102 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T16:52:21.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.102 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.103 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.103 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.103 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.103 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.103 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.104 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.104 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-04-25T16:52:21.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-04-25T16:52:21.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.106 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.106 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.106 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T16:52:21.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T16:52:21.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.107 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-04-25T16:52:21.107 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.107 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.107 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.107 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T16:52:21.108 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T16:52:21.109 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.109 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.109 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.109 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T16:52:21.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T16:52:21.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T16:52:21.110 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.110 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.110 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.110 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.110 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.111 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.111 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-04-25T16:52:21.111 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.113 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.113 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.113 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.113 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.113 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.113 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.114 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.114 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.114 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.114 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.114 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.114 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.115 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.115 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.115 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.115 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.115 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.116 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.116 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.117 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.117 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.117 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.117 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.118 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.118 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.119 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.119 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.119 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.119 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.119 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.119 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.119 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.119 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.120 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.120 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.120 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.120 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.120 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.120 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.120 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.121 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.121 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.121 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.121 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.121 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.121 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.121 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.122 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.123 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.123 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-04-25T16:52:21.123 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-04-25T16:52:21.123 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.123 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.123 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:21.124 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.124 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.124 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.124 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.124 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.124 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.125 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-04-25T16:52:21.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T16:52:21.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.125 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.125 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.125 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.125 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.126 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.126 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.126 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.126 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.126 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.126 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.127 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.127 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.127 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.127 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.127 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.127 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.127 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.128 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.128 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.128 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.128 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.128 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.128 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.128 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.129 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-04-25T16:52:21.129 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-04-25T16:52:21.129 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.130 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-04-25T16:52:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.131 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-04-25T16:52:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-04-25T16:52:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.132 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T16:52:21.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T16:52:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.135 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.136 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.137 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.140 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.141 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.142 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.143 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.144 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:21.145 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.146 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.147 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.148 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-04-25T16:52:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.152 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.153 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.154 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.155 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.156 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.157 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.159 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.160 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.161 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.162 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.163 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-04-25T16:52:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.166 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.167 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.168 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.169 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.170 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.170 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.171 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.172 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.172 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.172 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.172 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.172 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.172 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.172 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.173 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.173 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.173 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.173 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.173 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.173 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.174 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.174 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.174 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.175 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.176 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.176 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.176 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.176 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.176 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.177 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.177 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-04-25T16:52:21.177 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-04-25T16:52:21.177 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.177 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.177 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.178 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-04-25T16:52:21.178 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.178 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.178 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.178 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T16:52:21.178 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.178 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.179 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.179 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-04-25T16:52:21.179 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.179 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.180 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.180 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.180 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.180 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.180 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.181 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.181 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.181 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.181 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.181 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.181 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.181 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.181 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.182 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.182 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.182 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.182 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.182 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.182 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.182 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.183 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.183 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.183 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.183 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.183 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.184 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.184 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.185 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.186 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.186 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.186 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.186 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.186 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.186 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.186 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.186 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.187 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.187 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.187 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.187 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.187 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.187 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.188 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.188 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.188 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.188 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.188 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.188 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.189 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.189 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.189 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.189 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.189 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.190 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.190 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.190 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.190 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-04-25T16:52:21.190 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-04-25T16:52:21.190 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.191 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.191 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.191 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.191 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.191 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.192 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.192 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.192 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.192 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.192 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-04-25T16:52:21.193 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-04-25T16:52:21.193 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.193 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.193 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.193 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-04-25T16:52:21.193 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.194 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.194 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.194 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.194 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.194 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.195 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.195 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-04-25T16:52:21.195 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-04-25T16:52:21.195 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.196 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.196 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.196 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.197 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.197 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.197 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.198 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-04-25T16:52:21.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-04-25T16:52:21.198 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.198 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.198 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.199 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.199 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.199 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.199 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.199 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.200 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.200 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-04-25T16:52:21.200 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-04-25T16:52:21.200 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:21.200 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.200 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.201 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.201 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.201 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.201 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.201 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.201 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.202 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.202 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-04-25T16:52:21.202 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-04-25T16:52:21.202 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.202 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.202 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.203 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.203 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.203 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.203 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.203 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.204 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.204 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.204 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-04-25T16:52:21.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-04-25T16:52:21.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.204 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.205 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.205 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.206 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.206 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-04-25T16:52:21.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-04-25T16:52:21.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.207 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.207 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.207 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.207 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.207 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.207 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.207 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.208 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-04-25T16:52:21.208 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-04-25T16:52:21.208 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.208 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.208 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.208 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.208 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:21.208 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-04-25T16:52:21.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.209 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.209 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.209 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.209 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.209 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.209 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.210 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.210 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.210 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.210 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.210 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.210 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-04-25T16:52:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-04-25T16:52:21.211 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.212 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.212 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.212 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.212 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.212 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.212 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.213 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.213 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.213 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.213 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-04-25T16:52:21.213 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-04-25T16:52:21.213 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.214 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.214 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.214 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.214 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.215 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-04-25T16:52:21.215 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-04-25T16:52:21.215 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.215 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.216 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.216 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.216 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.217 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.217 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.217 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-04-25T16:52:21.218 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.218 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.218 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.218 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-04-25T16:52:21.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-04-25T16:52:21.218 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.218 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T16:52:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.219 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.220 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.220 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.220 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-04-25T16:52:21.220 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-04-25T16:52:21.220 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.220 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.220 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.220 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.221 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.221 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.221 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.221 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-04-25T16:52:21.221 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.221 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.222 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.222 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.222 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.222 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.223 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.223 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.223 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.223 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.223 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.223 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.224 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.224 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.224 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.224 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.224 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.225 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.225 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.225 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.225 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.225 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.225 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.225 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.226 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.226 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.226 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.226 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.226 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.226 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.226 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.227 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.227 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.227 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.228 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.228 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.228 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.228 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.229 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.229 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.229 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.229 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.229 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.229 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.230 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.230 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.230 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.230 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.230 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.231 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.231 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.231 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.231 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.231 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.231 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.231 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.232 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.232 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.232 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.232 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.232 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.233 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.233 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.233 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.233 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.233 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.234 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.234 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.234 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.234 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.235 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.235 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.235 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.236 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.236 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-04-25T16:52:21.236 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.236 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.236 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.236 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.237 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.237 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.237 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.237 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.237 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.238 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.238 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.238 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.238 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.238 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.239 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.239 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.239 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.239 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.239 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.239 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.239 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.239 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.240 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.240 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.240 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.240 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.240 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.240 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.241 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.242 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.243 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.247 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-04-25T16:52:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.249 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.250 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.251 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.251 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.251 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.251 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.251 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.251 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.251 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.252 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.252 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.252 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.252 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.252 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.253 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.253 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.253 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.253 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.253 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.254 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.254 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.254 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.254 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.254 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.255 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.255 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.255 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.255 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.255 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.257 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.257 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.258 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.258 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.259 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.259 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.259 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.259 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.259 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.260 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.260 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.260 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.260 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.260 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.260 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.261 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.261 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-04-25T16:52:21.261 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.261 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.261 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.261 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-04-25T16:52:21.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-04-25T16:52:21.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.262 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T16:52:21.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.263 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.263 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.263 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.263 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.264 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:21.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:21.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.264 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.264 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-04-25T16:52:21.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.265 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.265 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.266 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.266 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.266 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.266 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.266 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.266 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.267 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.267 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.267 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.267 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.267 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.268 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.268 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.268 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.268 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.268 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.268 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.269 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.269 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.269 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.270 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.270 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.273 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.274 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.274 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.274 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.274 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.275 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.276 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.276 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.277 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.278 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.278 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.278 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.278 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.278 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.278 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.279 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.280 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.280 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.280 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.280 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.280 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-04-25T16:52:21.280 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.280 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.281 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.281 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-04-25T16:52:21.281 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-04-25T16:52:21.281 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.281 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.281 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.281 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.281 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.282 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.282 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.282 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T16:52:21.282 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.282 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.282 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.282 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.282 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-04-25T16:52:21.283 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-04-25T16:52:21.283 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.283 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.283 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.283 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.283 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.283 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.284 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.284 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T16:52:21.284 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.284 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.284 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.284 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.284 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-04-25T16:52:21.284 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T16:52:21.285 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.286 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.286 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.286 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.286 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-04-25T16:52:21.286 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-04-25T16:52:21.286 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.286 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.287 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.287 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.287 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.287 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.287 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.287 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T16:52:21.287 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.287 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.288 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.288 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.288 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-04-25T16:52:21.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-04-25T16:52:21.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.288 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.288 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.288 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.289 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-04-25T16:52:21.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-04-25T16:52:21.290 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.290 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.290 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.290 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.290 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.290 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.291 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.291 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.291 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.291 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.291 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.291 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-04-25T16:52:21.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-04-25T16:52:21.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.292 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.292 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.292 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.292 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.292 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.292 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.292 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.293 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.293 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.293 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.293 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.293 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-04-25T16:52:21.293 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-04-25T16:52:21.293 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.293 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.294 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.294 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T16:52:21.294 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.294 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.294 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.294 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.294 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.294 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T16:52:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T16:52:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.295 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.296 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:21.297 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:21.297 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.297 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.297 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.297 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-04-25T16:52:21.297 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-04-25T16:52:21.298 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.298 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.298 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.298 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.298 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.298 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.299 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.299 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-04-25T16:52:21.299 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-04-25T16:52:21.299 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.299 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.299 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.300 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-04-25T16:52:21.300 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.300 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.300 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.300 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.301 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.301 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.301 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.301 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.301 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-04-25T16:52:21.301 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-04-25T16:52:21.301 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.301 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.302 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.302 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.302 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.302 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.302 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.303 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.303 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.303 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.303 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.303 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.303 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-04-25T16:52:21.303 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-04-25T16:52:21.304 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.304 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.304 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.304 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.304 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.304 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.305 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.305 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.305 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.305 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.305 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.305 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.305 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.306 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.306 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.306 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.306 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.306 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.307 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.307 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.307 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.307 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.307 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.307 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.308 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.308 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.308 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.308 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.308 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.308 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.309 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.309 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.309 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.309 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.309 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.310 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.310 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.310 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.310 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.310 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.310 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.311 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.311 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.311 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.311 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.311 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.311 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.311 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.311 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.312 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.313 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.313 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.313 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.313 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.313 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.313 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.313 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.314 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.314 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.314 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.314 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.314 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.314 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.314 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.315 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.316 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.316 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.316 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-04-25T16:52:21.316 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-04-25T16:52:21.316 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.316 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.316 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.317 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.317 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.317 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.317 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.317 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.317 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.317 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.318 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.318 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.318 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-04-25T16:52:21.318 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-04-25T16:52:21.318 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.318 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.318 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.318 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.319 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.320 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.320 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.320 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.320 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.320 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.320 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.320 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.320 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-04-25T16:52:21.321 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.322 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.322 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.322 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T16:52:21.322 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.322 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.322 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.322 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.323 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.323 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.323 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.323 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.323 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.323 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.323 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.324 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.325 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.326 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.327 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.328 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.329 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.330 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.331 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.332 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.333 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.334 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-04-25T16:52:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.335 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.336 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.337 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.338 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.339 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.340 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.341 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.342 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.343 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.344 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.345 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-04-25T16:52:21.346 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-04-25T16:52:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-04-25T16:52:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.347 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T16:52:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-04-25T16:52:21.348 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-04-25T16:52:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.349 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-04-25T16:52:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.350 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-04-25T16:52:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-04-25T16:52:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.351 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T16:52:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.352 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-04-25T16:52:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T16:52:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.353 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T16:52:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.354 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.355 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.356 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.357 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.357 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.357 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.357 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.357 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.357 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-04-25T16:52:21.358 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.358 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.358 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.358 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-04-25T16:52:21.358 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-04-25T16:52:21.359 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:21.359 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.359 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.359 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-04-25T16:52:21.359 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.359 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.359 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.360 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.360 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.360 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.360 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.360 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.360 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-04-25T16:52:21.360 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-04-25T16:52:21.360 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.361 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.361 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.361 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.361 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.361 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.361 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.361 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.361 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.362 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.362 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.362 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.362 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:21.362 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:21.362 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.362 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.363 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.363 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:21.363 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.363 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.363 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.363 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.363 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.363 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.364 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.364 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.364 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.364 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.364 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.364 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.364 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.365 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.365 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.365 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.365 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.365 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.365 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.365 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.365 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.366 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.366 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.366 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.366 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.366 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.366 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.366 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.367 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.367 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.367 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.367 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.367 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.367 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.367 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.367 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.368 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.369 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.370 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.371 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.372 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.373 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.374 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-04-25T16:52:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-04-25T16:52:21.375 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.376 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-04-25T16:52:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-04-25T16:52:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.377 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-04-25T16:52:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.378 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-04-25T16:52:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-04-25T16:52:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.379 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T16:52:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.380 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:21.381 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "modules": [ 2024-04-25T16:52:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-04-25T16:52:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard", 2024-04-25T16:52:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "iostat", 2024-04-25T16:52:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "nfs", 2024-04-25T16:52:21.382 INFO:teuthology.orchestra.run.smithi098.stdout: "restful" 2024-04-25T16:52:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "available_modules": [ 2024-04-25T16:52:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "alerts", 2024-04-25T16:52:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.383 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.384 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T16:52:21.385 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.386 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.387 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.388 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.389 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.390 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.391 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.392 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.393 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.394 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.395 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.396 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.397 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_destination": { 2024-04-25T16:52:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_destination", 2024-04-25T16:52:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.398 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email address to send alerts to", 2024-04-25T16:52:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.399 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_from_name": { 2024-04-25T16:52:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_from_name", 2024-04-25T16:52:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "Ceph", 2024-04-25T16:52:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.400 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Email From: name", 2024-04-25T16:52:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.401 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_host": { 2024-04-25T16:52:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_host", 2024-04-25T16:52:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.402 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP server", 2024-04-25T16:52:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.403 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_password": { 2024-04-25T16:52:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_password", 2024-04-25T16:52:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.404 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Password to authenticate with", 2024-04-25T16:52:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.405 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_port": { 2024-04-25T16:52:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_port", 2024-04-25T16:52:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "465", 2024-04-25T16:52:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.406 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP port", 2024-04-25T16:52:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.407 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_sender": { 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_sender", 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.408 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SMTP envelope sender", 2024-04-25T16:52:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_ssl": { 2024-04-25T16:52:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_ssl", 2024-04-25T16:52:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.409 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T16:52:21.410 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "smtp_user": { 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "smtp_user", 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.411 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "User to authenticate as", 2024-04-25T16:52:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.412 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.413 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.414 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "balancer", 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "active": { 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "active", 2024-04-25T16:52:21.415 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.416 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T16:52:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_time": { 2024-04-25T16:52:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_time", 2024-04-25T16:52:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.417 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0000", 2024-04-25T16:52:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.418 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T16:52:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:52:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.419 INFO:teuthology.orchestra.run.smithi098.stdout: "begin_weekday": { 2024-04-25T16:52:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "begin_weekday", 2024-04-25T16:52:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T16:52:21.420 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T16:52:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T16:52:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:52:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.421 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_max_iterations": { 2024-04-25T16:52:21.422 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_max_iterations", 2024-04-25T16:52:21.422 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:21.422 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.422 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.422 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "25", 2024-04-25T16:52:21.422 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:21.422 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "250", 2024-04-25T16:52:21.422 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.423 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T16:52:21.423 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.423 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.423 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.423 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.423 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_metrics": { 2024-04-25T16:52:21.423 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_metrics", 2024-04-25T16:52:21.424 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.424 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.424 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.424 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T16:52:21.424 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.424 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.425 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.425 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T16:52:21.425 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T16:52:21.425 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.425 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.425 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.425 INFO:teuthology.orchestra.run.smithi098.stdout: "crush_compat_step": { 2024-04-25T16:52:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crush_compat_step", 2024-04-25T16:52:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.5", 2024-04-25T16:52:21.426 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0.001", 2024-04-25T16:52:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "0.999", 2024-04-25T16:52:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "aggressiveness of optimization", 2024-04-25T16:52:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T16:52:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.427 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "end_time": { 2024-04-25T16:52:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_time", 2024-04-25T16:52:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.428 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2359", 2024-04-25T16:52:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T16:52:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:52:21.429 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "end_weekday": { 2024-04-25T16:52:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "end_weekday", 2024-04-25T16:52:21.430 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "0", 2024-04-25T16:52:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "6", 2024-04-25T16:52:21.431 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T16:52:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:52:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.432 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.433 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.434 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.435 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.436 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.437 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.438 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.439 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.440 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.441 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "min_score": { 2024-04-25T16:52:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_score", 2024-04-25T16:52:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:21.442 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T16:52:21.443 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T16:52:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T16:52:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.444 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "upmap", 2024-04-25T16:52:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "crush-compat", 2024-04-25T16:52:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "none", 2024-04-25T16:52:21.445 INFO:teuthology.orchestra.run.smithi098.stdout: "read", 2024-04-25T16:52:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap", 2024-04-25T16:52:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap-read" 2024-04-25T16:52:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Balancer mode", 2024-04-25T16:52:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.446 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_ids": { 2024-04-25T16:52:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_ids", 2024-04-25T16:52:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.447 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T16:52:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.448 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.449 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T16:52:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.450 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.451 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_deviation": { 2024-04-25T16:52:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_deviation", 2024-04-25T16:52:21.452 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.453 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T16:52:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T16:52:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "upmap_max_optimizations": { 2024-04-25T16:52:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "upmap_max_optimizations", 2024-04-25T16:52:21.454 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T16:52:21.455 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm", 2024-04-25T16:52:21.456 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_down_multiplier": { 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_down_multiplier", 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T16:52:21.457 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T16:52:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.458 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_refresh_rate": { 2024-04-25T16:52:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_refresh_rate", 2024-04-25T16:52:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "20", 2024-04-25T16:52:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.459 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T16:52:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.460 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "agent_starting_port": { 2024-04-25T16:52:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "agent_starting_port", 2024-04-25T16:52:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.461 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4721", 2024-04-25T16:52:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T16:52:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.462 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_ptrace": { 2024-04-25T16:52:21.463 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_ptrace", 2024-04-25T16:52:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.464 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T16:52:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T16:52:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.465 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_interval": { 2024-04-25T16:52:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_interval", 2024-04-25T16:52:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.466 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T16:52:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.467 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "autotune_memory_target_ratio": { 2024-04-25T16:52:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T16:52:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.468 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0.7", 2024-04-25T16:52:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.469 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T16:52:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm_log_destination": { 2024-04-25T16:52:21.470 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cephadm_log_destination", 2024-04-25T16:52:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.471 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "file", 2024-04-25T16:52:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "file,syslog", 2024-04-25T16:52:21.472 INFO:teuthology.orchestra.run.smithi098.stdout: "syslog" 2024-04-25T16:52:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T16:52:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.473 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "cgroups_split": { 2024-04-25T16:52:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cgroups_split", 2024-04-25T16:52:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.474 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T16:52:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.475 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "config_checks_enabled": { 2024-04-25T16:52:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_checks_enabled", 2024-04-25T16:52:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.476 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T16:52:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.477 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "config_dashboard": { 2024-04-25T16:52:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_dashboard", 2024-04-25T16:52:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.478 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_alertmanager": { 2024-04-25T16:52:21.479 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_alertmanager", 2024-04-25T16:52:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T16:52:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.480 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_base": { 2024-04-25T16:52:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_base", 2024-04-25T16:52:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.481 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T16:52:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Container image name, without the tag", 2024-04-25T16:52:21.482 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_elasticsearch": { 2024-04-25T16:52:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_elasticsearch", 2024-04-25T16:52:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.483 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T16:52:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "elasticsearch container image", 2024-04-25T16:52:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.484 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_grafana": { 2024-04-25T16:52:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_grafana", 2024-04-25T16:52:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.485 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T16:52:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.486 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_haproxy": { 2024-04-25T16:52:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_haproxy", 2024-04-25T16:52:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.487 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T16:52:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "HAproxy container image", 2024-04-25T16:52:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.488 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_agent": { 2024-04-25T16:52:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_agent", 2024-04-25T16:52:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T16:52:21.489 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger agent container image", 2024-04-25T16:52:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.490 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_collector": { 2024-04-25T16:52:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_collector", 2024-04-25T16:52:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T16:52:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.491 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger collector container image", 2024-04-25T16:52:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.492 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_jaeger_query": { 2024-04-25T16:52:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_jaeger_query", 2024-04-25T16:52:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T16:52:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.493 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Jaeger query container image", 2024-04-25T16:52:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.494 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_keepalived": { 2024-04-25T16:52:21.495 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_keepalived", 2024-04-25T16:52:21.495 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.495 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.495 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.495 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T16:52:21.495 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.497 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.498 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.498 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Keepalived container image", 2024-04-25T16:52:21.498 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.498 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.498 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.498 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.498 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_loki": { 2024-04-25T16:52:21.499 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_loki", 2024-04-25T16:52:21.499 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.499 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.499 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.499 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T16:52:21.499 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.499 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.499 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.500 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Loki container image", 2024-04-25T16:52:21.500 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.500 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.500 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.500 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.500 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_node_exporter": { 2024-04-25T16:52:21.500 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_node_exporter", 2024-04-25T16:52:21.501 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.501 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.501 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.501 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T16:52:21.501 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.502 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.502 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.502 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:21.502 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.502 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.502 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.503 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.503 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_nvmeof": { 2024-04-25T16:52:21.503 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_nvmeof", 2024-04-25T16:52:21.503 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.503 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.504 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.504 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T16:52:21.504 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.504 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.504 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.504 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Nvme-of container image", 2024-04-25T16:52:21.504 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.505 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.505 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.505 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.505 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_prometheus": { 2024-04-25T16:52:21.505 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_prometheus", 2024-04-25T16:52:21.505 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.505 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.506 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.506 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T16:52:21.506 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.506 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.506 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Prometheus container image", 2024-04-25T16:52:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_promtail": { 2024-04-25T16:52:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_promtail", 2024-04-25T16:52:21.507 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T16:52:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Promtail container image", 2024-04-25T16:52:21.508 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_samba": { 2024-04-25T16:52:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_samba", 2024-04-25T16:52:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.509 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T16:52:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Samba/SMB container image", 2024-04-25T16:52:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.510 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "container_image_snmp_gateway": { 2024-04-25T16:52:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_image_snmp_gateway", 2024-04-25T16:52:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.511 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T16:52:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "SNMP Gateway container image", 2024-04-25T16:52:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.512 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "container_init": { 2024-04-25T16:52:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "container_init", 2024-04-25T16:52:21.513 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.514 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T16:52:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "daemon_cache_timeout": { 2024-04-25T16:52:21.515 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "daemon_cache_timeout", 2024-04-25T16:52:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.516 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T16:52:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.517 INFO:teuthology.orchestra.run.smithi098.stdout: "default_cephadm_command_timeout": { 2024-04-25T16:52:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T16:52:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.518 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "900", 2024-04-25T16:52:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T16:52:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.519 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_registry": { 2024-04-25T16:52:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default_registry", 2024-04-25T16:52:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.520 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "docker.io", 2024-04-25T16:52:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T16:52:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.521 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "device_cache_timeout": { 2024-04-25T16:52:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_cache_timeout", 2024-04-25T16:52:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.522 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1800", 2024-04-25T16:52:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache device inventory", 2024-04-25T16:52:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.523 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "device_enhanced_scan": { 2024-04-25T16:52:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_enhanced_scan", 2024-04-25T16:52:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.524 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T16:52:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.525 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "facts_cache_timeout": { 2024-04-25T16:52:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "facts_cache_timeout", 2024-04-25T16:52:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.526 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "seconds to cache host facts data", 2024-04-25T16:52:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.527 INFO:teuthology.orchestra.run.smithi098.stdout: "grafana_dashboards_path": { 2024-04-25T16:52:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "grafana_dashboards_path", 2024-04-25T16:52:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T16:52:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.528 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T16:52:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "host_check_interval": { 2024-04-25T16:52:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "host_check_interval", 2024-04-25T16:52:21.529 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.530 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.530 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.530 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:21.530 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.530 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.530 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.530 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to perform a host check", 2024-04-25T16:52:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: "hw_monitoring": { 2024-04-25T16:52:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hw_monitoring", 2024-04-25T16:52:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.531 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T16:52:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.532 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.533 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.533 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.533 INFO:teuthology.orchestra.run.smithi098.stdout: "inventory_list_all": { 2024-04-25T16:52:21.533 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "inventory_list_all", 2024-04-25T16:52:21.533 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.533 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.533 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.533 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T16:52:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.534 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.535 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.535 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.535 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.535 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.535 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.535 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.535 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.535 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.536 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.536 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.536 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.536 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.536 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.536 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.536 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.537 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.537 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.537 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.537 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.537 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.537 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.537 INFO:teuthology.orchestra.run.smithi098.stdout: "log_refresh_metadata": { 2024-04-25T16:52:21.537 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_refresh_metadata", 2024-04-25T16:52:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.538 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T16:52:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.539 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T16:52:21.540 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.541 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.542 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.543 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.544 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.545 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.545 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.545 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.545 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.545 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.546 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.546 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.546 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.546 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.546 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.546 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T16:52:21.547 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T16:52:21.547 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.547 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.547 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.547 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.547 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.548 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.548 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.548 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T16:52:21.548 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.548 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.548 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.549 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.549 INFO:teuthology.orchestra.run.smithi098.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T16:52:21.549 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T16:52:21.549 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.549 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.549 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.549 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "*", 2024-04-25T16:52:21.549 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.550 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.550 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.550 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T16:52:21.550 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.550 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.550 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.551 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.551 INFO:teuthology.orchestra.run.smithi098.stdout: "max_count_per_host": { 2024-04-25T16:52:21.551 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_count_per_host", 2024-04-25T16:52:21.551 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.551 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.551 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.552 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:21.552 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.552 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.552 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.552 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of daemons per service per host", 2024-04-25T16:52:21.552 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.553 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.553 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.553 INFO:teuthology.orchestra.run.smithi098.stdout: "max_osd_draining_count": { 2024-04-25T16:52:21.553 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_osd_draining_count", 2024-04-25T16:52:21.553 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.554 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.554 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.554 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:21.554 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.554 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.554 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.554 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T16:52:21.555 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.555 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.555 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.555 INFO:teuthology.orchestra.run.smithi098.stdout: "migration_current": { 2024-04-25T16:52:21.556 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "migration_current", 2024-04-25T16:52:21.556 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.556 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.556 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.556 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.556 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.556 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.557 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.557 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "internal - do not modify", 2024-04-25T16:52:21.557 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.557 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.557 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.558 INFO:teuthology.orchestra.run.smithi098.stdout: "mode": { 2024-04-25T16:52:21.558 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mode", 2024-04-25T16:52:21.558 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.558 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.558 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.558 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "root", 2024-04-25T16:52:21.559 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.559 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.559 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.559 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm-package", 2024-04-25T16:52:21.559 INFO:teuthology.orchestra.run.smithi098.stdout: "root" 2024-04-25T16:52:21.560 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.560 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T16:52:21.560 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.560 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.560 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.561 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.561 INFO:teuthology.orchestra.run.smithi098.stdout: "oob_default_addr": { 2024-04-25T16:52:21.561 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "oob_default_addr", 2024-04-25T16:52:21.561 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.561 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.561 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.561 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "169.254.1.1", 2024-04-25T16:52:21.561 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.562 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.562 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.562 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T16:52:21.562 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.562 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.562 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.562 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.563 INFO:teuthology.orchestra.run.smithi098.stdout: "prometheus_alerts_path": { 2024-04-25T16:52:21.563 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus_alerts_path", 2024-04-25T16:52:21.563 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.563 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.563 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.563 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T16:52:21.564 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.564 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.564 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.564 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T16:52:21.564 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.564 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.564 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.564 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.565 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_insecure": { 2024-04-25T16:52:21.565 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_insecure", 2024-04-25T16:52:21.565 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.565 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.565 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.565 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.565 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T16:52:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.566 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_password": { 2024-04-25T16:52:21.567 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_password", 2024-04-25T16:52:21.567 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.567 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.567 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.567 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.567 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.567 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.568 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T16:52:21.568 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.568 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.568 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.568 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.568 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_url": { 2024-04-25T16:52:21.568 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_url", 2024-04-25T16:52:21.568 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.569 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.570 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.570 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.570 INFO:teuthology.orchestra.run.smithi098.stdout: "registry_username": { 2024-04-25T16:52:21.570 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "registry_username", 2024-04-25T16:52:21.570 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.570 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.570 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.570 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.571 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.571 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.571 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.571 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T16:52:21.571 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.571 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.571 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.571 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.572 INFO:teuthology.orchestra.run.smithi098.stdout: "secure_monitoring_stack": { 2024-04-25T16:52:21.572 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "secure_monitoring_stack", 2024-04-25T16:52:21.572 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.572 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.572 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.572 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.572 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.572 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.573 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.573 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T16:52:21.573 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.573 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.573 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.573 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.573 INFO:teuthology.orchestra.run.smithi098.stdout: "service_discovery_port": { 2024-04-25T16:52:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "service_discovery_port", 2024-04-25T16:52:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8765", 2024-04-25T16:52:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.574 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "cephadm service discovery port", 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.575 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.576 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.576 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.576 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.576 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.576 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.576 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.576 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.576 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: "ssh_config_file": { 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssh_config_file", 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.577 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.578 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.578 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.578 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T16:52:21.578 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.578 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.578 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "use_agent": { 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_agent", 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.579 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.580 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T16:52:21.580 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.580 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.580 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.580 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.580 INFO:teuthology.orchestra.run.smithi098.stdout: "use_repo_digest": { 2024-04-25T16:52:21.580 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "use_repo_digest", 2024-04-25T16:52:21.580 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.581 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.582 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.582 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.582 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_failed_host_check": { 2024-04-25T16:52:21.582 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_failed_host_check", 2024-04-25T16:52:21.582 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.582 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.582 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.583 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.583 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.583 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.583 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.583 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T16:52:21.583 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.583 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.583 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.584 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.584 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_daemons": { 2024-04-25T16:52:21.584 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_daemons", 2024-04-25T16:52:21.584 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.585 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.585 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.585 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.585 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.585 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.585 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.586 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T16:52:21.586 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.586 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.586 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.586 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.587 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_on_stray_hosts": { 2024-04-25T16:52:21.587 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_on_stray_hosts", 2024-04-25T16:52:21.587 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.587 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.587 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.587 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.587 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.588 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.588 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.588 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T16:52:21.588 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.588 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.588 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.589 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.589 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.589 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.589 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crash", 2024-04-25T16:52:21.589 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.590 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.590 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.590 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.590 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.590 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.591 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.591 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.591 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.591 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.591 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.591 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.592 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.592 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.592 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.592 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.592 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.593 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.593 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.593 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.593 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.593 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.593 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.593 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.593 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.594 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.594 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.594 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.594 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.594 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.594 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.594 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.595 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.595 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.595 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.595 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.595 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.595 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.595 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.596 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.596 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.596 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.596 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.596 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.596 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.596 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.596 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.597 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.597 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.597 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.597 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.597 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.598 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.598 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.598 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.598 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.598 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.598 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.599 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.599 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.599 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.599 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.599 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.600 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.600 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.600 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.600 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.600 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.601 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.601 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.601 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.601 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.601 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.601 INFO:teuthology.orchestra.run.smithi098.stdout: "retain_interval": { 2024-04-25T16:52:21.601 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retain_interval", 2024-04-25T16:52:21.602 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.602 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.602 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.602 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "31536000", 2024-04-25T16:52:21.602 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.602 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.602 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.603 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T16:52:21.603 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.603 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.603 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.603 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.603 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.604 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.604 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.604 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.604 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.604 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.604 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.604 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.604 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_recent_interval": { 2024-04-25T16:52:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_recent_interval", 2024-04-25T16:52:21.605 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.606 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.606 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.606 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1209600", 2024-04-25T16:52:21.606 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.606 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.606 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.606 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T16:52:21.606 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dashboard", 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.607 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.608 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.608 INFO:teuthology.orchestra.run.smithi098.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T16:52:21.608 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T16:52:21.608 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.608 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.608 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.608 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:21.608 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T16:52:21.609 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T16:52:21.610 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.610 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.610 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.610 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.610 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.610 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.610 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.610 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.611 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.612 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_ENABLED": { 2024-04-25T16:52:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T16:52:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.613 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.614 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.614 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.614 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.614 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.614 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.614 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.614 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.614 INFO:teuthology.orchestra.run.smithi098.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.615 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T16:52:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T16:52:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.616 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.617 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.618 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.618 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T16:52:21.618 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T16:52:21.618 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.618 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.618 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.618 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.618 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.619 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.619 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.619 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.619 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.619 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.619 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.619 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.619 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T16:52:21.620 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T16:52:21.620 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.620 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.620 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.620 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.620 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.620 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.620 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.621 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.621 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.621 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.621 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.621 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.621 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T16:52:21.621 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T16:52:21.621 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.622 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.622 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.622 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.622 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.622 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.623 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.623 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.623 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.623 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.623 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.623 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.623 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T16:52:21.623 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T16:52:21.624 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.624 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.624 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.624 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.624 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.624 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.625 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T16:52:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T16:52:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.626 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.627 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.627 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.627 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.627 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.627 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.627 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.627 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.627 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.628 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T16:52:21.628 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T16:52:21.628 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.628 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.628 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.629 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.629 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.629 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.629 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.629 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.629 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.629 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.630 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.630 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.630 INFO:teuthology.orchestra.run.smithi098.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T16:52:21.630 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T16:52:21.630 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.630 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.630 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.631 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.631 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.631 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.631 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.631 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.631 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.632 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.632 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.632 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.632 INFO:teuthology.orchestra.run.smithi098.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T16:52:21.632 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T16:52:21.633 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.633 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.633 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.633 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.633 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.633 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.634 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.634 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.634 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.634 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.634 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.634 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T16:52:21.635 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T16:52:21.635 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.635 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.635 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.635 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:21.635 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.636 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.636 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.636 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.636 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.636 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.636 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.636 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.636 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.637 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_URL": { 2024-04-25T16:52:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_URL", 2024-04-25T16:52:21.638 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.640 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.640 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T16:52:21.640 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T16:52:21.640 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.640 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.640 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.640 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:21.641 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.641 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.641 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.641 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.641 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.641 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.642 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T16:52:21.642 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T16:52:21.642 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.642 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.642 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.642 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.642 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T16:52:21.643 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.644 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.645 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.645 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.645 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.645 INFO:teuthology.orchestra.run.smithi098.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T16:52:21.645 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T16:52:21.645 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.645 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.645 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.646 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T16:52:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T16:52:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.647 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.648 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T16:52:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T16:52:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "[]", 2024-04-25T16:52:21.649 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.650 INFO:teuthology.orchestra.run.smithi098.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T16:52:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T16:52:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "{}", 2024-04-25T16:52:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.651 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T16:52:21.652 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T16:52:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.653 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T16:52:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T16:52:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.654 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.655 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.656 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T16:52:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T16:52:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.657 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.658 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T16:52:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T16:52:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.659 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.660 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T16:52:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T16:52:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.661 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.662 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T16:52:21.663 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T16:52:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.664 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.665 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T16:52:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T16:52:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.666 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T16:52:21.667 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.668 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T16:52:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T16:52:21.669 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.670 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.671 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T16:52:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T16:52:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.672 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.673 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T16:52:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T16:52:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T16:52:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.674 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T16:52:21.675 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T16:52:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.676 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T16:52:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T16:52:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.677 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8", 2024-04-25T16:52:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.678 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T16:52:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T16:52:21.679 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "45", 2024-04-25T16:52:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.680 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T16:52:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T16:52:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.681 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.682 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T16:52:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T16:52:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.683 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "admin", 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.684 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T16:52:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T16:52:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.685 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T16:52:21.686 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T16:52:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.687 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T16:52:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T16:52:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.688 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.689 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T16:52:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T16:52:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.690 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.691 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T16:52:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T16:52:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.692 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T16:52:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T16:52:21.693 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.694 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "cross_origin_url": { 2024-04-25T16:52:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cross_origin_url", 2024-04-25T16:52:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.695 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.696 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "crt_file": { 2024-04-25T16:52:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "crt_file", 2024-04-25T16:52:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.697 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.698 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "debug": { 2024-04-25T16:52:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "debug", 2024-04-25T16:52:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.699 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.702 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Enable/disable debug options", 2024-04-25T16:52:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.703 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "jwt_token_ttl": { 2024-04-25T16:52:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "jwt_token_ttl", 2024-04-25T16:52:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.704 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "28800", 2024-04-25T16:52:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.705 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T16:52:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T16:52:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.706 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.707 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.708 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.709 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.709 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.709 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.709 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.709 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.710 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.711 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.712 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.713 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.714 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.715 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.716 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.717 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.718 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "motd": { 2024-04-25T16:52:21.719 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "motd", 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "The message of the day", 2024-04-25T16:52:21.720 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_resolve_ip_addr": { 2024-04-25T16:52:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T16:52:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.721 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.722 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.723 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T16:52:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.724 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8080", 2024-04-25T16:52:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.725 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.726 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.727 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T16:52:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T16:52:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.728 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.729 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl_server_port": { 2024-04-25T16:52:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl_server_port", 2024-04-25T16:52:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.730 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8443", 2024-04-25T16:52:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.731 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T16:52:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T16:52:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.732 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "redirect", 2024-04-25T16:52:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect" 2024-04-25T16:52:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.733 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T16:52:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T16:52:21.734 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T16:52:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T16:52:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T16:52:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.735 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "url_prefix": { 2024-04-25T16:52:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url_prefix", 2024-04-25T16:52:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.736 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.737 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.738 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-04-25T16:52:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_monitoring": { 2024-04-25T16:52:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_monitoring", 2024-04-25T16:52:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.739 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "monitor device health metrics", 2024-04-25T16:52:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.740 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.741 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.742 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.743 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.744 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.745 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.746 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.747 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.748 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.749 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.750 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.751 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "mark_out_threshold": { 2024-04-25T16:52:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mark_out_threshold", 2024-04-25T16:52:21.752 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "2419200", 2024-04-25T16:52:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.753 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T16:52:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.754 INFO:teuthology.orchestra.run.smithi098.stdout: "pool_name": { 2024-04-25T16:52:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pool_name", 2024-04-25T16:52:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "device_health_metrics", 2024-04-25T16:52:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.755 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T16:52:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "retention_period": { 2024-04-25T16:52:21.756 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "retention_period", 2024-04-25T16:52:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15552000", 2024-04-25T16:52:21.757 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long to retain device health metrics", 2024-04-25T16:52:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.758 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_frequency": { 2024-04-25T16:52:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_frequency", 2024-04-25T16:52:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.759 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T16:52:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T16:52:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.760 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "self_heal": { 2024-04-25T16:52:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "self_heal", 2024-04-25T16:52:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.761 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T16:52:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.762 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:21.763 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T16:52:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.764 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.765 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "warn_threshold": { 2024-04-25T16:52:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "warn_threshold", 2024-04-25T16:52:21.766 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7257600", 2024-04-25T16:52:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T16:52:21.767 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.768 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "diskprediction_local", 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.769 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.770 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.771 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.772 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.773 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.774 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.775 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.776 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.777 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "predict_interval": { 2024-04-25T16:52:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predict_interval", 2024-04-25T16:52:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "86400", 2024-04-25T16:52:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.778 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "predictor_model": { 2024-04-25T16:52:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "predictor_model", 2024-04-25T16:52:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.780 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "prophetstor", 2024-04-25T16:52:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.781 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:21.782 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "600", 2024-04-25T16:52:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.783 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.784 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.785 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.786 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "influx", 2024-04-25T16:52:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": false, 2024-04-25T16:52:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "influxdb python module not found", 2024-04-25T16:52:21.787 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.788 INFO:teuthology.orchestra.run.smithi098.stdout: "batch_size": { 2024-04-25T16:52:21.788 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "batch_size", 2024-04-25T16:52:21.788 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.788 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.788 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.788 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5000", 2024-04-25T16:52:21.788 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.789 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.789 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.789 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T16:52:21.789 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.789 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.789 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.789 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.790 INFO:teuthology.orchestra.run.smithi098.stdout: "database": { 2024-04-25T16:52:21.790 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "database", 2024-04-25T16:52:21.790 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.790 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.790 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.791 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "ceph", 2024-04-25T16:52:21.791 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.791 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.791 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.791 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T16:52:21.791 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.792 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.792 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.792 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.792 INFO:teuthology.orchestra.run.smithi098.stdout: "hostname": { 2024-04-25T16:52:21.792 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "hostname", 2024-04-25T16:52:21.792 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.792 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.792 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.793 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.793 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.793 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.793 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.793 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server hostname", 2024-04-25T16:52:21.793 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.793 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.794 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.794 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.794 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:21.794 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:21.794 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.794 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.794 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.794 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "30", 2024-04-25T16:52:21.795 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "5", 2024-04-25T16:52:21.795 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.795 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.795 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T16:52:21.795 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.795 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.795 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.795 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.796 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.796 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.796 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.796 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.796 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.796 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.796 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.796 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.797 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.798 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.798 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.798 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.798 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.798 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.798 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.798 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.798 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.799 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.799 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.799 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.799 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.799 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.799 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.799 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.800 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.801 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.802 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.802 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.802 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.802 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.802 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.802 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.802 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.802 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.803 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: "password": { 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "password", 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.804 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "password of InfluxDB server user", 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.805 INFO:teuthology.orchestra.run.smithi098.stdout: "port": { 2024-04-25T16:52:21.806 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "port", 2024-04-25T16:52:21.806 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.806 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.806 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.806 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "8086", 2024-04-25T16:52:21.806 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.806 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.806 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.807 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "InfluxDB server port", 2024-04-25T16:52:21.807 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.807 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.807 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.807 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.807 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.807 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.808 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.809 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.809 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.809 INFO:teuthology.orchestra.run.smithi098.stdout: "ssl": { 2024-04-25T16:52:21.809 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ssl", 2024-04-25T16:52:21.809 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.809 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.809 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.809 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "false", 2024-04-25T16:52:21.810 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.810 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.810 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.810 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:52:21.810 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.810 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.810 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: "threads": { 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threads", 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:21.811 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "32", 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: "username": { 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "username", 2024-04-25T16:52:21.812 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "username of InfluxDB server user", 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.813 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.814 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.814 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.814 INFO:teuthology.orchestra.run.smithi098.stdout: "verify_ssl": { 2024-04-25T16:52:21.814 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "verify_ssl", 2024-04-25T16:52:21.814 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.814 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.815 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.815 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "true", 2024-04-25T16:52:21.815 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.815 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.815 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.815 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:52:21.815 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.815 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.816 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.816 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.816 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.816 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.816 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "insights", 2024-04-25T16:52:21.816 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.816 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.817 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.817 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.817 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.817 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.817 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.817 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.818 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.818 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.818 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.818 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.818 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.818 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.818 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.819 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.819 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.819 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.819 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.819 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.819 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.820 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.820 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.820 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.820 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.820 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.820 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.820 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.820 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.821 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.821 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.821 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.821 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.821 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.821 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.821 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.822 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.822 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.822 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.822 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.822 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.822 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.822 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.823 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.823 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.823 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.823 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.823 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.823 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.824 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.824 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.824 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.824 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.824 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.824 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.825 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.825 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.825 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.825 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.825 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.825 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.826 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.826 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.826 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.826 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.826 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.826 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.827 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.827 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.827 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.827 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.827 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.827 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.828 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.828 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.828 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.828 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.828 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.828 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.828 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.828 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.829 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.829 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.829 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.829 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.829 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "iostat", 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.830 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.831 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.831 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.831 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.831 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.831 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.831 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.831 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.832 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.832 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.832 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.832 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.832 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.832 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.832 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.832 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.833 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.834 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.834 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.834 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.834 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.834 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.834 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.834 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.834 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.835 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.836 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.837 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.838 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.838 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.838 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.838 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.838 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.838 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.838 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.838 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.839 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "k8sevents", 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.840 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.841 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.841 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_event_retention_days": { 2024-04-25T16:52:21.841 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "ceph_event_retention_days", 2024-04-25T16:52:21.841 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.841 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.841 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.841 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "7", 2024-04-25T16:52:21.841 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: "config_check_secs": { 2024-04-25T16:52:21.842 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "config_check_secs", 2024-04-25T16:52:21.843 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.843 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.843 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.843 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:21.843 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "10", 2024-04-25T16:52:21.843 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.843 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.843 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T16:52:21.844 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.844 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.844 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.844 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.844 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.844 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.844 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.844 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.845 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.846 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.846 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.846 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.846 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.846 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.846 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.846 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.847 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.848 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.848 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.848 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.848 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.848 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.848 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.848 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.849 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.851 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.852 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.853 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.853 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.853 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.853 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.853 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.853 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.853 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.854 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "localpool", 2024-04-25T16:52:21.855 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.855 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.855 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.855 INFO:teuthology.orchestra.run.smithi098.stdout: "failure_domain": { 2024-04-25T16:52:21.855 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "failure_domain", 2024-04-25T16:52:21.855 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.855 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.856 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.856 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "host", 2024-04-25T16:52:21.856 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.856 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.856 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.856 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "failure domain for any created local pool", 2024-04-25T16:52:21.856 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T16:52:21.857 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.857 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.857 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.857 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.857 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.857 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.857 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.858 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.858 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.858 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.858 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.858 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.858 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.858 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.859 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.859 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.859 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.859 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.859 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.859 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.859 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.859 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.860 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.860 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.860 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.860 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.860 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.860 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.861 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.861 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.861 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.861 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.861 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.861 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.861 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.862 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.862 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.862 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.862 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.862 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.862 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.863 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.863 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.863 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.863 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.863 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.864 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.864 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.864 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.864 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.864 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.864 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.864 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.865 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.865 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.865 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.866 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.866 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.866 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.866 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.866 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.866 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.866 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.866 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.867 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.867 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.867 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.867 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.867 INFO:teuthology.orchestra.run.smithi098.stdout: "min_size": { 2024-04-25T16:52:21.867 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "min_size", 2024-04-25T16:52:21.867 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default min_size for any created local pool", 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T16:52:21.868 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.869 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.869 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.869 INFO:teuthology.orchestra.run.smithi098.stdout: "num_rep": { 2024-04-25T16:52:21.869 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "num_rep", 2024-04-25T16:52:21.869 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.869 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.869 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3", 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default replica count for any created local pool", 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.870 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.871 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_num": { 2024-04-25T16:52:21.871 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_num", 2024-04-25T16:52:21.871 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.871 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.871 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.871 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "128", 2024-04-25T16:52:21.871 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.871 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: "prefix": { 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prefix", 2024-04-25T16:52:21.872 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.873 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.873 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.873 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.873 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.873 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.873 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.873 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "name prefix for any created local pool", 2024-04-25T16:52:21.873 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.874 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.875 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "subtree": { 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "subtree", 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "rack", 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.876 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.877 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mds_autoscaler", 2024-04-25T16:52:21.878 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.878 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.878 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.878 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.878 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.878 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.878 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.878 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.879 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.880 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.881 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.882 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.882 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.882 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.882 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.882 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.882 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.882 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.883 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.884 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.884 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.884 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.884 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.884 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.884 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.884 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.884 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.885 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.886 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.887 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.887 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.887 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.887 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.887 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.887 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.887 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.887 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.888 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.888 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.888 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.888 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirroring", 2024-04-25T16:52:21.888 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.888 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.889 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.889 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.889 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.889 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.889 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.889 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.889 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.890 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.890 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.890 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.890 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.890 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.890 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.890 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.890 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.891 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.891 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.891 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.891 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.891 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.891 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.892 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.893 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.893 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.893 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.893 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.893 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.893 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.894 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.894 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.894 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.894 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.894 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.894 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.894 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.895 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.895 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.895 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.895 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.895 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.895 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.895 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.896 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.896 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.896 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.896 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.896 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.896 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.896 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.896 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.897 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.897 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.897 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.897 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.897 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.897 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.897 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.898 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.898 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.898 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.898 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.898 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.898 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.898 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.899 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.899 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.899 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.899 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.899 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.899 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.900 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.900 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.900 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.900 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.900 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "nfs", 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.901 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.902 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.902 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.902 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.902 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.902 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.902 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.902 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.902 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.903 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.903 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.903 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.903 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.903 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.905 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.905 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.905 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.906 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.906 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.906 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.906 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.906 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.906 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.906 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.906 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.907 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.908 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.909 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.910 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.910 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.910 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.910 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.910 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.910 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.910 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.910 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.911 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.912 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.913 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T16:52:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: "fail_fs": { 2024-04-25T16:52:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "fail_fs", 2024-04-25T16:52:21.914 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.915 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.916 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.917 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.917 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.917 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.917 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.917 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.917 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.917 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.917 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.918 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.919 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.920 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.921 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.921 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.921 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.921 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.921 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.921 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.921 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.921 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.922 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.923 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator": { 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "orchestrator", 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.924 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: "cephadm", 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: "rook", 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: "test_orchestrator" 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Orchestrator backend", 2024-04-25T16:52:21.925 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.926 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.926 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.926 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.926 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.926 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.927 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.927 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.927 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.927 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.927 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.927 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.927 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.927 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.928 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.928 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.928 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.928 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.928 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.928 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_perf_query", 2024-04-25T16:52:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.929 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.930 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.930 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.930 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.930 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.930 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.930 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.930 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.930 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.931 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.932 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.932 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.932 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.932 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.932 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.932 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.932 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.932 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.933 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.934 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.934 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.934 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.934 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.934 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.934 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.934 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.934 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.935 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.935 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.935 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.935 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.935 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.935 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.935 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.935 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.936 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.936 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.936 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.936 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.936 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.936 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.937 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.937 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.937 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.937 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.937 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.937 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.937 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.937 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.938 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.938 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.938 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.938 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.938 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.938 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.939 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.939 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.939 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.939 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.939 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.939 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.940 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.940 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.940 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.940 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.940 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.940 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "osd_support", 2024-04-25T16:52:21.941 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.941 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.941 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.941 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.941 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.942 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.942 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.942 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.942 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.942 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.942 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.942 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.943 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.943 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.943 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.943 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.943 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.943 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.943 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.943 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.944 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.944 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.944 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.944 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.944 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.944 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.944 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.944 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.945 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.946 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.946 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.946 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.946 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.946 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.946 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.946 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.946 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.947 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.948 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.949 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.949 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.949 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.949 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.949 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.949 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.949 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.950 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.950 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.950 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.950 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.950 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.950 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.950 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.950 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.951 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.952 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.952 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.952 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.952 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "pg_autoscaler", 2024-04-25T16:52:21.952 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.952 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.952 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.953 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.954 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.954 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.954 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.954 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.954 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.954 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.954 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.954 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.955 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.955 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.955 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.955 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.955 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.955 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.955 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.955 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.956 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.957 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.957 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.957 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.957 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.957 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.957 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.957 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.957 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.958 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.958 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.958 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.958 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.958 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.958 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.958 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.958 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.959 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.959 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.959 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.959 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.959 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.959 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.959 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.959 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.960 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.960 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.960 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.960 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.960 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.960 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.960 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.960 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.961 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.961 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.961 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.961 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.961 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.961 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:21.961 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:21.962 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.962 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.962 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.962 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:21.962 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.962 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.962 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.962 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.963 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.963 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.963 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.963 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.963 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.963 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.963 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.963 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.964 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.965 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.965 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.965 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.965 INFO:teuthology.orchestra.run.smithi098.stdout: "threshold": { 2024-04-25T16:52:21.965 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "threshold", 2024-04-25T16:52:21.965 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:21.965 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.966 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.966 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "3.0", 2024-04-25T16:52:21.966 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1.0", 2024-04-25T16:52:21.966 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.966 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.966 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "scaling threshold", 2024-04-25T16:52:21.966 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T16:52:21.966 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.967 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.967 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.967 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.967 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.967 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "progress", 2024-04-25T16:52:21.968 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.968 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.968 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.968 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_pg_recovery_event": { 2024-04-25T16:52:21.968 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_pg_recovery_event", 2024-04-25T16:52:21.968 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.968 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.968 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.969 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.969 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.969 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.969 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.969 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T16:52:21.969 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.970 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.970 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.970 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.970 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T16:52:21.970 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T16:52:21.970 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.970 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.971 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.971 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.971 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.971 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.971 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.971 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.971 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.972 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.972 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.972 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.972 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.972 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.972 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.972 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.973 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.973 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.973 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.973 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.973 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.973 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.974 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.974 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.974 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.974 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.974 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.974 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.974 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.975 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.975 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.975 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.975 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.975 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.975 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.976 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.976 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.976 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.976 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.976 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.976 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.977 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.977 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.977 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.977 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.977 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.977 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.977 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.978 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.978 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.978 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.978 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.978 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.978 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.978 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.978 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.979 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.980 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.981 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.982 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.982 INFO:teuthology.orchestra.run.smithi098.stdout: "max_completed_events": { 2024-04-25T16:52:21.982 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_completed_events", 2024-04-25T16:52:21.982 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.982 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.982 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.982 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "50", 2024-04-25T16:52:21.982 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.983 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.983 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.983 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "number of past completed events to remember", 2024-04-25T16:52:21.983 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.983 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.983 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.983 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.983 INFO:teuthology.orchestra.run.smithi098.stdout: "sleep_interval": { 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sleep_interval", 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "5", 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.984 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "how long the module is going to sleep", 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:21.985 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.986 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "prometheus", 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: "cache": { 2024-04-25T16:52:21.987 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "cache", 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.988 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.989 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.989 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.989 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.989 INFO:teuthology.orchestra.run.smithi098.stdout: "exclude_perf_counters": { 2024-04-25T16:52:21.989 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "exclude_perf_counters", 2024-04-25T16:52:21.989 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.989 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.989 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.990 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:21.990 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.990 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.990 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.990 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T16:52:21.990 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T16:52:21.990 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.990 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.991 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.992 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.992 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.992 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.992 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.992 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.992 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.992 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.992 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.993 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.994 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.994 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.994 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.994 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.994 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.994 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.994 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.994 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:21.995 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:21.996 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:21.997 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:21.998 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.998 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:21.998 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:21.998 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:21.998 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:21.998 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:21.998 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:21.998 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:21.999 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:21.999 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:21.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:21.999 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools": { 2024-04-25T16:52:21.999 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools", 2024-04-25T16:52:21.999 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:21.999 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:21.999 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.000 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.000 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.000 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.000 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.000 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.000 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.001 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.001 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.001 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.001 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T16:52:22.001 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T16:52:22.001 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.001 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.001 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.002 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "300", 2024-04-25T16:52:22.002 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.002 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.002 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.002 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.002 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.002 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.003 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.003 INFO:teuthology.orchestra.run.smithi098.stdout: "scrape_interval": { 2024-04-25T16:52:22.003 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "scrape_interval", 2024-04-25T16:52:22.003 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:22.003 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.003 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.004 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15.0", 2024-04-25T16:52:22.004 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.004 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.004 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.004 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.004 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.004 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.004 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.005 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.005 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:22.005 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:22.005 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.005 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.005 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.006 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "::", 2024-04-25T16:52:22.006 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.006 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.006 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.006 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T16:52:22.006 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.006 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.006 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "9283", 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.007 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.008 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.009 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.009 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.009 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.009 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.009 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.009 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.009 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.009 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.010 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.010 INFO:teuthology.orchestra.run.smithi098.stdout: "stale_cache_strategy": { 2024-04-25T16:52:22.010 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stale_cache_strategy", 2024-04-25T16:52:22.010 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.010 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.011 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.011 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "log", 2024-04-25T16:52:22.011 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.011 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.011 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.011 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.011 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.012 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.012 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.012 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.012 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_behaviour": { 2024-04-25T16:52:22.012 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_behaviour", 2024-04-25T16:52:22.012 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.013 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.013 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.013 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "default", 2024-04-25T16:52:22.013 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.013 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.013 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.013 INFO:teuthology.orchestra.run.smithi098.stdout: "default", 2024-04-25T16:52:22.014 INFO:teuthology.orchestra.run.smithi098.stdout: "error" 2024-04-25T16:52:22.014 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.014 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.014 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.014 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.015 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.015 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.015 INFO:teuthology.orchestra.run.smithi098.stdout: "standby_error_status_code": { 2024-04-25T16:52:22.015 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "standby_error_status_code", 2024-04-25T16:52:22.015 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.015 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.016 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.016 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "500", 2024-04-25T16:52:22.016 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "400", 2024-04-25T16:52:22.016 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "599", 2024-04-25T16:52:22.016 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.016 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.016 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.017 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.018 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.018 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.018 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.018 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.018 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.018 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.018 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.018 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.019 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.019 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.019 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.019 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.019 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.019 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.019 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.020 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.021 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.021 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.021 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.021 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.021 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.021 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.021 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.021 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.022 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.023 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.024 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.024 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.024 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.024 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.024 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.024 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.024 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.025 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.026 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.026 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.026 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.026 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_snap_create": { 2024-04-25T16:52:22.026 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_snap_create", 2024-04-25T16:52:22.026 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.026 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10", 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.027 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.028 INFO:teuthology.orchestra.run.smithi098.stdout: "mirror_snapshot_schedule": { 2024-04-25T16:52:22.028 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T16:52:22.028 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.028 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.028 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.028 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.028 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.029 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.029 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.029 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.029 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.029 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.029 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.029 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.030 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: "trash_purge_schedule": { 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "trash_purge_schedule", 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.031 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.032 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "restful", 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: "enable_auth": { 2024-04-25T16:52:22.033 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enable_auth", 2024-04-25T16:52:22.034 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.034 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.034 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.034 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:22.034 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.034 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.034 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.034 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: "key_file": { 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "key_file", 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.035 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.037 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.037 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.037 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.037 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.037 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.037 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.037 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.037 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.038 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.038 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.038 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.038 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.038 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.038 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.038 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.038 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.039 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.039 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.039 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.039 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.039 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.039 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.039 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.040 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.040 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.040 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.040 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.040 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.040 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.040 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.040 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.041 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.041 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.041 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.041 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.041 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.041 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.042 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.042 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.042 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.042 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.042 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.042 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.042 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.043 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.043 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.043 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.043 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.043 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.043 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.043 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.044 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.044 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.044 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.044 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.044 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.044 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.044 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.045 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.045 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.045 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.045 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.045 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.045 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.045 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.046 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.046 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.046 INFO:teuthology.orchestra.run.smithi098.stdout: "server_addr": { 2024-04-25T16:52:22.046 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_addr", 2024-04-25T16:52:22.046 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.046 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.047 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "server_port": { 2024-04-25T16:52:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "server_port", 2024-04-25T16:52:22.048 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.049 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.050 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.051 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rgw", 2024-04-25T16:52:22.052 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.053 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.054 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.055 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.056 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.057 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.058 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.059 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.060 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.061 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.062 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rook", 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.063 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.064 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.065 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.066 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.067 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.068 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.069 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.070 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.071 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.072 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.073 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_class": { 2024-04-25T16:52:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "storage_class", 2024-04-25T16:52:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.074 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "local", 2024-04-25T16:52:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T16:52:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.075 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "selftest", 2024-04-25T16:52:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.076 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.077 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.078 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.079 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.080 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.081 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.082 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.083 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.084 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.085 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "roption1": { 2024-04-25T16:52:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption1", 2024-04-25T16:52:22.086 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.087 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.088 INFO:teuthology.orchestra.run.smithi098.stdout: "roption2": { 2024-04-25T16:52:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "roption2", 2024-04-25T16:52:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "xyz", 2024-04-25T16:52:22.089 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.090 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption1": { 2024-04-25T16:52:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption1", 2024-04-25T16:52:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.091 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.092 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption2": { 2024-04-25T16:52:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption2", 2024-04-25T16:52:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.093 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption3": { 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption3", 2024-04-25T16:52:22.094 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "float", 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.095 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption4": { 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption4", 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.096 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption5": { 2024-04-25T16:52:22.097 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption5", 2024-04-25T16:52:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.098 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption6": { 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption6", 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.099 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.100 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "rwoption7": { 2024-04-25T16:52:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rwoption7", 2024-04-25T16:52:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "1", 2024-04-25T16:52:22.101 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "42", 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.102 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.103 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "testkey": { 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testkey", 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.104 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.105 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "testlkey": { 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testlkey", 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.106 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "testnewline": { 2024-04-25T16:52:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "testnewline", 2024-04-25T16:52:22.107 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.108 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snap_schedule", 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.109 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "allow_m_granularity": { 2024-04-25T16:52:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "allow_m_granularity", 2024-04-25T16:52:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.110 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "dump_on_update": { 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "dump_on_update", 2024-04-25T16:52:22.111 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "dump database to debug log on update", 2024-04-25T16:52:22.112 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.113 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.114 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.115 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.116 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.117 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.118 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.119 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.120 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.121 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.122 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.123 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.124 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.125 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.126 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.127 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "stats", 2024-04-25T16:52:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.128 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.129 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.130 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.131 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.132 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.133 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.134 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.135 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.136 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.137 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.138 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.139 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "status", 2024-04-25T16:52:22.140 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.141 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.142 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.143 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.144 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.145 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.146 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.147 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.148 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.149 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.150 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.151 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.152 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.153 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telegraf", 2024-04-25T16:52:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.154 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "address": { 2024-04-25T16:52:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "address", 2024-04-25T16:52:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.155 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T16:52:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.156 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:22.157 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "15", 2024-04-25T16:52:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.158 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.159 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.160 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.161 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.162 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.163 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.164 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.165 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.165 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.165 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.165 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.165 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.165 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.165 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.166 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.166 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.166 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.166 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.166 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.166 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.167 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.167 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.167 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.167 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.167 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.167 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.167 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.168 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.168 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.168 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.168 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.168 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.168 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.169 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.169 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.169 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.169 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.169 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.169 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.169 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.170 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.170 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.170 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.170 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.170 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.170 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.170 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.171 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.171 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.171 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.171 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.171 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.171 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.172 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.173 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.173 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.173 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "telemetry", 2024-04-25T16:52:22.173 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.173 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.173 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.173 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_basic": { 2024-04-25T16:52:22.174 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_basic", 2024-04-25T16:52:22.174 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.174 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.174 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.174 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:22.174 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.175 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.175 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.175 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T16:52:22.175 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.175 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.175 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.175 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.176 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_crash": { 2024-04-25T16:52:22.176 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_crash", 2024-04-25T16:52:22.176 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.176 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.176 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.176 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:22.176 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.176 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.177 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.177 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T16:52:22.177 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.177 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.177 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.177 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.177 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_device": { 2024-04-25T16:52:22.178 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_device", 2024-04-25T16:52:22.178 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.178 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.178 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.178 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:22.178 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.178 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.179 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.179 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T16:52:22.179 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.179 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.179 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.179 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.179 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_ident": { 2024-04-25T16:52:22.179 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_ident", 2024-04-25T16:52:22.180 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.180 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.180 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.180 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.180 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.180 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.180 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.181 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T16:52:22.181 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.181 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.181 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.181 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.181 INFO:teuthology.orchestra.run.smithi098.stdout: "channel_perf": { 2024-04-25T16:52:22.181 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "channel_perf", 2024-04-25T16:52:22.182 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.182 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.182 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.182 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.182 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.182 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.182 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.183 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T16:52:22.183 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.183 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.183 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.183 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.183 INFO:teuthology.orchestra.run.smithi098.stdout: "contact": { 2024-04-25T16:52:22.183 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "contact", 2024-04-25T16:52:22.183 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.184 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.185 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.185 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.185 INFO:teuthology.orchestra.run.smithi098.stdout: "description": { 2024-04-25T16:52:22.185 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "description", 2024-04-25T16:52:22.185 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.185 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.185 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.185 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.186 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.186 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.186 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.186 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.186 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.186 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.186 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.186 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.187 INFO:teuthology.orchestra.run.smithi098.stdout: "device_url": { 2024-04-25T16:52:22.187 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "device_url", 2024-04-25T16:52:22.187 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.187 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.187 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.187 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T16:52:22.188 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.188 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.188 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.188 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.188 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.188 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.188 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.189 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.189 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": { 2024-04-25T16:52:22.189 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "enabled", 2024-04-25T16:52:22.189 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.189 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.190 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.190 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.190 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.190 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.190 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.190 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.190 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.191 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.191 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.191 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.191 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:22.191 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:22.191 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.191 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.192 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.192 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "24", 2024-04-25T16:52:22.192 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "8", 2024-04-25T16:52:22.192 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.192 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.192 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.193 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.193 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.193 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.193 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.193 INFO:teuthology.orchestra.run.smithi098.stdout: "last_opt_revision": { 2024-04-25T16:52:22.194 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "last_opt_revision", 2024-04-25T16:52:22.194 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.194 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.194 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.194 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "1", 2024-04-25T16:52:22.194 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.194 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.195 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.195 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.195 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.195 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.195 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.196 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.196 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard": { 2024-04-25T16:52:22.196 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard", 2024-04-25T16:52:22.196 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.196 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.196 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.197 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.197 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.197 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.197 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.197 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.197 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.198 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.198 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.198 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.198 INFO:teuthology.orchestra.run.smithi098.stdout: "leaderboard_description": { 2024-04-25T16:52:22.198 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "leaderboard_description", 2024-04-25T16:52:22.199 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.199 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.199 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.199 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.199 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.200 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.200 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.200 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.200 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.200 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.200 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.200 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.201 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.201 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.201 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.201 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.201 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.201 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.201 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.202 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.202 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.202 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.202 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.202 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.202 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.203 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.203 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.203 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.203 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.203 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.203 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.203 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.203 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.204 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.204 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.204 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.204 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.204 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.204 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.204 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.205 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.205 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.205 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.205 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.205 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.205 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.205 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.206 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.206 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.206 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.206 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.206 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.206 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.206 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.207 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.207 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.207 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.207 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.207 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.207 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.207 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.208 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.208 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.208 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.208 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.208 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.208 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.208 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.208 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.209 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.209 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.209 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.209 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.209 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.209 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.209 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.209 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.210 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.210 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.210 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.210 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.210 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.210 INFO:teuthology.orchestra.run.smithi098.stdout: "organization": { 2024-04-25T16:52:22.210 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "organization", 2024-04-25T16:52:22.210 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.211 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.211 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.211 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.211 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.211 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.211 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.211 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.211 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.212 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.212 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.212 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.212 INFO:teuthology.orchestra.run.smithi098.stdout: "proxy": { 2024-04-25T16:52:22.212 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "proxy", 2024-04-25T16:52:22.212 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.212 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.213 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.214 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.214 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.214 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.214 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.214 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.214 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.214 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.214 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.215 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "url": { 2024-04-25T16:52:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "url", 2024-04-25T16:52:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T16:52:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.216 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.217 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.218 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.218 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "test_orchestrator", 2024-04-25T16:52:22.218 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.218 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.218 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.218 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.218 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.219 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.219 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.219 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.219 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.219 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.219 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.219 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.219 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.220 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.221 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.221 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.221 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.221 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.221 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.222 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.222 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.222 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.222 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.222 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.222 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.223 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.223 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.223 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.223 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.223 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.224 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.224 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.224 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.224 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.224 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.224 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.225 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.225 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.225 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.225 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.225 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.225 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.225 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.226 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.226 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.226 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.226 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.226 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.226 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.227 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.227 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.227 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.227 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.227 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.227 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.228 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.228 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.228 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.228 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.228 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.228 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.229 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.229 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.229 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.229 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.229 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.230 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.230 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.230 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.230 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.230 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.230 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.231 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.231 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.231 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.231 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.231 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.231 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.232 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.232 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.232 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.232 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-04-25T16:52:22.233 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.233 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.233 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.233 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.233 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.233 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.233 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.234 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.234 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.234 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.234 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.234 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.234 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.234 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.234 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.235 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.235 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.235 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.235 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.235 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.235 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.235 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.235 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.236 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.236 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.236 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.236 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.236 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.237 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.237 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.237 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.237 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.237 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.237 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.237 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.237 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.238 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.238 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.238 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.238 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.238 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.238 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.239 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.239 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.239 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.239 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.239 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.239 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.239 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.239 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.240 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.240 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.240 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.240 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.240 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.240 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.240 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.241 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.242 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.242 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.242 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.242 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.242 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.242 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.242 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max_concurrent_clones": { 2024-04-25T16:52:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "max_concurrent_clones", 2024-04-25T16:52:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "4", 2024-04-25T16:52:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.243 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.244 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.244 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T16:52:22.244 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.244 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.244 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.244 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.244 INFO:teuthology.orchestra.run.smithi098.stdout: "periodic_async_work": { 2024-04-25T16:52:22.245 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "periodic_async_work", 2024-04-25T16:52:22.245 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.245 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.245 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.245 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.245 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.245 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.245 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.246 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Periodically check for async work", 2024-04-25T16:52:22.246 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.246 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.246 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.246 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.246 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_delay": { 2024-04-25T16:52:22.246 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_delay", 2024-04-25T16:52:22.247 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.247 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.247 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.247 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.247 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.247 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.247 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.247 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T16:52:22.248 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.248 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.248 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.248 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.248 INFO:teuthology.orchestra.run.smithi098.stdout: "snapshot_clone_no_wait": { 2024-04-25T16:52:22.248 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T16:52:22.248 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.249 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.249 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.249 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "True", 2024-04-25T16:52:22.249 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.249 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.249 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.249 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T16:52:22.249 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.250 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.250 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.250 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.250 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.250 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.250 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.250 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.251 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.252 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.252 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.252 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.252 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix", 2024-04-25T16:52:22.252 INFO:teuthology.orchestra.run.smithi098.stdout: "can_run": true, 2024-04-25T16:52:22.252 INFO:teuthology.orchestra.run.smithi098.stdout: "error_string": "", 2024-04-25T16:52:22.253 INFO:teuthology.orchestra.run.smithi098.stdout: "module_options": { 2024-04-25T16:52:22.253 INFO:teuthology.orchestra.run.smithi098.stdout: "discovery_interval": { 2024-04-25T16:52:22.253 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "discovery_interval", 2024-04-25T16:52:22.253 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "uint", 2024-04-25T16:52:22.253 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.253 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.253 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "100", 2024-04-25T16:52:22.254 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.254 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.254 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.254 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.254 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.254 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.254 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: "identifier": { 2024-04-25T16:52:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "identifier", 2024-04-25T16:52:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.255 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.256 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.256 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.256 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.256 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.256 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.256 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.256 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.257 INFO:teuthology.orchestra.run.smithi098.stdout: "interval": { 2024-04-25T16:52:22.257 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "interval", 2024-04-25T16:52:22.257 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "secs", 2024-04-25T16:52:22.257 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.257 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.257 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "60", 2024-04-25T16:52:22.257 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.257 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.258 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.258 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.258 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.258 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.258 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.258 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.258 INFO:teuthology.orchestra.run.smithi098.stdout: "log_level": { 2024-04-25T16:52:22.258 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_level", 2024-04-25T16:52:22.259 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.259 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.259 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.259 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.260 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.260 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.260 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.260 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.260 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.260 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.260 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.260 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.261 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.261 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.261 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.261 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.261 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.262 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.262 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.262 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster": { 2024-04-25T16:52:22.262 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster", 2024-04-25T16:52:22.262 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.262 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.262 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.263 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.263 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.263 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.263 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.263 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.264 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.264 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.264 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.264 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.264 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_cluster_level": { 2024-04-25T16:52:22.264 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_cluster_level", 2024-04-25T16:52:22.264 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.265 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.265 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.265 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "info", 2024-04-25T16:52:22.265 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.265 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.265 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [ 2024-04-25T16:52:22.266 INFO:teuthology.orchestra.run.smithi098.stdout: "", 2024-04-25T16:52:22.266 INFO:teuthology.orchestra.run.smithi098.stdout: "critical", 2024-04-25T16:52:22.266 INFO:teuthology.orchestra.run.smithi098.stdout: "debug", 2024-04-25T16:52:22.266 INFO:teuthology.orchestra.run.smithi098.stdout: "error", 2024-04-25T16:52:22.266 INFO:teuthology.orchestra.run.smithi098.stdout: "info", 2024-04-25T16:52:22.266 INFO:teuthology.orchestra.run.smithi098.stdout: "warning" 2024-04-25T16:52:22.266 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.267 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.267 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.267 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.267 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.267 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.267 INFO:teuthology.orchestra.run.smithi098.stdout: "log_to_file": { 2024-04-25T16:52:22.268 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "log_to_file", 2024-04-25T16:52:22.268 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "bool", 2024-04-25T16:52:22.268 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.268 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.268 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "False", 2024-04-25T16:52:22.268 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.269 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.269 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.269 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.269 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.269 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.270 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.270 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.270 INFO:teuthology.orchestra.run.smithi098.stdout: "sqlite3_killpoint": { 2024-04-25T16:52:22.270 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:52:22.271 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.271 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "dev", 2024-04-25T16:52:22.271 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 1, 2024-04-25T16:52:22.271 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "0", 2024-04-25T16:52:22.271 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.271 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.271 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.272 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.272 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.272 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.272 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.272 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.272 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_host": { 2024-04-25T16:52:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_host", 2024-04-25T16:52:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "", 2024-04-25T16:52:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.273 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.274 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.274 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.274 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.274 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.274 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.274 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.274 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_port": { 2024-04-25T16:52:22.275 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_port", 2024-04-25T16:52:22.275 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "int", 2024-04-25T16:52:22.275 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.275 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.275 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "10051", 2024-04-25T16:52:22.275 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.275 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.275 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.276 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.276 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.276 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.276 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.276 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.276 INFO:teuthology.orchestra.run.smithi098.stdout: "zabbix_sender": { 2024-04-25T16:52:22.276 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "zabbix_sender", 2024-04-25T16:52:22.276 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "str", 2024-04-25T16:52:22.277 INFO:teuthology.orchestra.run.smithi098.stdout: "level": "advanced", 2024-04-25T16:52:22.277 INFO:teuthology.orchestra.run.smithi098.stdout: "flags": 0, 2024-04-25T16:52:22.277 INFO:teuthology.orchestra.run.smithi098.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T16:52:22.277 INFO:teuthology.orchestra.run.smithi098.stdout: "min": "", 2024-04-25T16:52:22.277 INFO:teuthology.orchestra.run.smithi098.stdout: "max": "", 2024-04-25T16:52:22.277 INFO:teuthology.orchestra.run.smithi098.stdout: "enum_allowed": [], 2024-04-25T16:52:22.277 INFO:teuthology.orchestra.run.smithi098.stdout: "desc": "", 2024-04-25T16:52:22.278 INFO:teuthology.orchestra.run.smithi098.stdout: "long_desc": "", 2024-04-25T16:52:22.278 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-04-25T16:52:22.278 INFO:teuthology.orchestra.run.smithi098.stdout: "see_also": [] 2024-04-25T16:52:22.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.278 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.278 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.279 INFO:teuthology.orchestra.run.smithi098.stdout: "services": { 2024-04-25T16:52:22.279 INFO:teuthology.orchestra.run.smithi098.stdout: "dashboard": "https://172.21.15.98:8443/" 2024-04-25T16:52:22.279 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.279 INFO:teuthology.orchestra.run.smithi098.stdout: "always_on_modules": { 2024-04-25T16:52:22.279 INFO:teuthology.orchestra.run.smithi098.stdout: "octopus": [ 2024-04-25T16:52:22.279 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T16:52:22.279 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T16:52:22.279 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T16:52:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T16:52:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T16:52:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T16:52:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T16:52:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T16:52:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T16:52:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T16:52:22.280 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.281 INFO:teuthology.orchestra.run.smithi098.stdout: "pacific": [ 2024-04-25T16:52:22.281 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T16:52:22.281 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T16:52:22.281 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T16:52:22.281 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T16:52:22.281 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T16:52:22.281 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T16:52:22.281 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T16:52:22.282 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T16:52:22.282 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T16:52:22.282 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T16:52:22.282 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.282 INFO:teuthology.orchestra.run.smithi098.stdout: "quincy": [ 2024-04-25T16:52:22.282 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T16:52:22.282 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T16:52:22.283 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T16:52:22.283 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T16:52:22.283 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T16:52:22.283 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T16:52:22.283 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T16:52:22.283 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T16:52:22.283 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T16:52:22.284 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T16:52:22.284 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.284 INFO:teuthology.orchestra.run.smithi098.stdout: "reef": [ 2024-04-25T16:52:22.284 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T16:52:22.284 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T16:52:22.284 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T16:52:22.284 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T16:52:22.285 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T16:52:22.285 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T16:52:22.285 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T16:52:22.285 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T16:52:22.285 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T16:52:22.285 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T16:52:22.285 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-04-25T16:52:22.285 INFO:teuthology.orchestra.run.smithi098.stdout: "squid": [ 2024-04-25T16:52:22.286 INFO:teuthology.orchestra.run.smithi098.stdout: "balancer", 2024-04-25T16:52:22.286 INFO:teuthology.orchestra.run.smithi098.stdout: "crash", 2024-04-25T16:52:22.286 INFO:teuthology.orchestra.run.smithi098.stdout: "devicehealth", 2024-04-25T16:52:22.286 INFO:teuthology.orchestra.run.smithi098.stdout: "orchestrator", 2024-04-25T16:52:22.286 INFO:teuthology.orchestra.run.smithi098.stdout: "pg_autoscaler", 2024-04-25T16:52:22.286 INFO:teuthology.orchestra.run.smithi098.stdout: "progress", 2024-04-25T16:52:22.286 INFO:teuthology.orchestra.run.smithi098.stdout: "rbd_support", 2024-04-25T16:52:22.286 INFO:teuthology.orchestra.run.smithi098.stdout: "status", 2024-04-25T16:52:22.287 INFO:teuthology.orchestra.run.smithi098.stdout: "telemetry", 2024-04-25T16:52:22.287 INFO:teuthology.orchestra.run.smithi098.stdout: "volumes" 2024-04-25T16:52:22.287 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:22.287 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.287 INFO:teuthology.orchestra.run.smithi098.stdout: "last_failure_osd_epoch": 3, 2024-04-25T16:52:22.287 INFO:teuthology.orchestra.run.smithi098.stdout: "active_clients": [ 2024-04-25T16:52:22.287 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.288 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "devicehealth", 2024-04-25T16:52:22.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T16:52:22.288 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.288 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T16:52:22.288 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T16:52:22.288 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 1611661772 2024-04-25T16:52:22.288 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.289 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:22.289 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.289 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "libcephsqlite", 2024-04-25T16:52:22.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T16:52:22.289 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.289 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T16:52:22.289 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T16:52:22.290 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 3811231551 2024-04-25T16:52:22.290 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.290 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:22.290 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.290 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "rbd_support", 2024-04-25T16:52:22.290 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T16:52:22.290 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.291 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T16:52:22.291 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T16:52:22.291 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 79813565 2024-04-25T16:52:22.291 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.291 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:22.291 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:22.291 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.291 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "volumes", 2024-04-25T16:52:22.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addrvec": [ 2024-04-25T16:52:22.292 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-04-25T16:52:22.292 INFO:teuthology.orchestra.run.smithi098.stdout: "type": "v2", 2024-04-25T16:52:22.292 INFO:teuthology.orchestra.run.smithi098.stdout: "addr": "172.21.15.98:0", 2024-04-25T16:52:22.292 INFO:teuthology.orchestra.run.smithi098.stdout: "nonce": 2544586759 2024-04-25T16:52:22.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.292 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:22.292 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:22.293 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-04-25T16:52:22.293 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:52:22.294 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:20 smithi173 ceph-mon[32556]: pgmap v971: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 118 KiB/s wr, 56 op/s 2024-04-25T16:52:22.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:20 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2229712887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:52:22.295 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:20 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3428876341' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T16:52:22.295 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:20 smithi203 ceph-mon[33115]: pgmap v971: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 118 KiB/s wr, 56 op/s 2024-04-25T16:52:22.295 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:20 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2229712887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:52:22.295 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:20 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3428876341' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T16:52:22.296 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffaa40afb70 msgr2=0x7ffaa40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:22.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffaa40afb70 0x7ffaa40b2080 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8005fd0 tx=0x7ffaa8005f40 comp rx=0 tx=0).stop 2024-04-25T16:52:22.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 msgr2=0x7ffab8198fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:22.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 0x7ffab8198fe0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ffab4031460 tx=0x7ffab4004270 comp rx=0 tx=0).stop 2024-04-25T16:52:22.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 shutdown_connections 2024-04-25T16:52:22.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7ffaa40afb70 0x7ffaa40b2080 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:22.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffab8104480 0x7ffab81a04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:22.297 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffab8103960 0x7ffab8199520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:22.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 --2- 172.21.15.98:0/3428876341 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7ffab8102f80 0x7ffab8198fe0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:22.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 >> 172.21.15.98:0/3428876341 conn(0x7ffab80fd360 msgr2=0x7ffab8104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:22.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 shutdown_connections 2024-04-25T16:52:22.298 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:19.573+0000 7ffabff0d640 1 -- 172.21.15.98:0/3428876341 wait complete. 2024-04-25T16:52:22.307 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T16:52:22.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T16:52:22.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:20 smithi098 ceph-mon[29162]: pgmap v971: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 118 KiB/s wr, 56 op/s 2024-04-25T16:52:22.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:20 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2229712887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:52:22.308 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:20 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3428876341' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T16:52:22.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:22 smithi203 ceph-mon[33115]: pgmap v972: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 95 KiB/s wr, 53 op/s 2024-04-25T16:52:22.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:22 smithi098 ceph-mon[29162]: pgmap v972: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 95 KiB/s wr, 53 op/s 2024-04-25T16:52:22.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:22 smithi173 ceph-mon[32556]: pgmap v972: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 95 KiB/s wr, 53 op/s 2024-04-25T16:52:22.608 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:52:24.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:24 smithi098 ceph-mon[29162]: pgmap v973: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 72 KiB/s wr, 48 op/s 2024-04-25T16:52:24.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:24 smithi203 ceph-mon[33115]: pgmap v973: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 72 KiB/s wr, 48 op/s 2024-04-25T16:52:24.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:24 smithi173 ceph-mon[32556]: pgmap v973: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 72 KiB/s wr, 48 op/s 2024-04-25T16:52:24.989 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:52:24.989 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:24.989 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 2313, 2024-04-25T16:52:24.989 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2315, 2024-04-25T16:52:24.989 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 4885883, 2024-04-25T16:52:24.989 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 735535, 2024-04-25T16:52:24.989 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-04-25T16:52:24.990 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-04-25T16:52:24.990 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.221000032, 2024-04-25T16:52:24.990 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.071125244, 2024-04-25T16:52:24.990 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.057993760, 2024-04-25T16:52:24.990 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.074181926, 2024-04-25T16:52:24.990 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:24.990 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2313, 2024-04-25T16:52:24.990 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.070864444, 2024-04-25T16:52:24.991 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000030637 2024-04-25T16:52:24.991 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.991 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:24.991 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:24.991 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:24.992 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:24.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.992 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:24.992 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:24.992 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.992 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:24.992 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1783, 2024-04-25T16:52:24.993 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1631, 2024-04-25T16:52:24.993 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 9404550, 2024-04-25T16:52:24.993 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 621280, 2024-04-25T16:52:24.993 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T16:52:24.993 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 3, 2024-04-25T16:52:24.993 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.169132445, 2024-04-25T16:52:24.993 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.052637785, 2024-04-25T16:52:24.994 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.071874754, 2024-04-25T16:52:24.994 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.043932139, 2024-04-25T16:52:24.994 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:24.994 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1631, 2024-04-25T16:52:24.994 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.049431314, 2024-04-25T16:52:24.994 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000030307 2024-04-25T16:52:24.994 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 1771968, 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 226256 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.995 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:24.996 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 420, 2024-04-25T16:52:24.996 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 691, 2024-04-25T16:52:24.996 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 4302482, 2024-04-25T16:52:24.996 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 778492, 2024-04-25T16:52:24.996 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 7, 2024-04-25T16:52:24.996 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 4, 2024-04-25T16:52:24.996 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.052988682, 2024-04-25T16:52:24.996 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.026810723, 2024-04-25T16:52:24.997 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.012066563, 2024-04-25T16:52:24.997 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.009495455, 2024-04-25T16:52:24.997 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:24.997 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 689, 2024-04-25T16:52:24.997 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.022412940, 2024-04-25T16:52:24.997 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000032529 2024-04-25T16:52:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-25T16:52:24.998 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 645568 2024-04-25T16:52:24.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.999 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-04-25T16:52:24.999 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-04-25T16:52:24.999 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-04-25T16:52:24.999 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:24.999 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-04-25T16:52:25.000 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-04-25T16:52:25.000 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-04-25T16:52:25.000 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 6582, 2024-04-25T16:52:25.000 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.195818843, 2024-04-25T16:52:25.000 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000029750 2024-04-25T16:52:25.000 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.000 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.001 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:25.001 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-04-25T16:52:25.001 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-04-25T16:52:25.001 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.001 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.001 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.001 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.002 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.002 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-04-25T16:52:25.002 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 0, 2024-04-25T16:52:25.002 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 0, 2024-04-25T16:52:25.003 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-04-25T16:52:25.003 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.003 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.003 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.003 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.003 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-04-25T16:52:25.003 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 0, 2024-04-25T16:52:25.004 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 0, 2024-04-25T16:52:25.004 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 0, 2024-04-25T16:52:25.004 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-04-25T16:52:25.004 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-04-25T16:52:25.004 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-04-25T16:52:25.004 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 4678, 2024-04-25T16:52:25.004 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 194, 2024-04-25T16:52:25.004 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 4484, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 4400, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 0, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 0, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 160, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 0, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 0, 2024-04-25T16:52:25.005 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 0, 2024-04-25T16:52:25.006 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-04-25T16:52:25.006 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:25.006 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:25.006 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-04-25T16:52:25.006 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-04-25T16:52:25.006 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-04-25T16:52:25.006 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-04-25T16:52:25.006 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-04-25T16:52:25.007 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-04-25T16:52:25.007 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-04-25T16:52:25.007 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:25.007 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:52:25.007 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:25.007 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 0, 2024-04-25T16:52:25.007 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 0, 2024-04-25T16:52:25.008 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-04-25T16:52:25.008 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:25.008 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:25.008 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:25.008 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:25.008 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:25.008 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:25.008 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:25.009 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:25.009 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:25.009 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:52:25.009 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:52:25.009 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:52:25.009 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:25.009 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:25.010 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 0, 2024-04-25T16:52:25.010 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:52:25.010 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 0, 2024-04-25T16:52:25.010 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 0 2024-04-25T16:52:25.010 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.010 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-04-25T16:52:25.010 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-04-25T16:52:25.010 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:25.011 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-04-25T16:52:25.011 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-04-25T16:52:25.011 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-04-25T16:52:25.011 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-04-25T16:52:25.011 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:25.011 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:25.011 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 0, 2024-04-25T16:52:25.011 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 0, 2024-04-25T16:52:25.012 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:25.012 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-04-25T16:52:25.012 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:25.012 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:25.012 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:25.012 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-04-25T16:52:25.013 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-04-25T16:52:25.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:25.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:25.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:25.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-04-25T16:52:25.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-04-25T16:52:25.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:52:25.013 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-04-25T16:52:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-04-25T16:52:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-04-25T16:52:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 0, 2024-04-25T16:52:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-04-25T16:52:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-04-25T16:52:25.014 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 7440, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 10, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 2, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-04-25T16:52:25.015 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-04-25T16:52:25.016 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-04-25T16:52:25.016 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 19894996, 2024-04-25T16:52:25.016 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 19894422, 2024-04-25T16:52:25.016 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-04-25T16:52:25.016 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.016 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 7440 2024-04-25T16:52:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-04-25T16:52:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 4679, 2024-04-25T16:52:25.017 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 5622, 2024-04-25T16:52:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 943, 2024-04-25T16:52:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 440, 2024-04-25T16:52:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 485, 2024-04-25T16:52:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 45, 2024-04-25T16:52:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 4678, 2024-04-25T16:52:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 5619, 2024-04-25T16:52:25.018 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 941, 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 0, 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 0, 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 0, 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 136200, 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 215440 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 0, 2024-04-25T16:52:25.019 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 0, 2024-04-25T16:52:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 0, 2024-04-25T16:52:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-04-25T16:52:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 0, 2024-04-25T16:52:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-04-25T16:52:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.020 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.021 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.022 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-04-25T16:52:25.022 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.022 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.022 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.022 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.022 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-04-25T16:52:25.023 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.023 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.023 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.023 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.023 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:25.023 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.023 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-04-25T16:52:25.024 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.025 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.025 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.025 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.025 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:25.025 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.025 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.026 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.026 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:25.026 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.026 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.026 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.026 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.026 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-04-25T16:52:25.027 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.027 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.027 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.027 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.028 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-04-25T16:52:25.028 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.028 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.028 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.028 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.028 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-04-25T16:52:25.029 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.029 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.029 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.029 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.029 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-04-25T16:52:25.029 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.029 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.029 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.030 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-04-25T16:52:25.030 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.030 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.030 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.030 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.030 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-04-25T16:52:25.030 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.031 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.031 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.031 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.031 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-04-25T16:52:25.031 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.031 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.031 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.032 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.033 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.033 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.033 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-04-25T16:52:25.033 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.033 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.033 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.034 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:25.034 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.034 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.034 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.034 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.034 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:25.034 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.035 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.035 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.035 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-04-25T16:52:25.035 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.035 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.035 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.035 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.036 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.037 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-04-25T16:52:25.037 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.037 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.037 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.037 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.037 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-04-25T16:52:25.037 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.038 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.038 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.038 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.038 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-04-25T16:52:25.038 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.038 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.038 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.039 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-04-25T16:52:25.039 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.039 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.039 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.039 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.039 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:25.039 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 1, 2024-04-25T16:52:25.040 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 2, 2024-04-25T16:52:25.041 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 1, 2024-04-25T16:52:25.041 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 2, 2024-04-25T16:52:25.041 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-04-25T16:52:25.041 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 0, 2024-04-25T16:52:25.041 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 0, 2024-04-25T16:52:25.041 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 3430, 2024-04-25T16:52:25.041 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:25.041 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.042 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-04-25T16:52:25.042 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:25.042 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:25.042 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:25.042 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:25.042 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:25.042 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:25.042 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:25.043 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:25.043 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:25.043 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:25.043 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:25.043 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:25.043 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:25.043 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:25.044 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:25.044 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:25.044 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:25.044 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:25.044 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:25.044 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:25.045 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:25.045 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:25.045 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:25.045 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:25.045 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:25.045 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:25.045 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:25.046 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:25.046 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:25.046 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:25.046 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:25.046 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-04-25T16:52:25.046 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:25.046 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:25.046 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:25.047 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:25.047 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 207895, 2024-04-25T16:52:25.047 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 64, 2024-04-25T16:52:25.047 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:25.047 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:25.047 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-04-25T16:52:25.047 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-04-25T16:52:25.047 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:25.048 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:25.048 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:25.048 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:25.048 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:25.048 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 276, 2024-04-25T16:52:25.048 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:25.048 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:25.048 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 13827414, 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 97085, 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.049 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-04-25T16:52:25.050 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-04-25T16:52:25.050 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-04-25T16:52:25.050 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 3973, 2024-04-25T16:52:25.050 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 21, 2024-04-25T16:52:25.050 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 3, 2024-04-25T16:52:25.050 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 3973, 2024-04-25T16:52:25.050 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-04-25T16:52:25.050 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3973, 2024-04-25T16:52:25.051 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 6.890907773, 2024-04-25T16:52:25.051 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001734434 2024-04-25T16:52:25.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.051 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-04-25T16:52:25.051 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-04-25T16:52:25.051 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 3973, 2024-04-25T16:52:25.051 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 3974 2024-04-25T16:52:25.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.052 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 3973, 2024-04-25T16:52:25.052 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 3973, 2024-04-25T16:52:25.052 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 0, 2024-04-25T16:52:25.052 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-04-25T16:52:25.052 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-04-25T16:52:25.052 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 2491, 2024-04-25T16:52:25.052 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 1480, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 0, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 0, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 0, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 0, 2024-04-25T16:52:25.053 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-04-25T16:52:25.054 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:25.054 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:25.054 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 1, 2024-04-25T16:52:25.054 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-04-25T16:52:25.054 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:25.055 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-04-25T16:52:25.056 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-04-25T16:52:25.057 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-04-25T16:52:25.057 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-04-25T16:52:25.057 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-04-25T16:52:25.057 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-04-25T16:52:25.057 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-04-25T16:52:25.057 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-04-25T16:52:25.057 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-04-25T16:52:25.057 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 106, 2024-04-25T16:52:25.058 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-04-25T16:52:25.058 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 25, 2024-04-25T16:52:25.058 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 7, 2024-04-25T16:52:25.058 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 8, 2024-04-25T16:52:25.058 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 1, 2024-04-25T16:52:25.058 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-04-25T16:52:25.058 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 0, 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 2, 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 0 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-04-25T16:52:25.059 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.060 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-04-25T16:52:25.060 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-04-25T16:52:25.060 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.060 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-04-25T16:52:25.060 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:25.060 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:52:25.060 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-04-25T16:52:25.060 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 0, 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 0, 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 0, 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:25.061 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 4512, 2024-04-25T16:52:25.062 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 18157297, 2024-04-25T16:52:25.062 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:25.062 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 4512, 2024-04-25T16:52:25.062 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T16:52:25.062 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T16:52:25.062 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 4512, 2024-04-25T16:52:25.062 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 18157297, 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:25.063 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T16:52:25.064 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:25.064 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-04-25T16:52:25.064 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-04-25T16:52:25.064 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:25.064 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:25.064 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 3973, 2024-04-25T16:52:25.064 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 15703843, 2024-04-25T16:52:25.065 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 3973, 2024-04-25T16:52:25.065 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 15703843, 2024-04-25T16:52:25.065 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:25.065 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.065 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.065 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.065 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.066 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.066 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:25.066 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:25.066 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-04-25T16:52:25.066 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:25.066 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-04-25T16:52:25.066 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-04-25T16:52:25.066 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 3973, 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 3973, 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 3973, 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 3973, 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.067 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-04-25T16:52:25.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T16:52:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T16:52:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-04-25T16:52:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-04-25T16:52:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.069 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.070 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.070 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.070 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:25.070 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x560815d42750": { 2024-04-25T16:52:25.070 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:25.070 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-04-25T16:52:25.070 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:25.070 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-04-25T16:52:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-04-25T16:52:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T16:52:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T16:52:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 0, 2024-04-25T16:52:25.071 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 0, 2024-04-25T16:52:25.072 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:25.072 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:25.072 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:25.072 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:25.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.072 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:25.072 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:52:25.933 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-25T16:52:25.933 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-25T16:52:25.933 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-04-25T16:52:25.933 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T16:52:25.934 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T16:52:26.185 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:52:26.250 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:26 smithi098 ceph-mon[29162]: pgmap v974: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 70 KiB/s wr, 47 op/s 2024-04-25T16:52:26.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:26 smithi203 ceph-mon[33115]: pgmap v974: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 70 KiB/s wr, 47 op/s 2024-04-25T16:52:26.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:26 smithi173 ceph-mon[32556]: pgmap v974: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 70 KiB/s wr, 47 op/s 2024-04-25T16:52:28.048 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:52:28.049 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:28.049 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-04-25T16:52:28.049 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-04-25T16:52:28.049 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-04-25T16:52:28.049 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-04-25T16:52:28.049 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T16:52:28.049 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-04-25T16:52:28.049 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001131628, 2024-04-25T16:52:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T16:52:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000223822, 2024-04-25T16:52:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T16:52:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:28.050 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.051 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:28.052 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 368, 2024-04-25T16:52:28.052 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 363, 2024-04-25T16:52:28.052 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 844704, 2024-04-25T16:52:28.052 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 224416, 2024-04-25T16:52:28.052 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-04-25T16:52:28.052 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-04-25T16:52:28.052 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.052212103, 2024-04-25T16:52:28.052 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.020280931, 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.026218219, 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.007485390, 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 362, 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.012753857, 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000035231 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:28.053 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:28.054 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:28.054 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:28.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.054 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 844704, 2024-04-25T16:52:28.054 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 224416 2024-04-25T16:52:28.054 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.054 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:28.054 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 6, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 280, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3280, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 206560, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.019953858, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.015728173, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.001994279, 2024-04-25T16:52:28.055 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 279, 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.012269714, 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000043977 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:28.056 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:28.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.057 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3280, 2024-04-25T16:52:28.057 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 206560 2024-04-25T16:52:28.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.057 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-04-25T16:52:28.057 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-04-25T16:52:28.057 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-04-25T16:52:28.057 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.058 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-04-25T16:52:28.058 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:28.058 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:28.058 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:28.058 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:28.058 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:28.058 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:28.058 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:28.059 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:28.060 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:28.060 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:28.060 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:28.060 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:28.060 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:28.060 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:28.060 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:28.060 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-04-25T16:52:28.061 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 148704, 2024-04-25T16:52:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 29, 2024-04-25T16:52:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:28.062 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-04-25T16:52:28.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-04-25T16:52:28.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:28.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:28.065 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:28.066 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:28.066 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-04-25T16:52:28.066 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-04-25T16:52:28.066 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:28.066 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:28.066 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:28.066 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:28.067 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:28.068 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T16:52:28.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:28.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 370, 2024-04-25T16:52:28.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 792689, 2024-04-25T16:52:28.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:28.068 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 370, 2024-04-25T16:52:28.068 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T16:52:28.068 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 370, 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 792689, 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:28.069 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:52:28.413 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:28 smithi098 ceph-mon[29162]: pgmap v975: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T16:52:28.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:28 smithi203 ceph-mon[33115]: pgmap v975: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T16:52:28.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:28 smithi173 ceph-mon[32556]: pgmap v975: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-04-25T16:52:28.808 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T16:52:28.809 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T16:52:28.809 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-25T16:52:28.809 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T16:52:28.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T16:52:29.053 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:52:30.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:30 smithi098 ceph-mon[29162]: pgmap v976: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-04-25T16:52:30.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:30 smithi203 ceph-mon[33115]: pgmap v976: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-04-25T16:52:30.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:30 smithi173 ceph-mon[32556]: pgmap v976: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 16 KiB/s wr, 17 op/s 2024-04-25T16:52:30.818 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:52:30.819 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:30.819 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 31483, 2024-04-25T16:52:30.819 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 31680, 2024-04-25T16:52:30.819 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 18040841, 2024-04-25T16:52:30.819 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 5694885, 2024-04-25T16:52:30.819 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 11, 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 7, 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 2.535308512, 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 1.189517947, 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 1.777912772, 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.032568489, 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 31677, 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.604078444, 2024-04-25T16:52:30.820 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000019069 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 59714, 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.013782800, 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000230 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 277120, 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 83664 2024-04-25T16:52:30.821 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.822 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:30.822 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 331, 2024-04-25T16:52:30.822 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 453, 2024-04-25T16:52:30.822 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 21113368, 2024-04-25T16:52:30.822 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 447211, 2024-04-25T16:52:30.822 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 9, 2024-04-25T16:52:30.822 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 8, 2024-04-25T16:52:30.822 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.051741189, 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.014752152, 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.026845783, 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.007923048, 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 451, 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.017671394, 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000039182 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.823 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 60, 2024-04-25T16:52:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000013793, 2024-04-25T16:52:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000229 2024-04-25T16:52:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:52:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 260432 2024-04-25T16:52:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.824 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 1209, 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 1031, 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 4822045, 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 497976, 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 8, 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 7, 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.083760347, 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.031016479, 2024-04-25T16:52:30.825 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.040533974, 2024-04-25T16:52:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.015665533, 2024-04-25T16:52:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1032, 2024-04-25T16:52:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.037088510, 2024-04-25T16:52:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000035938 2024-04-25T16:52:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:30.826 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 958, 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000174817, 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000182 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 2960, 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 1, 2024-04-25T16:52:30.827 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-MDSRank": { 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 2710, 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.456892507, 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000168595 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.828 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "queue_len": 0, 2024-04-25T16:52:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "complete_latency": { 2024-04-25T16:52:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 140, 2024-04-25T16:52:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.008735854, 2024-04-25T16:52:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000062398 2024-04-25T16:52:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": { 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "request": 30090, 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "reply": 29727, 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "reply_latency": { 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 29727, 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 20.142496399, 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000677582 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.830 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_reply": 0, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "forward": 360, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_complete": 11, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_fetch_keys": 4, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_commit": 0, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_split": 0, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_merge": 0, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes": 10807, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_top": 128, 2024-04-25T16:52:30.831 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_bottom": 10679, 2024-04-25T16:52:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_pinned": 10624, 2024-04-25T16:52:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_expired": 0, 2024-04-25T16:52:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "inodes_with_caps": 607, 2024-04-25T16:52:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "caps": 607, 2024-04-25T16:52:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "subtrees": 175, 2024-04-25T16:52:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse": 59684, 2024-04-25T16:52:30.832 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_hit": 1255, 2024-04-25T16:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_forward": 360, 2024-04-25T16:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_discover": 0, 2024-04-25T16:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "traverse_lock": 0, 2024-04-25T16:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "load_cent": 0, 2024-04-25T16:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "q": 0, 2024-04-25T16:52:30.833 INFO:teuthology.orchestra.run.smithi098.stdout: "exported": 0, 2024-04-25T16:52:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "exported_inodes": 0, 2024-04-25T16:52:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "imported": 0, 2024-04-25T16:52:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "imported_inodes": 0, 2024-04-25T16:52:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:52:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rfiles": 84381, 2024-04-25T16:52:30.834 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rbytes": 1535916169, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "root_rsnaps": 0, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:30.835 INFO:teuthology.orchestra.run.smithi098.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_revoke": 61, 2024-04-25T16:52:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_grant": 124, 2024-04-25T16:52:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_trunc": 2, 2024-04-25T16:52:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps": 250, 2024-04-25T16:52:30.836 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_caps_dirty": 114, 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_cap_release": 10, 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "process_request_cap_release": 162 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_cache": { 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update": 0, 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_try_discover": 0, 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_send_discover": 0, 2024-04-25T16:52:30.837 INFO:teuthology.orchestra.run.smithi098.stdout: "dir_handle_discover": 0, 2024-04-25T16:52:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays": 0, 2024-04-25T16:52:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_created": 32, 2024-04-25T16:52:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_enqueued": 36, 2024-04-25T16:52:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "strays_migrated": 0, 2024-04-25T16:52:30.838 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_started": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "recovery_completed": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_exportdir": 0, 2024-04-25T16:52:30.839 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_flush": 0, 2024-04-25T16:52:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:52:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "ireq_inodestats": 0 2024-04-25T16:52:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_log": { 2024-04-25T16:52:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "evlrg": 0, 2024-04-25T16:52:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "evadd": 629, 2024-04-25T16:52:30.840 INFO:teuthology.orchestra.run.smithi098.stdout: "evex": 0, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "evtrm": 0, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "ev": 16004, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "evexg": 0, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "evexd": 0, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "segadd": 0, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "segex": 0, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "segtrm": 0, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "seg": 21, 2024-04-25T16:52:30.841 INFO:teuthology.orchestra.run.smithi098.stdout: "segmjr": 3, 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "segexg": 0, 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "segexd": 0, 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "expos": 4194304, 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "wrpos": 38797410, 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "rdpos": 37778339, 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "jlat": { 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 203, 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 7.752017641, 2024-04-25T16:52:30.842 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.038187279 2024-04-25T16:52:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "replayed": 15375 2024-04-25T16:52:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_mem": { 2024-04-25T16:52:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "ino": 10204, 2024-04-25T16:52:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "ino+": 10848, 2024-04-25T16:52:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "ino-": 644, 2024-04-25T16:52:30.843 INFO:teuthology.orchestra.run.smithi098.stdout: "dir": 861, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "dir+": 906, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "dir-": 45, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "dn": 10807, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "dn+": 11208, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "dn-": 401, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "cap": 607, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "cap+": 809, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "cap-": 202, 2024-04-25T16:52:30.844 INFO:teuthology.orchestra.run.smithi098.stdout: "rss": 160952, 2024-04-25T16:52:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "heap": 240016 2024-04-25T16:52:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_server": { 2024-04-25T16:52:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_client_request": 30116, 2024-04-25T16:52:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "dispatch_server_request": 21, 2024-04-25T16:52:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_request": 30090, 2024-04-25T16:52:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_client_session": 0, 2024-04-25T16:52:30.845 INFO:teuthology.orchestra.run.smithi098.stdout: "handle_peer_request": 50, 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "req_create_latency": { 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 98, 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.165416686, 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.001687925 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getattr_latency": { 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.846 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "req_link_latency": { 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.847 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookup_latency": { 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 29576, 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 19.892374981, 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000672585 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:30.848 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.849 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.849 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.849 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupino_latency": { 2024-04-25T16:52:30.849 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.849 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.849 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.849 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.850 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupname_latency": { 2024-04-25T16:52:30.850 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.850 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.850 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.850 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.850 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:30.850 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.850 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.851 INFO:teuthology.orchestra.run.smithi098.stdout: "req_lssnap_latency": { 2024-04-25T16:52:30.852 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.852 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.852 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.852 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mkdir_latency": { 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mknod_latency": { 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.853 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.854 INFO:teuthology.orchestra.run.smithi098.stdout: "req_mksnap_latency": { 2024-04-25T16:52:30.854 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.854 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.854 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.854 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.854 INFO:teuthology.orchestra.run.smithi098.stdout: "req_open_latency": { 2024-04-25T16:52:30.854 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 21, 2024-04-25T16:52:30.855 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.010278992, 2024-04-25T16:52:30.855 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000489475 2024-04-25T16:52:30.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.855 INFO:teuthology.orchestra.run.smithi098.stdout: "req_readdir_latency": { 2024-04-25T16:52:30.855 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.855 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.855 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.855 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rename_latency": { 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.856 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.857 INFO:teuthology.orchestra.run.smithi098.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:30.857 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.857 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.857 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.857 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.857 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmdir_latency": { 2024-04-25T16:52:30.857 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.858 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.858 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:30.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.858 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.858 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.858 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setattr_latency": { 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.859 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.860 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:30.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.860 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.860 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.860 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:30.860 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setlayout_latency": { 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.861 INFO:teuthology.orchestra.run.smithi098.stdout: "req_setxattr_latency": { 2024-04-25T16:52:30.862 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.862 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.862 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.862 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.862 INFO:teuthology.orchestra.run.smithi098.stdout: "req_symlink_latency": { 2024-04-25T16:52:30.862 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.862 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.862 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.863 INFO:teuthology.orchestra.run.smithi098.stdout: "req_unlink_latency": { 2024-04-25T16:52:30.863 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 32, 2024-04-25T16:52:30.863 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.074425740, 2024-04-25T16:52:30.863 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.002325804 2024-04-25T16:52:30.863 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.863 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:30.863 INFO:teuthology.orchestra.run.smithi098.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_sessions": { 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: "session_count": 1, 2024-04-25T16:52:30.864 INFO:teuthology.orchestra.run.smithi098.stdout: "session_add": 1, 2024-04-25T16:52:30.865 INFO:teuthology.orchestra.run.smithi098.stdout: "session_remove": 0, 2024-04-25T16:52:30.865 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_open": 1, 2024-04-25T16:52:30.865 INFO:teuthology.orchestra.run.smithi098.stdout: "sessions_stale": 0, 2024-04-25T16:52:30.865 INFO:teuthology.orchestra.run.smithi098.stdout: "total_load": 24147, 2024-04-25T16:52:30.865 INFO:teuthology.orchestra.run.smithi098.stdout: "average_load": 24147, 2024-04-25T16:52:30.865 INFO:teuthology.orchestra.run.smithi098.stdout: "avg_session_uptime": 2313, 2024-04-25T16:52:30.865 INFO:teuthology.orchestra.run.smithi098.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:30.865 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.866 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-04-25T16:52:30.866 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:30.866 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:30.866 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:30.866 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:30.866 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:30.866 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:30.866 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:30.867 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:30.868 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-04-25T16:52:30.869 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 313289, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 128, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-04-25T16:52:30.870 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 276, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:30.871 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:30.872 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-04-25T16:52:30.872 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 30683907, 2024-04-25T16:52:30.872 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 156663, 2024-04-25T16:52:30.872 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:30.872 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-04-25T16:52:30.872 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:30.872 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-04-25T16:52:30.872 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "objecter": { 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "op_active": 0, 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "op_laggy": 0, 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send": 1870, 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "op_send_bytes": 1046115, 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "op_resend": 0, 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "op_reply": 1870, 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "op_latency": { 2024-04-25T16:52:30.873 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1870, 2024-04-25T16:52:30.874 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 9.114881900, 2024-04-25T16:52:30.874 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.004874268 2024-04-25T16:52:30.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.874 INFO:teuthology.orchestra.run.smithi098.stdout: "op_inflight": 0, 2024-04-25T16:52:30.874 INFO:teuthology.orchestra.run.smithi098.stdout: "oplen_avg": { 2024-04-25T16:52:30.874 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 1870, 2024-04-25T16:52:30.874 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 1961 2024-04-25T16:52:30.874 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.875 INFO:teuthology.orchestra.run.smithi098.stdout: "op": 1870, 2024-04-25T16:52:30.875 INFO:teuthology.orchestra.run.smithi098.stdout: "op_r": 1512, 2024-04-25T16:52:30.875 INFO:teuthology.orchestra.run.smithi098.stdout: "op_w": 358, 2024-04-25T16:52:30.875 INFO:teuthology.orchestra.run.smithi098.stdout: "op_rmw": 0, 2024-04-25T16:52:30.875 INFO:teuthology.orchestra.run.smithi098.stdout: "op_pg": 0, 2024-04-25T16:52:30.875 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_stat": 852, 2024-04-25T16:52:30.875 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_create": 0, 2024-04-25T16:52:30.875 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_read": 619, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_write": 221, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writefull": 15, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_writesame": 0, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_append": 0, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_zero": 2, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_truncate": 0, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_delete": 82, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_mapext": 0, 2024-04-25T16:52:30.876 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_getxattr": 16, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_setxattr": 0, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_call": 0, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_watch": 0, 2024-04-25T16:52:30.877 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_notify": 0, 2024-04-25T16:52:30.878 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:30.878 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls": 0, 2024-04-25T16:52:30.878 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:30.878 INFO:teuthology.orchestra.run.smithi098.stdout: "osdop_other": 2, 2024-04-25T16:52:30.878 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_active": 0, 2024-04-25T16:52:30.878 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_send": 0, 2024-04-25T16:52:30.878 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_resend": 0, 2024-04-25T16:52:30.878 INFO:teuthology.orchestra.run.smithi098.stdout: "linger_ping": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_active": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_send": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "poolop_resend": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_active": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_send": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "poolstat_resend": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_active": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_send": 0, 2024-04-25T16:52:30.879 INFO:teuthology.orchestra.run.smithi098.stdout: "statfs_resend": 0, 2024-04-25T16:52:30.880 INFO:teuthology.orchestra.run.smithi098.stdout: "command_active": 0, 2024-04-25T16:52:30.880 INFO:teuthology.orchestra.run.smithi098.stdout: "command_send": 0, 2024-04-25T16:52:30.880 INFO:teuthology.orchestra.run.smithi098.stdout: "command_resend": 0, 2024-04-25T16:52:30.880 INFO:teuthology.orchestra.run.smithi098.stdout: "map_epoch": 106, 2024-04-25T16:52:30.880 INFO:teuthology.orchestra.run.smithi098.stdout: "map_full": 0, 2024-04-25T16:52:30.880 INFO:teuthology.orchestra.run.smithi098.stdout: "map_inc": 1, 2024-04-25T16:52:30.880 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_sessions": 12, 2024-04-25T16:52:30.880 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_open": 12, 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_session_close": 0, 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_laggy": 0, 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_wr": 66, 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_rd": 57, 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_del": 29 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: "oft": { 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_objs": 0, 2024-04-25T16:52:30.881 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:30.882 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_updates": 0, 2024-04-25T16:52:30.882 INFO:teuthology.orchestra.run.smithi098.stdout: "omap_total_removes": 0 2024-04-25T16:52:30.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.882 INFO:teuthology.orchestra.run.smithi098.stdout: "osd-slow-ops": { 2024-04-25T16:52:30.882 INFO:teuthology.orchestra.run.smithi098.stdout: "slow_ops_count": 0 2024-04-25T16:52:30.882 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.882 INFO:teuthology.orchestra.run.smithi098.stdout: "purge_queue": { 2024-04-25T16:52:30.882 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:30.883 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_ops_high_water": 10, 2024-04-25T16:52:30.883 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing": 0, 2024-04-25T16:52:30.883 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executing_high_water": 5, 2024-04-25T16:52:30.883 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed_ops": 72, 2024-04-25T16:52:30.883 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_executed": 36, 2024-04-25T16:52:30.883 INFO:teuthology.orchestra.run.smithi098.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:30.883 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.883 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 33019, 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 40993829, 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 33019, 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T16:52:30.884 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T16:52:30.885 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 33019, 2024-04-25T16:52:30.885 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 40993829, 2024-04-25T16:52:30.885 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:30.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.885 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.885 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.885 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.885 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:30.886 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 1870, 2024-04-25T16:52:30.887 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 34637462, 2024-04-25T16:52:30.887 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1870, 2024-04-25T16:52:30.887 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 34637462, 2024-04-25T16:52:30.887 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:30.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.887 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.887 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.887 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 1024, 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 0, 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 0, 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:30.888 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 1870, 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 1870, 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 1870, 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1870, 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.889 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.890 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.890 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:30.890 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:30.890 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-04-25T16:52:30.890 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:30.890 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 36, 2024-04-25T16:52:30.890 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 3924, 2024-04-25T16:52:30.890 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 36, 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 18, 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 3924, 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.891 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-write_buf_throttle-0x5556dfedec30": { 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 3758096384, 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 629, 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 1018472, 2024-04-25T16:52:30.892 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 629, 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 203, 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 1018472, 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:30.893 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:30.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.894 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:30.894 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:52:31.521 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-25T16:52:31.521 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-25T16:52:31.521 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-04-25T16:52:31.521 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T16:52:31.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T16:52:31.762 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:52:32.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:32 smithi203 ceph-mon[33115]: pgmap v977: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.9 KiB/s wr, 13 op/s 2024-04-25T16:52:32.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:32 smithi098 ceph-mon[29162]: pgmap v977: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.9 KiB/s wr, 13 op/s 2024-04-25T16:52:32.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:32 smithi173 ceph-mon[32556]: pgmap v977: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.9 KiB/s wr, 13 op/s 2024-04-25T16:52:33.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:33 smithi098 ceph-mon[29162]: pgmap v978: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-04-25T16:52:33.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:33 smithi173 ceph-mon[32556]: pgmap v978: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-04-25T16:52:33.550 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-04-25T16:52:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 0, 2024-04-25T16:52:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 2, 2024-04-25T16:52:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 0, 2024-04-25T16:52:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 0, 2024-04-25T16:52:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T16:52:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 0, 2024-04-25T16:52:33.551 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.001148343, 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.000298952, 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.552 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:33.553 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 370, 2024-04-25T16:52:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 365, 2024-04-25T16:52:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 845248, 2024-04-25T16:52:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 225696, 2024-04-25T16:52:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 2, 2024-04-25T16:52:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 1, 2024-04-25T16:52:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.053075188, 2024-04-25T16:52:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.020537526, 2024-04-25T16:52:33.554 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.027152917, 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.007698040, 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 364, 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.012001608, 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000032971 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:33.555 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 845248, 2024-04-25T16:52:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 225696 2024-04-25T16:52:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_messages": 6, 2024-04-25T16:52:33.556 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages": 281, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_bytes": 3280, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_bytes": 206736, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_created_connections": 4, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_active_connections": 2, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_total_time": 0.020267857, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_send_time": 0.015915967, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_recv_time": 0.002077558, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T16:52:33.557 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 280, 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.014222005, 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000050792 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:33.558 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_recv_encrypted_bytes": 3280, 2024-04-25T16:52:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "msgr_send_encrypted_bytes": 206736 2024-04-25T16:52:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "cct": { 2024-04-25T16:52:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "total_workers": 0, 2024-04-25T16:52:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "unhealthy_workers": 0 2024-04-25T16:52:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.559 INFO:teuthology.orchestra.run.smithi098.stdout: "mempool": { 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:33.560 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:33.561 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:33.562 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_items": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:33.563 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_bytes": 149775, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_anon_items": 31, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_bytes": 0, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_items": 0, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:33.564 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_bytes": 0, 2024-04-25T16:52:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_items": 0, 2024-04-25T16:52:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:33.565 INFO:teuthology.orchestra.run.smithi098.stdout: "pgmap_items": 0, 2024-04-25T16:52:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_bytes": 928, 2024-04-25T16:52:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "mds_co_items": 2, 2024-04-25T16:52:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_1_items": 0, 2024-04-25T16:52:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "unittest_2_items": 0 2024-04-25T16:52:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-04-25T16:52:33.566 INFO:teuthology.orchestra.run.smithi098.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 0, 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "max": 104857600, 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "get_started": 0, 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "get": 372, 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "get_sum": 792949, 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "get_or_fail_success": 372, 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "take": 0, 2024-04-25T16:52:33.567 INFO:teuthology.orchestra.run.smithi098.stdout: "take_sum": 0, 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "put": 372, 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "put_sum": 792949, 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "wait": { 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "avgcount": 0, 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "sum": 0.000000000, 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: "avgtime": 0.000000000 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-04-25T16:52:33.568 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-04-25T16:52:33.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:33 smithi203 ceph-mon[33115]: pgmap v978: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.7 KiB/s wr, 12 op/s 2024-04-25T16:52:34.139 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T16:52:34.139 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T16:52:34.139 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-25T16:52:34.140 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T16:52:34.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T16:52:34.389 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:52:35.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:35 smithi203 ceph-mon[33115]: pgmap v979: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.1 KiB/s wr, 12 op/s 2024-04-25T16:52:35.705 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:35 smithi173 ceph-mon[32556]: pgmap v979: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.1 KiB/s wr, 12 op/s 2024-04-25T16:52:35.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:35 smithi098 ceph-mon[29162]: pgmap v979: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.1 KiB/s wr, 12 op/s 2024-04-25T16:52:36.334 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T16:52:36.335 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:36.335 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 7883, 2024-04-25T16:52:36.335 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 10882, 2024-04-25T16:52:36.335 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 62635949, 2024-04-25T16:52:36.335 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 11262335, 2024-04-25T16:52:36.335 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 10, 2024-04-25T16:52:36.335 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-04-25T16:52:36.336 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.677345348, 2024-04-25T16:52:36.336 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.248793315, 2024-04-25T16:52:36.336 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.294730052, 2024-04-25T16:52:36.336 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.146139341, 2024-04-25T16:52:36.336 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:36.336 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 10871, 2024-04-25T16:52:36.336 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 5.764897415, 2024-04-25T16:52:36.337 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000530300 2024-04-25T16:52:36.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.337 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:36.337 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 10015, 2024-04-25T16:52:36.337 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.002348373, 2024-04-25T16:52:36.337 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000234 2024-04-25T16:52:36.337 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.337 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 600176, 2024-04-25T16:52:36.338 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 120672 2024-04-25T16:52:36.338 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.338 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:36.338 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 13433, 2024-04-25T16:52:36.338 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 19605, 2024-04-25T16:52:36.338 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 110468764, 2024-04-25T16:52:36.338 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 11297208, 2024-04-25T16:52:36.338 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 9, 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 8, 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 1.240150949, 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.516679456, 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.476208121, 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.236631895, 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 19604, 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 16.963779871, 2024-04-25T16:52:36.339 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000865322 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 16851, 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.005696536, 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000338 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3120, 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 339664 2024-04-25T16:52:36.340 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 63393, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 66231, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 142054477, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 34787752, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 10, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 8, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 5.195984350, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 1.853933783, 2024-04-25T16:52:36.341 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 4.197906373, 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.232589003, 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 66226, 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 12.893055863, 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000194682 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 115046, 2024-04-25T16:52:36.342 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.022916680, 2024-04-25T16:52:36.343 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000199 2024-04-25T16:52:36.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.343 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:36.343 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:36.343 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.343 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-25T16:52:36.343 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-25T16:52:36.343 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 4815, 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 6.396476707, 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001328447 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.344 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.345 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:36.345 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T16:52:36.345 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T16:52:36.345 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1898, 2024-04-25T16:52:36.345 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.083715589, 2024-04-25T16:52:36.345 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000044107 2024-04-25T16:52:36.345 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.345 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.346 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-25T16:52:36.346 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 51256, 2024-04-25T16:52:36.346 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 51039, 2024-04-25T16:52:36.346 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-25T16:52:36.346 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 51039, 2024-04-25T16:52:36.346 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 28.347170213, 2024-04-25T16:52:36.346 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000555402 2024-04-25T16:52:36.347 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.347 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-25T16:52:36.347 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 216, 2024-04-25T16:52:36.347 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 346, 2024-04-25T16:52:36.347 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 41, 2024-04-25T16:52:36.347 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 362, 2024-04-25T16:52:36.348 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 1, 2024-04-25T16:52:36.348 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-25T16:52:36.348 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 66152, 2024-04-25T16:52:36.348 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 16086, 2024-04-25T16:52:36.348 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 50066, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 43171, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 484, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 11014, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 11014, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 553, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 99281, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 14479, 2024-04-25T16:52:36.349 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 82, 2024-04-25T16:52:36.350 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-25T16:52:36.350 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 6, 2024-04-25T16:52:36.350 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:36.350 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 5, 2024-04-25T16:52:36.350 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 28, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 484, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 91, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 55, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 146, 2024-04-25T16:52:36.351 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 84381, 2024-04-25T16:52:36.352 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 1535916169, 2024-04-25T16:52:36.352 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-04-25T16:52:36.352 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:36.352 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:36.352 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:36.352 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:36.352 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:36.353 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:36.353 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:36.353 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:36.353 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:36.353 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 750, 2024-04-25T16:52:36.353 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 1540, 2024-04-25T16:52:36.353 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 27, 2024-04-25T16:52:36.354 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:36.354 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:36.354 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 3233, 2024-04-25T16:52:36.354 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 1231, 2024-04-25T16:52:36.354 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 384, 2024-04-25T16:52:36.354 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 2099 2024-04-25T16:52:36.354 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.354 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-25T16:52:36.355 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-25T16:52:36.355 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:36.355 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-25T16:52:36.355 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-25T16:52:36.355 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 28, 2024-04-25T16:52:36.355 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-25T16:52:36.355 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:36.355 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:36.356 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 413, 2024-04-25T16:52:36.356 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 413, 2024-04-25T16:52:36.356 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:36.356 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-25T16:52:36.356 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:36.356 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:36.356 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 10, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 10, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 28, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 1, 2024-04-25T16:52:36.357 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:36.358 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-25T16:52:36.358 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.358 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-25T16:52:36.358 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-25T16:52:36.358 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 5857, 2024-04-25T16:52:36.358 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 11873, 2024-04-25T16:52:36.358 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-25T16:52:36.358 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 144971, 2024-04-25T16:52:36.359 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-25T16:52:36.359 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 11873, 2024-04-25T16:52:36.359 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 7, 2024-04-25T16:52:36.359 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 17, 2024-04-25T16:52:36.359 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-25T16:52:36.359 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 145, 2024-04-25T16:52:36.359 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 12, 2024-04-25T16:52:36.359 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 17, 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 128990685, 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 353201302, 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 338858206, 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2025, 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 348.123634149, 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.171912905 2024-04-25T16:52:36.360 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 139114 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 64893, 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 78096, 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 13203, 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 5007, 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 5820, 2024-04-25T16:52:36.361 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 813, 2024-04-25T16:52:36.362 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 66152, 2024-04-25T16:52:36.362 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 79655, 2024-04-25T16:52:36.362 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 13503, 2024-04-25T16:52:36.362 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 11014, 2024-04-25T16:52:36.362 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 16980, 2024-04-25T16:52:36.362 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 5966, 2024-04-25T16:52:36.362 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 363768, 2024-04-25T16:52:36.362 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 248208 2024-04-25T16:52:36.363 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.363 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-25T16:52:36.363 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 52158, 2024-04-25T16:52:36.363 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 36, 2024-04-25T16:52:36.363 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 51256, 2024-04-25T16:52:36.363 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 26, 2024-04-25T16:52:36.363 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 4987, 2024-04-25T16:52:36.363 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1234, 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.537643717, 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001246064 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 3130, 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.909327604, 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000290520 2024-04-25T16:52:36.364 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.365 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:36.365 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.365 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.365 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.365 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.365 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-25T16:52:36.365 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.365 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.366 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.366 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-25T16:52:36.366 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 44711, 2024-04-25T16:52:36.366 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 16.816416532, 2024-04-25T16:52:36.366 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000376113 2024-04-25T16:52:36.366 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.366 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-25T16:52:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.367 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.368 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.369 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.369 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.369 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:36.369 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.369 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.370 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.370 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-25T16:52:36.370 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.370 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.370 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.370 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.371 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-25T16:52:36.371 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 23, 2024-04-25T16:52:36.371 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.026608323, 2024-04-25T16:52:36.371 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001156883 2024-04-25T16:52:36.371 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.371 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-25T16:52:36.371 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.371 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 387, 2024-04-25T16:52:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.391994219, 2024-04-25T16:52:36.373 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001012904 2024-04-25T16:52:36.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.373 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-25T16:52:36.373 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1136, 2024-04-25T16:52:36.373 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 7.746177829, 2024-04-25T16:52:36.373 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.006818818 2024-04-25T16:52:36.373 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.374 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-25T16:52:36.374 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5, 2024-04-25T16:52:36.374 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.007677114, 2024-04-25T16:52:36.374 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001535422 2024-04-25T16:52:36.374 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.374 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:36.374 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.374 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.375 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.375 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:36.375 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.375 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.375 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.375 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.375 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 23, 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.118448003, 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.005149913 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.376 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.377 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:36.377 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.377 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.377 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.377 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.377 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.378 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.379 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:36.379 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.379 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.379 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.379 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.379 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-25T16:52:36.379 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.379 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.380 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.380 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-25T16:52:36.380 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.380 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.380 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.380 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.380 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 390, 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.792876872, 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002033017 2024-04-25T16:52:36.381 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.382 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:36.382 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:36.382 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:36.382 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.382 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.382 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.383 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.383 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.383 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-25T16:52:36.383 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-04-25T16:52:36.383 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 1, 2024-04-25T16:52:36.383 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2024-04-25T16:52:36.383 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 0, 2024-04-25T16:52:36.383 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-25T16:52:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 7077, 2024-04-25T16:52:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 7077, 2024-04-25T16:52:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 2323, 2024-04-25T16:52:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-25T16:52:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:36.384 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:36.385 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:36.386 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:36.386 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:36.386 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:36.386 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:36.386 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:36.386 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:36.386 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:36.386 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:36.387 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-25T16:52:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 314945, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 126, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:36.389 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:36.390 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:36.390 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:36.390 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 276, 2024-04-25T16:52:36.390 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:36.390 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:36.390 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:36.390 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-25T16:52:36.390 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 195306316, 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 983416, 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-25T16:52:36.391 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 18179, 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 24330810, 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 0, 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 18179, 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 18179, 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 9006.696308682, 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.495445090 2024-04-25T16:52:36.392 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.393 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-25T16:52:36.393 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-25T16:52:36.393 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 18179, 2024-04-25T16:52:36.393 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 38019 2024-04-25T16:52:36.393 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.393 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 18179, 2024-04-25T16:52:36.393 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 2139, 2024-04-25T16:52:36.393 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 16040, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 455, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 11916, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 655, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 2291, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 195, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-25T16:52:36.394 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-25T16:52:36.395 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 2, 2024-04-25T16:52:36.395 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 1, 2024-04-25T16:52:36.395 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 817, 2024-04-25T16:52:36.395 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-25T16:52:36.395 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:36.395 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:36.395 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 372, 2024-04-25T16:52:36.395 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 18034, 2024-04-25T16:52:36.396 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:36.396 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:36.396 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:36.396 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-25T16:52:36.396 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-25T16:52:36.396 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-25T16:52:36.396 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:36.396 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 27, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-25T16:52:36.397 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-25T16:52:36.398 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-25T16:52:36.399 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 106, 2024-04-25T16:52:36.399 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-25T16:52:36.399 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 11, 2024-04-25T16:52:36.399 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 12, 2024-04-25T16:52:36.399 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 12, 2024-04-25T16:52:36.399 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-25T16:52:36.399 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-25T16:52:36.399 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 1506, 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 1377, 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 371 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-25T16:52:36.400 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 5 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 10, 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 7, 2024-04-25T16:52:36.401 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 803, 2024-04-25T16:52:36.402 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 413, 2024-04-25T16:52:36.402 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:36.402 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.402 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:36.402 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:36.402 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T16:52:36.402 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:36.402 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 84705, 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 307519315, 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 84705, 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 84705, 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 307519315, 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:36.403 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:36.404 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:36.405 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:36.405 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:36.405 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:36.405 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 18179, 2024-04-25T16:52:36.405 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 309559685, 2024-04-25T16:52:36.405 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 18179, 2024-04-25T16:52:36.405 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 309559685, 2024-04-25T16:52:36.405 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:36.406 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 18179, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 18179, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 18179, 2024-04-25T16:52:36.407 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 18179, 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:36.408 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 413, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 44833, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 413, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 266, 2024-04-25T16:52:36.409 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 44833, 2024-04-25T16:52:36.410 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:36.410 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.410 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.410 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.410 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.410 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:36.410 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55eec54a2750": { 2024-04-25T16:52:36.410 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 5857, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 14331971, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 5857, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:36.411 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 2025, 2024-04-25T16:52:36.412 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 14331971, 2024-04-25T16:52:36.412 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:36.412 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:36.412 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:36.412 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:36.412 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.412 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:36.412 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T16:52:37.202 INFO:tasks.check_counter:Daemon mds.b mds.exported=28 2024-04-25T16:52:37.202 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-25T16:52:37.202 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-04-25T16:52:37.202 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T16:52:37.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T16:52:37.448 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:52:37.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:37 smithi173 ceph-mon[32556]: pgmap v980: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T16:52:37.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:37 smithi203 ceph-mon[33115]: pgmap v980: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T16:52:37.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:37 smithi098 ceph-mon[29162]: pgmap v980: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2024-04-25T16:52:39.259 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T16:52:39.259 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:39.260 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 21169, 2024-04-25T16:52:39.260 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 11916, 2024-04-25T16:52:39.260 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 10929684, 2024-04-25T16:52:39.260 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 13160560, 2024-04-25T16:52:39.260 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 22, 2024-04-25T16:52:39.260 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-25T16:52:39.260 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 1.045868535, 2024-04-25T16:52:39.260 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.459695144, 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.753357732, 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.029305177, 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 11823, 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.304254302, 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000025734 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:39.261 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 29301, 2024-04-25T16:52:39.262 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.005886587, 2024-04-25T16:52:39.262 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000200 2024-04-25T16:52:39.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.262 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:39.262 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:39.262 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.262 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:39.262 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 343, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 700, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 425897, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 4569102, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 25, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 6, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.056356175, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.028264557, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.023715303, 2024-04-25T16:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.003809243, 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 694, 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.056522248, 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000081444 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 487, 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000092975, 2024-04-25T16:52:39.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000190 2024-04-25T16:52:39.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.265 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 2496, 2024-04-25T16:52:39.265 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 661632 2024-04-25T16:52:39.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.265 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:39.265 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 8584, 2024-04-25T16:52:39.265 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 16631, 2024-04-25T16:52:39.265 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 4669949, 2024-04-25T16:52:39.266 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 12073729, 2024-04-25T16:52:39.266 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 20, 2024-04-25T16:52:39.266 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-25T16:52:39.266 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.564891218, 2024-04-25T16:52:39.266 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.242623900, 2024-04-25T16:52:39.266 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.364123565, 2024-04-25T16:52:39.266 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.015824448, 2024-04-25T16:52:39.266 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 16627, 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.256179858, 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000015407 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 22735, 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.004131353, 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000181 2024-04-25T16:52:39.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 1793360, 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 240064 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-25T16:52:39.268 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T16:52:39.269 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T16:52:39.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1052, 2024-04-25T16:52:39.269 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2.912117581, 2024-04-25T16:52:39.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002768172 2024-04-25T16:52:39.269 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.269 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.269 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 15, 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000441025, 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000029401 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-25T16:52:39.270 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 2862, 2024-04-25T16:52:39.271 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 2770, 2024-04-25T16:52:39.271 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-25T16:52:39.271 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2770, 2024-04-25T16:52:39.271 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 29.798867881, 2024-04-25T16:52:39.271 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.010757714 2024-04-25T16:52:39.271 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.271 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-25T16:52:39.271 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 92, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 56, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 5, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 4689, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 130, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 4499, 2024-04-25T16:52:39.272 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 60, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 4503, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 937, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 1235, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 1235, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 176, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 3431, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 3026, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 59, 2024-04-25T16:52:39.273 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 193, 2024-04-25T16:52:39.274 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:39.274 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:39.274 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-25T16:52:39.274 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-25T16:52:39.274 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-25T16:52:39.274 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 12, 2024-04-25T16:52:39.274 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 941, 2024-04-25T16:52:39.274 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 133, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 3176, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 86, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 84381, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 1535916169, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 86, 2024-04-25T16:52:39.275 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 3, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 3, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 83, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 129, 2024-04-25T16:52:39.276 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 2158, 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 5541, 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 5409, 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 153, 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 2207 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.277 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 159, 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 18, 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:39.278 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 2, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 2, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-25T16:52:39.279 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 12, 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 5, 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.280 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 7440, 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 7440, 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 10, 2024-04-25T16:52:39.281 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 10, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 19894996, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 0, 2024-04-25T16:52:39.282 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 368, 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 9.433422641, 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.025634300 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 0 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 4692, 2024-04-25T16:52:39.283 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 5631, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 939, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 510, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 538, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 28, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 4689, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 5628, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 939, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 1235, 2024-04-25T16:52:39.284 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 9086, 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 7851, 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 155920, 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 223632 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 3012, 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 5313, 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 2862, 2024-04-25T16:52:39.285 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 31, 2024-04-25T16:52:39.286 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 10628, 2024-04-25T16:52:39.286 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-25T16:52:39.286 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2155, 2024-04-25T16:52:39.286 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 14.832372865, 2024-04-25T16:52:39.286 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.006882771 2024-04-25T16:52:39.286 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.286 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-25T16:52:39.286 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 186, 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.048690990, 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000261779 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.287 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-25T16:52:39.288 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.288 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.288 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.288 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.288 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-25T16:52:39.288 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 128, 2024-04-25T16:52:39.288 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.074708352, 2024-04-25T16:52:39.288 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000583659 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.289 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.290 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.290 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-25T16:52:39.290 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.290 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.290 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.290 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.290 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.291 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.292 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-25T16:52:39.292 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.292 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.293 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.293 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.293 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-25T16:52:39.293 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 42, 2024-04-25T16:52:39.293 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 14.644201225, 2024-04-25T16:52:39.294 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.348671457 2024-04-25T16:52:39.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.294 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-25T16:52:39.294 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.294 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.294 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.294 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.294 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 5, 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.001823841, 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000364768 2024-04-25T16:52:39.295 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 247, 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.185320550, 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000750285 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.296 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.297 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:39.298 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.299 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.299 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.299 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:39.299 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.299 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.299 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.299 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 3, 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.003227697, 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001075899 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.300 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.301 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.302 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.302 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-25T16:52:39.302 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.302 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.302 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.302 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.302 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.003470031, 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001735015 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2, 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.005052330, 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.002526165 2024-04-25T16:52:39.303 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.304 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-25T16:52:39.305 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-04-25T16:52:39.305 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 2, 2024-04-25T16:52:39.305 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 1, 2024-04-25T16:52:39.305 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-04-25T16:52:39.305 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-25T16:52:39.305 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 3, 2024-04-25T16:52:39.305 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 3, 2024-04-25T16:52:39.305 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 1204, 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:39.306 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:39.307 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:39.307 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:39.307 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:39.307 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:39.307 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:39.307 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:39.307 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:39.308 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:39.309 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:39.309 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:39.309 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:39.309 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:39.309 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:39.309 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:39.309 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:39.309 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 307395, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 138, 2024-04-25T16:52:39.310 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:39.311 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 276, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 14729959, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 138557, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-25T16:52:39.312 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:39.313 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-25T16:52:39.313 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.313 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-25T16:52:39.313 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-25T16:52:39.313 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-25T16:52:39.313 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 821, 2024-04-25T16:52:39.313 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 16915975, 2024-04-25T16:52:39.313 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 33, 2024-04-25T16:52:39.314 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 821, 2024-04-25T16:52:39.314 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-25T16:52:39.314 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 821, 2024-04-25T16:52:39.314 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 8.390826817, 2024-04-25T16:52:39.314 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.010220251 2024-04-25T16:52:39.314 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.315 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-25T16:52:39.315 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-25T16:52:39.318 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 821, 2024-04-25T16:52:39.318 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1149 2024-04-25T16:52:39.318 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.318 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 821, 2024-04-25T16:52:39.318 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 87, 2024-04-25T16:52:39.318 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 734, 2024-04-25T16:52:39.318 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-25T16:52:39.318 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 0, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 10, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 1, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 369, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 47, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-25T16:52:39.319 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 5, 2024-04-25T16:52:39.320 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 19, 2024-04-25T16:52:39.320 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-25T16:52:39.320 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:39.320 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:39.320 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 86, 2024-04-25T16:52:39.320 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 20, 2024-04-25T16:52:39.320 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:39.320 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-25T16:52:39.321 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-25T16:52:39.322 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-25T16:52:39.323 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-25T16:52:39.323 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-25T16:52:39.323 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-25T16:52:39.323 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-25T16:52:39.323 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-25T16:52:39.323 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-25T16:52:39.323 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 106, 2024-04-25T16:52:39.323 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 25, 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 12, 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 12, 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 442, 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 0, 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 150 2024-04-25T16:52:39.324 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.325 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-25T16:52:39.325 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 1, 2024-04-25T16:52:39.325 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:39.325 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-25T16:52:39.325 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-25T16:52:39.325 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.325 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-25T16:52:39.325 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 19 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 4, 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 2, 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 4, 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 2, 2024-04-25T16:52:39.326 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:39.327 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.327 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:39.327 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:39.327 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T16:52:39.327 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:39.327 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 30092, 2024-04-25T16:52:39.327 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 13285297, 2024-04-25T16:52:39.327 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 30092, 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 30092, 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 13285297, 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.328 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:39.329 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 821, 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 16916193, 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 821, 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 16916193, 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.330 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:39.331 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 821, 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 821, 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 821, 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 821, 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 2, 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 218, 2024-04-25T16:52:39.333 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:39.334 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 2, 2024-04-25T16:52:39.334 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:39.334 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:39.334 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1, 2024-04-25T16:52:39.334 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 218, 2024-04-25T16:52:39.334 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:39.334 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.334 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x5594e9dce8f0": { 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 7440, 2024-04-25T16:52:39.335 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 15700692, 2024-04-25T16:52:39.336 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:39.336 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 7440, 2024-04-25T16:52:39.336 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:39.336 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:39.336 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 368, 2024-04-25T16:52:39.336 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 15700692, 2024-04-25T16:52:39.336 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:39.336 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:39.337 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:39.337 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:39.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.337 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:39.337 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T16:52:39.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:39 smithi173 ceph-mon[32556]: pgmap v981: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-25T16:52:39.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:39 smithi203 ceph-mon[33115]: pgmap v981: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-25T16:52:39.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:39 smithi098 ceph-mon[29162]: pgmap v981: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.8 KiB/s wr, 11 op/s 2024-04-25T16:52:40.137 INFO:tasks.check_counter:Daemon mds.e mds.exported=12 2024-04-25T16:52:40.137 INFO:tasks.check_counter:Daemon mds.e mds.imported=133 2024-04-25T16:52:40.137 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=5 2024-04-25T16:52:40.137 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T16:52:40.138 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T16:52:40.377 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:52:41.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:41 smithi173 ceph-mon[32556]: pgmap v982: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T16:52:41.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:41 smithi203 ceph-mon[33115]: pgmap v982: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T16:52:41.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:41 smithi098 ceph-mon[29162]: pgmap v982: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-04-25T16:52:42.140 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T16:52:42.140 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:42.140 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 1020, 2024-04-25T16:52:42.141 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 1008, 2024-04-25T16:52:42.141 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 69567969, 2024-04-25T16:52:42.141 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 359306, 2024-04-25T16:52:42.141 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 7, 2024-04-25T16:52:42.141 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-04-25T16:52:42.141 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.147347784, 2024-04-25T16:52:42.141 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.037159372, 2024-04-25T16:52:42.141 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.066916497, 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.034704597, 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1007, 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.045165310, 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000044851 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.142 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 277056, 2024-04-25T16:52:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 82960 2024-04-25T16:52:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 350, 2024-04-25T16:52:42.143 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 450, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 79797261, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 376023, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.078955098, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.013481334, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.052635930, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.009736445, 2024-04-25T16:52:42.144 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 449, 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.020483895, 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000045621 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.145 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3120, 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 266256 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 490, 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 492, 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 75621936, 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 155836, 2024-04-25T16:52:42.146 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 6, 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 5, 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.085552203, 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.011436914, 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.055947383, 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.014409568, 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 490, 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.024705471, 2024-04-25T16:52:42.147 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000050419 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.148 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-25T16:52:42.149 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-25T16:52:42.149 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-25T16:52:42.149 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.149 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-25T16:52:42.149 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T16:52:42.149 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T16:52:42.149 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2817, 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.081386667, 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000028891 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.150 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.151 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-25T16:52:42.152 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 45436, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 0, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 45436, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 45436, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-25T16:52:42.153 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 478, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-25T16:52:42.154 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 84381, 2024-04-25T16:52:42.155 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 1535916169, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:42.156 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:52:42.157 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-25T16:52:42.158 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-25T16:52:42.158 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.158 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-25T16:52:42.158 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-25T16:52:42.158 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:42.158 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-25T16:52:42.158 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-25T16:52:42.159 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:42.160 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:42.160 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:42.160 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-25T16:52:42.160 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-25T16:52:42.160 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:42.160 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:42.160 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:42.161 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-25T16:52:42.161 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-25T16:52:42.161 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:52:42.161 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:42.161 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-25T16:52:42.161 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.161 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-25T16:52:42.161 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-25T16:52:42.162 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-04-25T16:52:42.162 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-25T16:52:42.162 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-25T16:52:42.162 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 108105, 2024-04-25T16:52:42.162 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-25T16:52:42.162 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-25T16:52:42.162 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-04-25T16:52:42.162 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-25T16:52:42.163 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-25T16:52:42.163 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 145, 2024-04-25T16:52:42.163 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 9, 2024-04-25T16:52:42.163 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-25T16:52:42.163 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-25T16:52:42.163 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 128990685, 2024-04-25T16:52:42.163 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 353211168, 2024-04-25T16:52:42.163 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 353208406, 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 108105 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-25T16:52:42.164 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 44287, 2024-04-25T16:52:42.165 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 52585, 2024-04-25T16:52:42.165 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 8298, 2024-04-25T16:52:42.165 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 3970, 2024-04-25T16:52:42.165 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 4499, 2024-04-25T16:52:42.165 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 529, 2024-04-25T16:52:42.165 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 45436, 2024-04-25T16:52:42.165 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 53190, 2024-04-25T16:52:42.165 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 7754, 2024-04-25T16:52:42.166 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-25T16:52:42.166 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-04-25T16:52:42.166 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-04-25T16:52:42.166 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 267336, 2024-04-25T16:52:42.166 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215440 2024-04-25T16:52:42.166 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.166 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-25T16:52:42.166 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.167 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.168 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-25T16:52:42.168 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.168 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.168 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.168 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.168 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:42.168 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.168 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-25T16:52:42.169 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.170 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.170 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.170 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:42.170 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.170 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.170 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.170 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.171 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-25T16:52:42.171 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.171 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.171 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.171 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.171 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-25T16:52:42.171 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.171 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:42.172 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.173 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.173 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.173 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.173 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-25T16:52:42.173 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.173 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.173 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.174 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.174 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-25T16:52:42.174 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.174 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.175 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.175 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.175 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-25T16:52:42.175 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.175 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-25T16:52:42.176 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-25T16:52:42.177 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.178 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.178 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.178 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:42.178 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.178 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.178 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.178 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.179 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.180 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.181 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-25T16:52:42.181 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.181 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.181 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.181 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.182 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:42.182 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.182 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.182 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.182 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.182 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:42.182 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.182 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.183 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.183 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-25T16:52:42.183 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.183 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.183 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.183 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.183 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-25T16:52:42.184 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.184 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.184 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.184 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.184 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-25T16:52:42.184 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.184 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.184 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:42.185 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:42.186 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.186 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.186 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.186 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.186 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.186 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-25T16:52:42.186 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-04-25T16:52:42.186 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 1, 2024-04-25T16:52:42.187 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 0, 2024-04-25T16:52:42.187 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 0, 2024-04-25T16:52:42.187 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-25T16:52:42.187 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-04-25T16:52:42.187 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-04-25T16:52:42.187 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 2329, 2024-04-25T16:52:42.187 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:42.187 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.188 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-25T16:52:42.188 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:42.188 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:42.188 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:42.188 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:42.188 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:42.188 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:42.188 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:42.189 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:42.190 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:42.191 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:42.191 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:42.191 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:42.191 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:42.191 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:42.191 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:42.191 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-25T16:52:42.192 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:42.192 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:42.192 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:42.192 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:42.192 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 229656, 2024-04-25T16:52:42.192 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 69, 2024-04-25T16:52:42.192 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:42.192 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 276, 2024-04-25T16:52:42.193 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 131219266, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 607204, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:42.194 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 1704, 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 26, 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 0, 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 1704, 2024-04-25T16:52:42.195 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-25T16:52:42.196 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1704, 2024-04-25T16:52:42.196 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 2.741943910, 2024-04-25T16:52:42.196 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001609122 2024-04-25T16:52:42.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.196 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-25T16:52:42.196 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-25T16:52:42.196 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1704, 2024-04-25T16:52:42.196 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1705 2024-04-25T16:52:42.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.197 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 1704, 2024-04-25T16:52:42.197 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 1704, 2024-04-25T16:52:42.197 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-04-25T16:52:42.197 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-25T16:52:42.197 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-25T16:52:42.197 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 958, 2024-04-25T16:52:42.197 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 743, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-04-25T16:52:42.198 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 2, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-25T16:52:42.199 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-25T16:52:42.200 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-25T16:52:42.201 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-25T16:52:42.201 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-25T16:52:42.201 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-25T16:52:42.201 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-25T16:52:42.201 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-25T16:52:42.201 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-25T16:52:42.202 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-25T16:52:42.202 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-25T16:52:42.202 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-25T16:52:42.202 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-25T16:52:42.202 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-25T16:52:42.202 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-25T16:52:42.202 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 106, 2024-04-25T16:52:42.202 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-25T16:52:42.203 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 1, 2024-04-25T16:52:42.203 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 10, 2024-04-25T16:52:42.203 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 10, 2024-04-25T16:52:42.203 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-25T16:52:42.203 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-25T16:52:42.203 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-04-25T16:52:42.203 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-04-25T16:52:42.203 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-04-25T16:52:42.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.204 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-25T16:52:42.204 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-25T16:52:42.204 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:42.204 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-25T16:52:42.204 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-25T16:52:42.204 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.204 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-25T16:52:42.205 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-25T16:52:42.206 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:42.206 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.206 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:42.206 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:42.206 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T16:52:42.206 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:42.206 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 1856, 2024-04-25T16:52:42.206 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 224809460, 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 1856, 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1856, 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 224809460, 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.207 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:42.208 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:42.209 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:42.209 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:42.209 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 1704, 2024-04-25T16:52:42.209 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 224222511, 2024-04-25T16:52:42.209 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1704, 2024-04-25T16:52:42.209 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 224222511, 2024-04-25T16:52:42.209 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:42.209 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.210 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.210 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.210 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.210 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.210 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:42.210 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:42.210 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-25T16:52:42.210 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 1704, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 1704, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1704, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1704, 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:42.211 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.212 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.212 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.212 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.212 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:42.212 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:42.212 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-25T16:52:42.213 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x55eab3c82c30": { 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:42.214 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-25T16:52:42.215 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-25T16:52:42.216 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:42.216 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:42.216 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:42.216 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:42.216 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.216 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:42.216 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T16:52:42.995 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-25T16:52:42.995 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-25T16:52:42.995 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-04-25T16:52:42.995 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T16:52:42.995 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T16:52:43.234 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:52:43.661 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:43 smithi173 ceph-mon[32556]: pgmap v983: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T16:52:43.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:43 smithi203 ceph-mon[33115]: pgmap v983: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T16:52:43.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:43 smithi098 ceph-mon[29162]: pgmap v983: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T16:52:45.190 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-04-25T16:52:45.191 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:45.191 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 684, 2024-04-25T16:52:45.191 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 663, 2024-04-25T16:52:45.191 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 505296, 2024-04-25T16:52:45.191 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 251562, 2024-04-25T16:52:45.191 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-04-25T16:52:45.191 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 1, 2024-04-25T16:52:45.192 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.079511187, 2024-04-25T16:52:45.192 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.032255235, 2024-04-25T16:52:45.192 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.022119125, 2024-04-25T16:52:45.192 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.018693986, 2024-04-25T16:52:45.192 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:45.192 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 662, 2024-04-25T16:52:45.192 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.035296223, 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000053317 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 332256, 2024-04-25T16:52:45.193 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 87792 2024-04-25T16:52:45.194 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.194 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:45.194 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 522, 2024-04-25T16:52:45.194 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 627, 2024-04-25T16:52:45.194 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 130366, 2024-04-25T16:52:45.194 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 439684, 2024-04-25T16:52:45.194 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 5, 2024-04-25T16:52:45.194 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 4, 2024-04-25T16:52:45.195 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.050280076, 2024-04-25T16:52:45.195 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.015189526, 2024-04-25T16:52:45.195 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.013223057, 2024-04-25T16:52:45.195 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.017703152, 2024-04-25T16:52:45.195 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:45.195 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 626, 2024-04-25T16:52:45.195 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.019529816, 2024-04-25T16:52:45.195 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000031197 2024-04-25T16:52:45.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.196 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:45.196 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.196 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.196 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.196 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.196 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 3120, 2024-04-25T16:52:45.196 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 275216 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_messages": 536, 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages": 538, 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_bytes": 9272274, 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_bytes": 170455, 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_created_connections": 4, 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_active_connections": 3, 2024-04-25T16:52:45.197 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_total_time": 0.048478963, 2024-04-25T16:52:45.198 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_send_time": 0.012431505, 2024-04-25T16:52:45.198 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_recv_time": 0.018820595, 2024-04-25T16:52:45.198 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_running_fast_dispatch_time": 0.013502248, 2024-04-25T16:52:45.198 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:45.198 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 536, 2024-04-25T16:52:45.198 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.015354018, 2024-04-25T16:52:45.198 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000028645 2024-04-25T16:52:45.198 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.199 INFO:teuthology.orchestra.run.smithi173.stdout: "cct": { 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: "total_workers": 1, 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: "unhealthy_workers": 0 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-MDSRank": { 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 2541, 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.077666240, 2024-04-25T16:52:45.200 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000030565 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: "queue_len": 0, 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: "complete_latency": { 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.201 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.202 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.202 INFO:teuthology.orchestra.run.smithi173.stdout: "mds": { 2024-04-25T16:52:45.202 INFO:teuthology.orchestra.run.smithi173.stdout: "request": 0, 2024-04-25T16:52:45.202 INFO:teuthology.orchestra.run.smithi173.stdout: "reply": 0, 2024-04-25T16:52:45.202 INFO:teuthology.orchestra.run.smithi173.stdout: "reply_latency": { 2024-04-25T16:52:45.202 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.202 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.202 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_reply": 0, 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: "forward": 0, 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_complete": 0, 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_fetch_keys": 0, 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_commit": 0, 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_split": 0, 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_merge": 0, 2024-04-25T16:52:45.203 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes": 3403, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_top": 49, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_bottom": 3354, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_pinned": 3332, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_expired": 0, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "inodes_with_caps": 0, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "caps": 0, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "subtrees": 126, 2024-04-25T16:52:45.204 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse": 0, 2024-04-25T16:52:45.205 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_hit": 0, 2024-04-25T16:52:45.205 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_forward": 0, 2024-04-25T16:52:45.205 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_discover": 0, 2024-04-25T16:52:45.205 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:45.205 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:45.205 INFO:teuthology.orchestra.run.smithi173.stdout: "traverse_lock": 0, 2024-04-25T16:52:45.205 INFO:teuthology.orchestra.run.smithi173.stdout: "load_cent": 0, 2024-04-25T16:52:45.205 INFO:teuthology.orchestra.run.smithi173.stdout: "q": 0, 2024-04-25T16:52:45.206 INFO:teuthology.orchestra.run.smithi173.stdout: "exported": 0, 2024-04-25T16:52:45.206 INFO:teuthology.orchestra.run.smithi173.stdout: "exported_inodes": 0, 2024-04-25T16:52:45.206 INFO:teuthology.orchestra.run.smithi173.stdout: "imported": 0, 2024-04-25T16:52:45.206 INFO:teuthology.orchestra.run.smithi173.stdout: "imported_inodes": 0, 2024-04-25T16:52:45.206 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:45.206 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:52:45.206 INFO:teuthology.orchestra.run.smithi173.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:45.206 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rfiles": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rbytes": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "root_rsnaps": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:45.207 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps": 0, 2024-04-25T16:52:45.208 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_cap_release": 0, 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: "process_request_cap_release": 0 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_cache": { 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update": 0, 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_try_discover": 0, 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_send_discover": 0, 2024-04-25T16:52:45.209 INFO:teuthology.orchestra.run.smithi173.stdout: "dir_handle_discover": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_created": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_enqueued": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "strays_migrated": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:45.210 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_started": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "recovery_completed": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_exportdir": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_flush": 0, 2024-04-25T16:52:45.211 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:52:45.212 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:45.212 INFO:teuthology.orchestra.run.smithi173.stdout: "ireq_inodestats": 0 2024-04-25T16:52:45.212 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.212 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_log": { 2024-04-25T16:52:45.212 INFO:teuthology.orchestra.run.smithi173.stdout: "evlrg": 0, 2024-04-25T16:52:45.212 INFO:teuthology.orchestra.run.smithi173.stdout: "evadd": 0, 2024-04-25T16:52:45.212 INFO:teuthology.orchestra.run.smithi173.stdout: "evex": 0, 2024-04-25T16:52:45.212 INFO:teuthology.orchestra.run.smithi173.stdout: "evtrm": 0, 2024-04-25T16:52:45.213 INFO:teuthology.orchestra.run.smithi173.stdout: "ev": 4014, 2024-04-25T16:52:45.213 INFO:teuthology.orchestra.run.smithi173.stdout: "evexg": 0, 2024-04-25T16:52:45.213 INFO:teuthology.orchestra.run.smithi173.stdout: "evexd": 0, 2024-04-25T16:52:45.213 INFO:teuthology.orchestra.run.smithi173.stdout: "segadd": 0, 2024-04-25T16:52:45.213 INFO:teuthology.orchestra.run.smithi173.stdout: "segex": 0, 2024-04-25T16:52:45.213 INFO:teuthology.orchestra.run.smithi173.stdout: "segtrm": 0, 2024-04-25T16:52:45.213 INFO:teuthology.orchestra.run.smithi173.stdout: "seg": 6, 2024-04-25T16:52:45.213 INFO:teuthology.orchestra.run.smithi173.stdout: "segmjr": 2, 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "segexg": 0, 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "segexd": 0, 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "expos": 4194304, 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "wrpos": 13325837, 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "rdpos": 13325263, 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "jlat": { 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.214 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.215 INFO:teuthology.orchestra.run.smithi173.stdout: "replayed": 4014 2024-04-25T16:52:45.215 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.215 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_mem": { 2024-04-25T16:52:45.215 INFO:teuthology.orchestra.run.smithi173.stdout: "ino": 3405, 2024-04-25T16:52:45.215 INFO:teuthology.orchestra.run.smithi173.stdout: "ino+": 3406, 2024-04-25T16:52:45.215 INFO:teuthology.orchestra.run.smithi173.stdout: "ino-": 1, 2024-04-25T16:52:45.215 INFO:teuthology.orchestra.run.smithi173.stdout: "dir": 502, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "dir+": 523, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "dir-": 21, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "dn": 3403, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "dn+": 3403, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "dn-": 0, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "cap": 0, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "cap+": 0, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "cap-": 0, 2024-04-25T16:52:45.216 INFO:teuthology.orchestra.run.smithi173.stdout: "rss": 131304, 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "heap": 215440 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_server": { 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_client_request": 0, 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "dispatch_server_request": 0, 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_request": 0, 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_client_session": 0, 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "handle_peer_request": 0, 2024-04-25T16:52:45.217 INFO:teuthology.orchestra.run.smithi173.stdout: "req_create_latency": { 2024-04-25T16:52:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getattr_latency": { 2024-04-25T16:52:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.219 INFO:teuthology.orchestra.run.smithi173.stdout: "req_link_latency": { 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookup_latency": { 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.220 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupino_latency": { 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.221 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupname_latency": { 2024-04-25T16:52:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:45.222 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.223 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "req_lssnap_latency": { 2024-04-25T16:52:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.224 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mkdir_latency": { 2024-04-25T16:52:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mknod_latency": { 2024-04-25T16:52:45.225 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "req_mksnap_latency": { 2024-04-25T16:52:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.226 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "req_open_latency": { 2024-04-25T16:52:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "req_readdir_latency": { 2024-04-25T16:52:45.227 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rename_latency": { 2024-04-25T16:52:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.228 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.229 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmdir_latency": { 2024-04-25T16:52:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.230 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.231 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setattr_latency": { 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.232 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setlayout_latency": { 2024-04-25T16:52:45.233 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "req_setxattr_latency": { 2024-04-25T16:52:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.234 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "req_symlink_latency": { 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "req_unlink_latency": { 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.235 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.236 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_sessions": { 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "session_count": 1, 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "session_add": 2, 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "session_remove": 1, 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_open": 1, 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "sessions_stale": 0, 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "total_load": 0, 2024-04-25T16:52:45.237 INFO:teuthology.orchestra.run.smithi173.stdout: "average_load": 0, 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "avg_session_uptime": 2331, 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "mempool": { 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:45.238 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:45.239 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:45.240 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:45.241 INFO:teuthology.orchestra.run.smithi173.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_items": 0, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_bytes": 190414, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_anon_items": 51, 2024-04-25T16:52:45.242 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_bytes": 0, 2024-04-25T16:52:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_items": 0, 2024-04-25T16:52:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:45.243 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_items": 276, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "pgmap_items": 0, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_bytes": 10294201, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "mds_co_items": 49926, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:45.244 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_1_items": 0, 2024-04-25T16:52:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "unittest_2_items": 0 2024-04-25T16:52:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "objecter": { 2024-04-25T16:52:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "op_active": 0, 2024-04-25T16:52:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "op_laggy": 0, 2024-04-25T16:52:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send": 1568, 2024-04-25T16:52:45.245 INFO:teuthology.orchestra.run.smithi173.stdout: "op_send_bytes": 21, 2024-04-25T16:52:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_resend": 0, 2024-04-25T16:52:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_reply": 1568, 2024-04-25T16:52:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_latency": { 2024-04-25T16:52:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1568, 2024-04-25T16:52:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1.965959803, 2024-04-25T16:52:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.001253800 2024-04-25T16:52:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.246 INFO:teuthology.orchestra.run.smithi173.stdout: "op_inflight": 0, 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "oplen_avg": { 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 1568, 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 1569 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "op": 1568, 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "op_r": 1568, 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "op_w": 0, 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "op_rmw": 0, 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "op_pg": 0, 2024-04-25T16:52:45.247 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_stat": 1030, 2024-04-25T16:52:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_create": 0, 2024-04-25T16:52:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_read": 536, 2024-04-25T16:52:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_write": 0, 2024-04-25T16:52:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writefull": 0, 2024-04-25T16:52:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_writesame": 0, 2024-04-25T16:52:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_append": 0, 2024-04-25T16:52:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_zero": 0, 2024-04-25T16:52:45.248 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_truncate": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_delete": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_mapext": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_getxattr": 1, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_setxattr": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:45.249 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_call": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_watch": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_notify": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "osdop_other": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_active": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_send": 0, 2024-04-25T16:52:45.250 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_resend": 0, 2024-04-25T16:52:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "linger_ping": 0, 2024-04-25T16:52:45.253 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_active": 0, 2024-04-25T16:52:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_send": 0, 2024-04-25T16:52:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "poolop_resend": 0, 2024-04-25T16:52:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_active": 0, 2024-04-25T16:52:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_send": 0, 2024-04-25T16:52:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "poolstat_resend": 0, 2024-04-25T16:52:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_active": 0, 2024-04-25T16:52:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_send": 0, 2024-04-25T16:52:45.254 INFO:teuthology.orchestra.run.smithi173.stdout: "statfs_resend": 0, 2024-04-25T16:52:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "command_active": 0, 2024-04-25T16:52:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "command_send": 0, 2024-04-25T16:52:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "command_resend": 0, 2024-04-25T16:52:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "map_epoch": 106, 2024-04-25T16:52:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "map_full": 0, 2024-04-25T16:52:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "map_inc": 3, 2024-04-25T16:52:45.255 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_sessions": 5, 2024-04-25T16:52:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_open": 5, 2024-04-25T16:52:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_session_close": 0, 2024-04-25T16:52:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "osd_laggy": 0, 2024-04-25T16:52:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_wr": 0, 2024-04-25T16:52:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_rd": 2, 2024-04-25T16:52:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_del": 0 2024-04-25T16:52:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.256 INFO:teuthology.orchestra.run.smithi173.stdout: "oft": { 2024-04-25T16:52:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_objs": 0, 2024-04-25T16:52:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_updates": 0, 2024-04-25T16:52:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "omap_total_removes": 0 2024-04-25T16:52:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "osd-slow-ops": { 2024-04-25T16:52:45.257 INFO:teuthology.orchestra.run.smithi173.stdout: "slow_ops_count": 0 2024-04-25T16:52:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "purge_queue": { 2024-04-25T16:52:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:52:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing": 0, 2024-04-25T16:52:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executing_high_water": 0, 2024-04-25T16:52:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed_ops": 0, 2024-04-25T16:52:45.258 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_executed": 0, 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 1738, 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 9739971, 2024-04-25T16:52:45.259 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 1738, 2024-04-25T16:52:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1738, 2024-04-25T16:52:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 9739971, 2024-04-25T16:52:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.260 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 104857600, 2024-04-25T16:52:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:45.261 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 1568, 2024-04-25T16:52:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 9131538, 2024-04-25T16:52:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1568, 2024-04-25T16:52:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 9131538, 2024-04-25T16:52:45.262 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 1024, 2024-04-25T16:52:45.263 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 1568, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 1568, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 1568, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 1568, 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:45.264 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:45.265 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-25T16:52:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-25T16:52:45.266 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: }, 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "throttle-write_buf_throttle-0x563fd1ee6c30": { 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "val": 0, 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "max": 3758096384, 2024-04-25T16:52:45.267 INFO:teuthology.orchestra.run.smithi173.stdout: "get_started": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "get": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "get_sum": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "take": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "take_sum": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "put": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "put_sum": 0, 2024-04-25T16:52:45.268 INFO:teuthology.orchestra.run.smithi173.stdout: "wait": { 2024-04-25T16:52:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgcount": 0, 2024-04-25T16:52:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "sum": 0.000000000, 2024-04-25T16:52:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: "avgtime": 0.000000000 2024-04-25T16:52:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.269 INFO:teuthology.orchestra.run.smithi173.stdout: } 2024-04-25T16:52:45.269 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-04-25T16:52:45.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:45 smithi203 ceph-mon[33115]: pgmap v984: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:52:45.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:45 smithi098 ceph-mon[29162]: pgmap v984: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:52:45.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:45 smithi173 ceph-mon[32556]: pgmap v984: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:52:46.113 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-25T16:52:46.113 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-25T16:52:46.113 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-04-25T16:52:46.113 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T16:52:46.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T16:52:46.439 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:52:47.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:47 smithi203 ceph-mon[33115]: pgmap v985: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:52:47.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:47 smithi098 ceph-mon[29162]: pgmap v985: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:52:47.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:47 smithi173 ceph-mon[32556]: pgmap v985: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:52:48.156 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T16:52:48.156 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:48.157 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 284, 2024-04-25T16:52:48.157 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 275, 2024-04-25T16:52:48.157 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 8984308, 2024-04-25T16:52:48.157 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 474657, 2024-04-25T16:52:48.157 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 10, 2024-04-25T16:52:48.157 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-25T16:52:48.157 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.034777390, 2024-04-25T16:52:48.157 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.010301146, 2024-04-25T16:52:48.158 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.017415687, 2024-04-25T16:52:48.158 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.005791550, 2024-04-25T16:52:48.158 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:48.158 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 273, 2024-04-25T16:52:48.158 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.015506610, 2024-04-25T16:52:48.158 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000056800 2024-04-25T16:52:48.158 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.158 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 141, 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000027152, 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000192 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:48.159 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 840, 2024-04-25T16:52:48.160 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 844, 2024-04-25T16:52:48.160 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 606571, 2024-04-25T16:52:48.160 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1054366, 2024-04-25T16:52:48.160 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 9, 2024-04-25T16:52:48.160 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-04-25T16:52:48.160 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.064719636, 2024-04-25T16:52:48.160 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.023985343, 2024-04-25T16:52:48.160 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.044445348, 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.004149262, 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 842, 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.024091285, 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000028611 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1199, 2024-04-25T16:52:48.161 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000193188, 2024-04-25T16:52:48.162 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000161 2024-04-25T16:52:48.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.162 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 392416, 2024-04-25T16:52:48.162 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 95696 2024-04-25T16:52:48.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.162 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:48.162 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 3256, 2024-04-25T16:52:48.162 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 1797, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 1664697, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1071039, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 11, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 5, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.163409911, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.063360073, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.129419263, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.002624994, 2024-04-25T16:52:48.163 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1772, 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.059321156, 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000033476 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 4589, 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001014655, 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000221 2024-04-25T16:52:48.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.165 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:52:48.165 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 283728 2024-04-25T16:52:48.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.165 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-25T16:52:48.165 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-25T16:52:48.165 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-25T16:52:48.165 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.165 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 301, 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.093533843, 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000310743 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:48.166 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T16:52:48.167 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T16:52:48.167 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5, 2024-04-25T16:52:48.167 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000078142, 2024-04-25T16:52:48.167 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000015628 2024-04-25T16:52:48.167 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.167 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.167 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-25T16:52:48.167 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 553, 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 516, 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 516, 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.189777748, 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000367786 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 37, 2024-04-25T16:52:48.168 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 11, 2024-04-25T16:52:48.169 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 9, 2024-04-25T16:52:48.169 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-25T16:52:48.169 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-25T16:52:48.169 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-25T16:52:48.169 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 3412, 2024-04-25T16:52:48.169 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 18, 2024-04-25T16:52:48.169 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 3394, 2024-04-25T16:52:48.169 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 3385, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 504, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 504, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 144, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 857, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 820, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 30, 2024-04-25T16:52:48.170 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 6, 2024-04-25T16:52:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-25T16:52:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-25T16:52:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-25T16:52:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-25T16:52:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-25T16:52:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 6, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 73, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 84381, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 1535916169, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:48.172 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:48.173 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:48.173 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:48.173 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:48.173 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:48.173 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:48.173 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:48.173 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:48.173 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 5, 2024-04-25T16:52:48.174 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:52:48.174 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:52:48.174 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:48.174 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:48.174 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 5, 2024-04-25T16:52:48.174 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:52:48.174 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 43, 2024-04-25T16:52:48.174 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 6, 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-25T16:52:48.175 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:48.176 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:48.176 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-25T16:52:48.176 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-25T16:52:48.176 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:48.176 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-25T16:52:48.176 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:48.176 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:48.176 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:52:48.177 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:48.178 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-25T16:52:48.178 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.178 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-25T16:52:48.178 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-25T16:52:48.178 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 202, 2024-04-25T16:52:48.178 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-25T16:52:48.178 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-25T16:52:48.178 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 4014, 2024-04-25T16:52:48.179 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-25T16:52:48.179 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-25T16:52:48.179 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-04-25T16:52:48.179 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-25T16:52:48.179 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-25T16:52:48.179 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 6, 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 13325837, 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 13030259, 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 86, 2024-04-25T16:52:48.180 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 2.341880740, 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.027231171 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 3812 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 3414, 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 3415, 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 1, 2024-04-25T16:52:48.181 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 546, 2024-04-25T16:52:48.182 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 572, 2024-04-25T16:52:48.182 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 26, 2024-04-25T16:52:48.182 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 3412, 2024-04-25T16:52:48.182 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 3412, 2024-04-25T16:52:48.182 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 0, 2024-04-25T16:52:48.182 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 504, 2024-04-25T16:52:48.182 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 1731, 2024-04-25T16:52:48.182 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 1227, 2024-04-25T16:52:48.183 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 134548, 2024-04-25T16:52:48.183 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 240016 2024-04-25T16:52:48.183 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.183 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-25T16:52:48.183 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 558, 2024-04-25T16:52:48.183 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-25T16:52:48.183 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 553, 2024-04-25T16:52:48.184 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 6, 2024-04-25T16:52:48.184 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-25T16:52:48.184 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-25T16:52:48.184 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.184 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.184 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 283, 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.079931675, 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000282444 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.185 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.186 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.186 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-25T16:52:48.186 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.186 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.186 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.186 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.186 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 15, 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.008055717, 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000537047 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.187 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.188 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.189 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:48.189 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.189 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.189 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.189 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.189 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:48.189 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.190 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.190 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.190 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.190 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-25T16:52:48.190 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.190 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.191 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.191 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.191 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-25T16:52:48.191 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.191 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-25T16:52:48.192 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.193 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.193 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.193 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-25T16:52:48.193 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.193 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.193 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.193 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.194 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-25T16:52:48.194 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 218, 2024-04-25T16:52:48.194 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.101790356, 2024-04-25T16:52:48.194 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000466928 2024-04-25T16:52:48.194 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.194 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-25T16:52:48.194 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.194 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:48.195 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.196 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.196 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.196 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-25T16:52:48.196 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.196 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.196 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.196 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.197 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.198 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-25T16:52:48.198 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.198 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.198 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.198 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.198 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:48.198 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.199 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.199 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.199 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.199 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:48.199 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.199 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.200 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.200 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-25T16:52:48.200 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.200 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.200 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.200 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.200 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.201 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.202 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-25T16:52:48.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.202 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.202 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.202 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.203 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:48.203 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:48.203 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:48.203 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.203 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.204 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.204 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.204 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.204 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-25T16:52:48.204 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-25T16:52:48.204 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-04-25T16:52:48.204 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-04-25T16:52:48.204 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-04-25T16:52:48.205 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-25T16:52:48.205 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 7, 2024-04-25T16:52:48.205 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 7, 2024-04-25T16:52:48.205 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2319, 2024-04-25T16:52:48.205 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:48.205 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.205 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-25T16:52:48.206 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:48.206 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:48.206 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:48.206 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:48.206 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:48.206 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:48.206 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:48.206 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:48.207 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:48.207 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:48.207 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:48.207 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:48.207 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:48.207 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:48.207 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:48.208 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:48.208 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:48.208 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:48.208 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:48.208 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:48.208 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:48.209 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:48.209 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:48.209 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:48.209 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:48.209 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:48.209 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:48.209 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:48.210 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:48.210 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:48.210 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:48.210 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-25T16:52:48.210 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:48.210 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:48.211 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:48.211 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:48.211 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 266851, 2024-04-25T16:52:48.211 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 114, 2024-04-25T16:52:48.211 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:48.211 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:48.211 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-25T16:52:48.212 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-25T16:52:48.212 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:48.212 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:48.212 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:48.212 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:48.212 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:48.212 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 276, 2024-04-25T16:52:48.213 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:48.213 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:48.213 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:48.213 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-25T16:52:48.213 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 10599513, 2024-04-25T16:52:48.213 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 54611, 2024-04-25T16:52:48.213 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:48.214 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-25T16:52:48.214 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:48.214 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-25T16:52:48.214 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.214 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-25T16:52:48.214 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-25T16:52:48.214 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-25T16:52:48.214 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 245, 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 320237, 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 0, 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 245, 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 245, 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 2.339944734, 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.009550794 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-25T16:52:48.215 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 245, 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 327 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 245, 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 114, 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 131, 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-25T16:52:48.216 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 46, 2024-04-25T16:52:48.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-25T16:52:48.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 34, 2024-04-25T16:52:48.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 86, 2024-04-25T16:52:48.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 4, 2024-04-25T16:52:48.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-25T16:52:48.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-25T16:52:48.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 1, 2024-04-25T16:52:48.217 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 5, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 21, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:48.218 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:48.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-25T16:52:48.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-25T16:52:48.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-25T16:52:48.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:48.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-25T16:52:48.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:48.219 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-25T16:52:48.219 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-25T16:52:48.220 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 106, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 5, 2024-04-25T16:52:48.221 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 9, 2024-04-25T16:52:48.222 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 9, 2024-04-25T16:52:48.222 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-25T16:52:48.222 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-25T16:52:48.222 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 53, 2024-04-25T16:52:48.222 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 55, 2024-04-25T16:52:48.222 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 22 2024-04-25T16:52:48.222 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.222 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.223 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-25T16:52:48.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:48.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:52:48.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-25T16:52:48.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-25T16:52:48.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-25T16:52:48.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-25T16:52:48.224 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:48.224 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 4376, 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 10849075, 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 4376, 2024-04-25T16:52:48.225 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 4376, 2024-04-25T16:52:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 10849075, 2024-04-25T16:52:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.226 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:48.227 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 245, 2024-04-25T16:52:48.228 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 9157302, 2024-04-25T16:52:48.228 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 245, 2024-04-25T16:52:48.228 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 9157302, 2024-04-25T16:52:48.228 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:48.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.228 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.228 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.228 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.229 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.229 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:48.229 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:48.229 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-25T16:52:48.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:48.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:48.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:48.229 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 245, 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 245, 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 245, 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 245, 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.230 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.231 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.231 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.231 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:48.231 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:48.231 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T16:52:48.231 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:48.231 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:48.231 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.232 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.233 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.233 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.233 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:48.233 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x5598d909c750": { 2024-04-25T16:52:48.233 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:48.233 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T16:52:48.233 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:48.233 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 202, 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 293927, 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 202, 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 86, 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 293927, 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:48.234 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:48.235 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:48.235 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:48.235 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.235 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:48.235 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T16:52:49.103 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-25T16:52:49.103 INFO:tasks.check_counter:Daemon mds.c mds.imported=6 2024-04-25T16:52:49.103 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-04-25T16:52:49.104 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T16:52:49.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T16:52:49.362 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:52:49.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:49 smithi203 ceph-mon[33115]: pgmap v986: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-25T16:52:49.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:49 smithi098 ceph-mon[29162]: pgmap v986: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-25T16:52:49.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:49 smithi173 ceph-mon[32556]: pgmap v986: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-04-25T16:52:51.091 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T16:52:51.091 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:51.091 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 167, 2024-04-25T16:52:51.091 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 166, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 8603812, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 58172, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 6, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 2, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.029360386, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.005662905, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.016639395, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.005886103, 2024-04-25T16:52:51.092 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:51.093 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 165, 2024-04-25T16:52:51.093 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.007989663, 2024-04-25T16:52:51.093 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000048422 2024-04-25T16:52:51.093 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.093 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:51.093 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.093 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.093 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.094 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 175136, 2024-04-25T16:52:51.094 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 14288 2024-04-25T16:52:51.094 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.094 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:51.094 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 90, 2024-04-25T16:52:51.094 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 107, 2024-04-25T16:52:51.094 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 13655380, 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 140237, 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.019469472, 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.003977502, 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.012287240, 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.002207480, 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:51.095 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 105, 2024-04-25T16:52:51.096 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.005312665, 2024-04-25T16:52:51.096 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000050596 2024-04-25T16:52:51.096 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.096 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:51.096 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.096 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.096 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.097 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:52:51.097 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 111872 2024-04-25T16:52:51.097 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.097 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:51.097 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 10, 2024-04-25T16:52:51.097 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 10, 2024-04-25T16:52:51.097 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 12587405, 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 2362, 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.011356712, 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.000108088, 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.010948514, 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.000184093, 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:51.098 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 9, 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.004228685, 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000469853 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.099 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 2960, 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T16:52:51.100 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 377, 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.013270532, 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000035200 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T16:52:51.101 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-25T16:52:51.102 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.103 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.103 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.103 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.103 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-25T16:52:51.103 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-04-25T16:52:51.103 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-25T16:52:51.103 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-25T16:52:51.104 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-25T16:52:51.104 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-25T16:52:51.104 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-25T16:52:51.104 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 10734, 2024-04-25T16:52:51.104 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 58, 2024-04-25T16:52:51.104 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 10676, 2024-04-25T16:52:51.104 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:51.104 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 10652, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 156, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-04-25T16:52:51.105 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:51.106 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:51.106 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-25T16:52:51.106 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-25T16:52:51.106 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-25T16:52:51.106 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-25T16:52:51.106 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-25T16:52:51.106 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-25T16:52:51.106 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:51.107 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:52:51.108 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:52:51.109 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:51.109 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:51.109 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-04-25T16:52:51.109 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:52:51.109 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-25T16:52:51.109 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-25T16:52:51.109 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.109 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:51.110 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-25T16:52:51.111 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:51.112 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:51.112 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:51.112 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-25T16:52:51.112 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-25T16:52:51.112 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:52:51.112 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:51.112 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-25T16:52:51.112 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 16004, 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-25T16:52:51.113 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-04-25T16:52:51.114 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-25T16:52:51.114 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-25T16:52:51.114 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 21, 2024-04-25T16:52:51.114 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 3, 2024-04-25T16:52:51.114 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-25T16:52:51.114 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-25T16:52:51.114 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-25T16:52:51.114 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 38797410, 2024-04-25T16:52:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 38796836, 2024-04-25T16:52:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-25T16:52:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 16004 2024-04-25T16:52:51.115 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.116 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-25T16:52:51.116 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 10200, 2024-04-25T16:52:51.116 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 10844, 2024-04-25T16:52:51.116 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 644, 2024-04-25T16:52:51.116 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 836, 2024-04-25T16:52:51.116 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 878, 2024-04-25T16:52:51.116 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 42, 2024-04-25T16:52:51.116 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 10734, 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 11103, 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 369, 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 154168, 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215440 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.117 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-25T16:52:51.118 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-04-25T16:52:51.118 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-25T16:52:51.118 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-04-25T16:52:51.118 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-04-25T16:52:51.118 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-25T16:52:51.118 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-25T16:52:51.118 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.118 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:51.119 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.120 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.120 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.120 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-25T16:52:51.120 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.120 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.120 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.120 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.121 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.122 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-25T16:52:51.122 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.122 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.122 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.122 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.122 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-25T16:52:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.123 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.124 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.125 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.125 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.125 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-25T16:52:51.126 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.126 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.126 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.126 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.126 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-25T16:52:51.126 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.126 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.126 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.127 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.128 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.128 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-25T16:52:51.128 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.128 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.128 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.128 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.128 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.129 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.130 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:51.130 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.130 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.130 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.130 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.130 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-25T16:52:51.130 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.130 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:51.131 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.132 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.132 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.132 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-25T16:52:51.132 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.132 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.132 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.132 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.133 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.134 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-25T16:52:51.134 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.134 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.134 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.134 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.134 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-25T16:52:51.134 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.135 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-25T16:52:51.136 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.136 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.136 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.136 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.136 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:51.136 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:51.136 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:51.136 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 1, 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 0, 2024-04-25T16:52:51.137 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-04-25T16:52:51.138 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-25T16:52:51.138 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-04-25T16:52:51.138 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-04-25T16:52:51.138 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2320, 2024-04-25T16:52:51.138 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:51.138 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.138 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-25T16:52:51.138 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:51.139 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:51.139 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:51.139 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:51.139 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:51.139 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:51.139 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:51.139 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:51.139 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:51.140 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:51.140 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:51.140 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:51.140 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:51.140 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:51.140 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:51.141 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:51.141 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:51.141 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:51.141 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:51.141 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:51.141 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:51.141 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:51.141 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:51.142 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:51.142 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:51.142 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:51.142 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:51.142 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:51.142 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:51.143 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:51.143 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:51.143 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-25T16:52:51.143 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:51.143 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:51.143 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:51.143 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:51.143 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 204639, 2024-04-25T16:52:51.144 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 54, 2024-04-25T16:52:51.144 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:51.144 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:51.144 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-25T16:52:51.144 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-25T16:52:51.144 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:51.144 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:51.144 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:51.145 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:51.145 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:51.145 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 276, 2024-04-25T16:52:51.145 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:51.145 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:51.145 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:51.145 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-25T16:52:51.145 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 30305019, 2024-04-25T16:52:51.146 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 148551, 2024-04-25T16:52:51.146 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:51.146 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-25T16:52:51.146 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:51.146 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-25T16:52:51.146 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.146 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 233, 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 0, 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 233, 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 233, 2024-04-25T16:52:51.147 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.247994173, 2024-04-25T16:52:51.148 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001064352 2024-04-25T16:52:51.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.148 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-25T16:52:51.148 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-25T16:52:51.148 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 233, 2024-04-25T16:52:51.148 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 234 2024-04-25T16:52:51.148 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.148 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 233, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 233, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 136, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 95, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-04-25T16:52:51.149 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-04-25T16:52:51.150 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-25T16:52:51.150 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-25T16:52:51.150 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-25T16:52:51.150 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-25T16:52:51.150 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-04-25T16:52:51.150 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-25T16:52:51.150 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:51.150 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:51.151 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-04-25T16:52:51.151 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-25T16:52:51.151 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:51.151 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:51.151 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:51.151 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-25T16:52:51.151 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-25T16:52:51.151 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-25T16:52:51.152 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-25T16:52:51.153 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-25T16:52:51.153 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-25T16:52:51.153 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-25T16:52:51.153 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-25T16:52:51.153 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-25T16:52:51.153 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-25T16:52:51.153 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-25T16:52:51.154 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-25T16:52:51.154 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-25T16:52:51.154 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-25T16:52:51.154 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-25T16:52:51.154 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 0, 2024-04-25T16:52:51.154 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-25T16:52:51.154 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 0, 2024-04-25T16:52:51.154 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 7, 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 7, 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-25T16:52:51.155 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-25T16:52:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-25T16:52:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-25T16:52:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-25T16:52:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-25T16:52:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.156 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.157 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T16:52:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 263, 2024-04-25T16:52:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 34818484, 2024-04-25T16:52:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 263, 2024-04-25T16:52:51.158 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 263, 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 34818484, 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.159 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:51.160 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 233, 2024-04-25T16:52:51.161 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 34603111, 2024-04-25T16:52:51.161 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 233, 2024-04-25T16:52:51.161 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 34603111, 2024-04-25T16:52:51.161 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:51.161 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.161 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.161 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.161 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.162 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.162 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:51.162 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:51.162 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-25T16:52:51.162 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:51.162 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:51.162 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:51.162 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 233, 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 233, 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 233, 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 233, 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.163 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:51.164 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.165 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.166 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.166 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:51.166 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55c52d2028f0": { 2024-04-25T16:52:51.166 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:51.166 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T16:52:51.166 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:51.166 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:51.166 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:51.167 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:51.168 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.168 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:51.168 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T16:52:51.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:51 smithi203 ceph-mon[33115]: pgmap v987: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T16:52:51.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:51 smithi098 ceph-mon[29162]: pgmap v987: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T16:52:51.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:51 smithi173 ceph-mon[32556]: pgmap v987: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-25T16:52:51.825 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T16:52:51.826 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T16:52:51.826 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-04-25T16:52:51.826 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T16:52:51.826 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T16:52:52.066 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:52:53.528 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:53 smithi203 ceph-mon[33115]: pgmap v988: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T16:52:53.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:53 smithi098 ceph-mon[29162]: pgmap v988: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T16:52:53.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:53 smithi173 ceph-mon[32556]: pgmap v988: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-04-25T16:52:53.809 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T16:52:53.809 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:53.809 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 3881, 2024-04-25T16:52:53.809 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 2079, 2024-04-25T16:52:53.809 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 9788500, 2024-04-25T16:52:53.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1342249, 2024-04-25T16:52:53.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 16, 2024-04-25T16:52:53.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 0, 2024-04-25T16:52:53.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.196352058, 2024-04-25T16:52:53.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.068797319, 2024-04-25T16:52:53.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.163748772, 2024-04-25T16:52:53.810 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.007880557, 2024-04-25T16:52:53.811 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:53.811 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2074, 2024-04-25T16:52:53.811 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.070680974, 2024-04-25T16:52:53.811 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000034079 2024-04-25T16:52:53.811 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.811 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:53.811 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 5268, 2024-04-25T16:52:53.811 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.001128470, 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000214 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 2960, 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 1377, 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 1448, 2024-04-25T16:52:53.812 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 872789, 2024-04-25T16:52:53.813 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 1414095, 2024-04-25T16:52:53.813 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 12, 2024-04-25T16:52:53.813 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 8, 2024-04-25T16:52:53.813 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.103915892, 2024-04-25T16:52:53.813 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.039258403, 2024-04-25T16:52:53.813 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.069609942, 2024-04-25T16:52:53.813 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.007737331, 2024-04-25T16:52:53.813 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:53.814 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1443, 2024-04-25T16:52:53.814 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.032145341, 2024-04-25T16:52:53.814 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000022276 2024-04-25T16:52:53.814 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.814 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:53.814 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2053, 2024-04-25T16:52:53.814 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000331535, 2024-04-25T16:52:53.814 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000161 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 389728, 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 95888 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 117, 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 227, 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 82287, 2024-04-25T16:52:53.815 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 354133, 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 9, 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 6, 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.017139560, 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.007323876, 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.007493465, 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.001949561, 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 225, 2024-04-25T16:52:53.816 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.015510197, 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000068934 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 83, 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000018598, 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000224 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:52:53.817 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 285696 2024-04-25T16:52:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-25T16:52:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-25T16:52:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-25T16:52:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-25T16:52:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T16:52:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T16:52:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 473, 2024-04-25T16:52:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.106462837, 2024-04-25T16:52:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000225079 2024-04-25T16:52:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T16:52:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T16:52:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 94, 2024-04-25T16:52:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.002828105, 2024-04-25T16:52:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000030086 2024-04-25T16:52:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-25T16:52:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 530, 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 501, 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 501, 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.375754501, 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000750008 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 28, 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 11, 2024-04-25T16:52:53.821 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 9, 2024-04-25T16:52:53.822 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-25T16:52:53.822 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-25T16:52:53.822 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-25T16:52:53.822 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 3451, 2024-04-25T16:52:53.822 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 17, 2024-04-25T16:52:53.822 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 3434, 2024-04-25T16:52:53.822 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:53.823 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 3426, 2024-04-25T16:52:53.823 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-25T16:52:53.823 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 563, 2024-04-25T16:52:53.823 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 563, 2024-04-25T16:52:53.823 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 175, 2024-04-25T16:52:53.823 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 1002, 2024-04-25T16:52:53.823 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 488, 2024-04-25T16:52:53.823 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 23, 2024-04-25T16:52:53.824 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 9, 2024-04-25T16:52:53.824 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:53.824 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:53.824 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-25T16:52:53.824 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-25T16:52:53.824 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-25T16:52:53.824 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-25T16:52:53.824 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 9, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 49, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 84381, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 1535916169, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-25T16:52:53.825 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:53.826 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:53.826 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:53.826 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:53.826 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:53.826 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:53.826 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:53.826 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:53.826 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 28, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 68, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 3, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 100, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 47, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 55, 2024-04-25T16:52:53.827 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 74 2024-04-25T16:52:53.828 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.828 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-25T16:52:53.828 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-25T16:52:53.828 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:53.828 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-25T16:52:53.828 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 9, 2024-04-25T16:52:53.828 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-25T16:52:53.828 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-25T16:52:53.829 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:53.829 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:53.829 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 12, 2024-04-25T16:52:53.829 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 12, 2024-04-25T16:52:53.829 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:53.829 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-25T16:52:53.829 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:53.829 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-25T16:52:53.830 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:52:53.831 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:53.831 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-25T16:52:53.831 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.831 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-25T16:52:53.831 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-25T16:52:53.831 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 433, 2024-04-25T16:52:53.832 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-25T16:52:53.832 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-25T16:52:53.832 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 4065, 2024-04-25T16:52:53.832 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-25T16:52:53.832 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-25T16:52:53.832 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 1, 2024-04-25T16:52:53.832 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-25T16:52:53.832 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 6, 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 12557344, 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 12095196, 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-25T16:52:53.833 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 123, 2024-04-25T16:52:53.834 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1.538483112, 2024-04-25T16:52:53.834 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.012507992 2024-04-25T16:52:53.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.834 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 3632 2024-04-25T16:52:53.834 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.834 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-25T16:52:53.834 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 3437, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 3469, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 32, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 504, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 554, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 50, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 3451, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 3497, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 46, 2024-04-25T16:52:53.835 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 563, 2024-04-25T16:52:53.836 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 2118, 2024-04-25T16:52:53.836 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 1555, 2024-04-25T16:52:53.836 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 135332, 2024-04-25T16:52:53.836 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 240016 2024-04-25T16:52:53.836 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.836 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-25T16:52:53.836 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 737, 2024-04-25T16:52:53.836 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 1, 2024-04-25T16:52:53.837 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 530, 2024-04-25T16:52:53.837 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 6, 2024-04-25T16:52:53.837 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 68, 2024-04-25T16:52:53.837 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-25T16:52:53.837 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 50, 2024-04-25T16:52:53.837 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.115801093, 2024-04-25T16:52:53.837 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.002316021 2024-04-25T16:52:53.837 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.838 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-25T16:52:53.838 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 77, 2024-04-25T16:52:53.838 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.021901552, 2024-04-25T16:52:53.838 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000284435 2024-04-25T16:52:53.838 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.838 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:53.838 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.838 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-25T16:52:53.839 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 201, 2024-04-25T16:52:53.840 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.084819285, 2024-04-25T16:52:53.840 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000421986 2024-04-25T16:52:53.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.840 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:53.840 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.840 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.840 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.840 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.841 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.842 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:53.842 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.842 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.842 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.842 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.842 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:53.842 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.843 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.843 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.843 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.843 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-25T16:52:53.845 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.845 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.846 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.846 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.846 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-25T16:52:53.846 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.847 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.847 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.847 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-25T16:52:53.847 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.847 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.847 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.847 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.848 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-25T16:52:53.848 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.848 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.848 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.848 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.848 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-25T16:52:53.848 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 8, 2024-04-25T16:52:53.848 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.006391714, 2024-04-25T16:52:53.849 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000798964 2024-04-25T16:52:53.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.849 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-25T16:52:53.849 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 153, 2024-04-25T16:52:53.849 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.108953296, 2024-04-25T16:52:53.849 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000712113 2024-04-25T16:52:53.849 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.849 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.850 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.851 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:53.851 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.851 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.851 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.851 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.851 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-25T16:52:53.851 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.851 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.852 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.852 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:53.852 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.852 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.852 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.852 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.852 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:53.853 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.853 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.853 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.853 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.853 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-25T16:52:53.853 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.853 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.853 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:53.854 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.855 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.856 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.857 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-25T16:52:53.857 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 12, 2024-04-25T16:52:53.857 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.037887561, 2024-04-25T16:52:53.857 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.003157296 2024-04-25T16:52:53.857 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.857 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:53.857 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:53.857 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:53.858 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.858 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.858 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.858 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.858 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.858 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-25T16:52:53.858 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-25T16:52:53.858 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 26, 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 26, 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2326, 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.859 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:53.860 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:53.861 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:53.861 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:53.861 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:53.861 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:53.861 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:53.861 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:53.861 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:53.861 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:53.862 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:53.863 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:53.863 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:53.863 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:53.863 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:53.863 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:53.863 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-25T16:52:53.863 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:53.863 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 283812, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 118, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-25T16:52:53.864 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:53.865 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:53.865 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:53.865 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:53.865 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:53.865 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 276, 2024-04-25T16:52:53.865 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:53.865 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:53.865 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:53.866 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-25T16:52:53.866 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 10646496, 2024-04-25T16:52:53.866 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 59544, 2024-04-25T16:52:53.866 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:53.866 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-25T16:52:53.866 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:53.866 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-25T16:52:53.866 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 391, 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 494219, 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 0, 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 391, 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-25T16:52:53.867 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 391, 2024-04-25T16:52:53.868 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1.986953120, 2024-04-25T16:52:53.868 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.005081721 2024-04-25T16:52:53.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.868 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-25T16:52:53.868 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-25T16:52:53.868 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 391, 2024-04-25T16:52:53.868 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 519 2024-04-25T16:52:53.868 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 391, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 122, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 269, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 46, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 43, 2024-04-25T16:52:53.869 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 135, 2024-04-25T16:52:53.870 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 22, 2024-04-25T16:52:53.870 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-25T16:52:53.870 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-25T16:52:53.870 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 2, 2024-04-25T16:52:53.870 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-25T16:52:53.870 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 34, 2024-04-25T16:52:53.870 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-25T16:52:53.870 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:53.871 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:53.871 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 21, 2024-04-25T16:52:53.871 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-25T16:52:53.871 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:53.871 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:53.871 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:53.871 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-25T16:52:53.871 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 3, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-25T16:52:53.872 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-25T16:52:53.873 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-25T16:52:53.873 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-25T16:52:53.873 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-25T16:52:53.873 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-25T16:52:53.873 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-25T16:52:53.873 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-25T16:52:53.873 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-25T16:52:53.873 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-25T16:52:53.874 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-25T16:52:53.874 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-25T16:52:53.874 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-25T16:52:53.874 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-25T16:52:53.874 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 106, 2024-04-25T16:52:53.874 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-25T16:52:53.874 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 5, 2024-04-25T16:52:53.874 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 12, 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 12, 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 115, 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 54, 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 44 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-25T16:52:53.875 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-25T16:52:53.876 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:53.876 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-25T16:52:53.876 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-25T16:52:53.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.876 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-25T16:52:53.876 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 1 2024-04-25T16:52:53.876 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.876 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 4, 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 2, 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 24, 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 12, 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.877 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 5371, 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 10247473, 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 5371, 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:53.878 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:53.879 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 5371, 2024-04-25T16:52:53.879 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 10247473, 2024-04-25T16:52:53.879 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:53.879 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.879 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.879 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.879 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.879 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T16:52:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:53.880 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 391, 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 8397043, 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 391, 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 8397043, 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.881 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.882 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.882 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:53.882 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:53.882 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-25T16:52:53.882 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:53.882 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:53.882 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:53.882 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 391, 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 391, 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 391, 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 391, 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.883 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 12, 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 1308, 2024-04-25T16:52:53.884 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 12, 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 12, 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1308, 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.885 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.886 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.886 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:53.886 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x5556cb5e6c30": { 2024-04-25T16:52:53.886 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:53.886 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T16:52:53.886 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:53.886 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 433, 2024-04-25T16:52:53.886 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 461010, 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 433, 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 123, 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 461010, 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:53.887 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:53.888 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:53.888 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.888 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:53.888 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T16:52:54.576 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-25T16:52:54.576 INFO:tasks.check_counter:Daemon mds.i mds.imported=9 2024-04-25T16:52:54.576 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-25T16:52:54.576 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T16:52:54.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid c520eaa8-031f-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T16:52:54.814 INFO:teuthology.orchestra.run.smithi203.stderr:Inferring config /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:52:55.686 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:55 smithi203 ceph-mon[33115]: pgmap v989: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T16:52:55.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:55 smithi098 ceph-mon[29162]: pgmap v989: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T16:52:55.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:55 smithi173 ceph-mon[32556]: pgmap v989: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T16:52:56.579 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2024-04-25T16:52:56.580 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:52:56.580 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 576, 2024-04-25T16:52:56.580 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 578, 2024-04-25T16:52:56.580 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 8513855, 2024-04-25T16:52:56.580 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 183232, 2024-04-25T16:52:56.580 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 7, 2024-04-25T16:52:56.580 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-04-25T16:52:56.581 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.052913131, 2024-04-25T16:52:56.581 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.013219766, 2024-04-25T16:52:56.581 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.020553115, 2024-04-25T16:52:56.581 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.014397592, 2024-04-25T16:52:56.581 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:56.581 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 576, 2024-04-25T16:52:56.581 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.018188661, 2024-04-25T16:52:56.581 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000031577 2024-04-25T16:52:56.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.582 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:56.582 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.582 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.582 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.582 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.582 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:52:56.582 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 533, 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 640, 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 133030, 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 446704, 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 5, 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 4, 2024-04-25T16:52:56.583 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.048734635, 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.014956108, 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.012127528, 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.017528601, 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 639, 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.020495127, 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000032073 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.584 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 3120, 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 278800 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:52:56.585 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_messages": 700, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages": 679, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_bytes": 510390, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_bytes": 257621, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_created_connections": 4, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_active_connections": 3, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_total_time": 0.070145664, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_send_time": 0.022086025, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_recv_time": 0.024937568, 2024-04-25T16:52:56.586 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_running_fast_dispatch_time": 0.019484242, 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 678, 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.027762836, 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000040948 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.587 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_recv_encrypted_bytes": 333072, 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: "msgr_send_encrypted_bytes": 89712 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: "cct": { 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: "total_workers": 1, 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: "unhealthy_workers": 0 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-MDSRank": { 2024-04-25T16:52:56.588 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 2673, 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.083745174, 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000031330 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: "finisher-PurgeQueue": { 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: "queue_len": 0, 2024-04-25T16:52:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: "complete_latency": { 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: "mds": { 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: "request": 0, 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: "reply": 0, 2024-04-25T16:52:56.590 INFO:teuthology.orchestra.run.smithi203.stdout: "reply_latency": { 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_reply": 0, 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: "forward": 0, 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_complete": 0, 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_fetch_keys": 0, 2024-04-25T16:52:56.591 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_commit": 0, 2024-04-25T16:52:56.592 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_split": 0, 2024-04-25T16:52:56.592 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_merge": 0, 2024-04-25T16:52:56.592 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes": 3450, 2024-04-25T16:52:56.592 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_top": 35, 2024-04-25T16:52:56.592 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_bottom": 3415, 2024-04-25T16:52:56.592 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pin_tail": 0, 2024-04-25T16:52:56.592 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_pinned": 3401, 2024-04-25T16:52:56.592 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_expired": 0, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "inodes_with_caps": 0, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "caps": 0, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "subtrees": 160, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse": 0, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_hit": 0, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_forward": 0, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_discover": 0, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:52:56.593 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_remote_ino": 0, 2024-04-25T16:52:56.594 INFO:teuthology.orchestra.run.smithi203.stdout: "traverse_lock": 0, 2024-04-25T16:52:56.594 INFO:teuthology.orchestra.run.smithi203.stdout: "load_cent": 0, 2024-04-25T16:52:56.594 INFO:teuthology.orchestra.run.smithi203.stdout: "q": 0, 2024-04-25T16:52:56.594 INFO:teuthology.orchestra.run.smithi203.stdout: "exported": 0, 2024-04-25T16:52:56.594 INFO:teuthology.orchestra.run.smithi203.stdout: "exported_inodes": 0, 2024-04-25T16:52:56.594 INFO:teuthology.orchestra.run.smithi203.stdout: "imported": 0, 2024-04-25T16:52:56.594 INFO:teuthology.orchestra.run.smithi203.stdout: "imported_inodes": 0, 2024-04-25T16:52:56.594 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_dir_fetch": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "openino_peer_discover": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rfiles": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rbytes": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "root_rsnaps": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_set_tag": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:52:56.595 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "scrub_file_inodes": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:52:56.596 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps": 0, 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_cap_release": 0, 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "process_request_cap_release": 0 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_cache": { 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update": 0, 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_update_receipt": 0, 2024-04-25T16:52:56.597 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_try_discover": 0, 2024-04-25T16:52:56.598 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_send_discover": 0, 2024-04-25T16:52:56.598 INFO:teuthology.orchestra.run.smithi203.stdout: "dir_handle_discover": 0, 2024-04-25T16:52:56.598 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays": 0, 2024-04-25T16:52:56.598 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_delayed": 0, 2024-04-25T16:52:56.598 INFO:teuthology.orchestra.run.smithi203.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:52:56.598 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_created": 0, 2024-04-25T16:52:56.598 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_enqueued": 0, 2024-04-25T16:52:56.598 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_reintegrated": 0, 2024-04-25T16:52:56.599 INFO:teuthology.orchestra.run.smithi203.stdout: "strays_migrated": 0, 2024-04-25T16:52:56.599 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_processing": 0, 2024-04-25T16:52:56.599 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:52:56.599 INFO:teuthology.orchestra.run.smithi203.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:52:56.599 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_started": 0, 2024-04-25T16:52:56.599 INFO:teuthology.orchestra.run.smithi203.stdout: "recovery_completed": 0, 2024-04-25T16:52:56.599 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:52:56.599 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:52:56.600 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:52:56.600 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_exportdir": 0, 2024-04-25T16:52:56.600 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_flush": 0, 2024-04-25T16:52:56.600 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:52:56.600 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_fragstats": 0, 2024-04-25T16:52:56.600 INFO:teuthology.orchestra.run.smithi203.stdout: "ireq_inodestats": 0 2024-04-25T16:52:56.600 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.600 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_log": { 2024-04-25T16:52:56.601 INFO:teuthology.orchestra.run.smithi203.stdout: "evlrg": 0, 2024-04-25T16:52:56.601 INFO:teuthology.orchestra.run.smithi203.stdout: "evadd": 0, 2024-04-25T16:52:56.601 INFO:teuthology.orchestra.run.smithi203.stdout: "evex": 0, 2024-04-25T16:52:56.601 INFO:teuthology.orchestra.run.smithi203.stdout: "evtrm": 0, 2024-04-25T16:52:56.601 INFO:teuthology.orchestra.run.smithi203.stdout: "ev": 4065, 2024-04-25T16:52:56.601 INFO:teuthology.orchestra.run.smithi203.stdout: "evexg": 0, 2024-04-25T16:52:56.601 INFO:teuthology.orchestra.run.smithi203.stdout: "evexd": 0, 2024-04-25T16:52:56.601 INFO:teuthology.orchestra.run.smithi203.stdout: "segadd": 0, 2024-04-25T16:52:56.602 INFO:teuthology.orchestra.run.smithi203.stdout: "segex": 0, 2024-04-25T16:52:56.602 INFO:teuthology.orchestra.run.smithi203.stdout: "segtrm": 0, 2024-04-25T16:52:56.602 INFO:teuthology.orchestra.run.smithi203.stdout: "seg": 6, 2024-04-25T16:52:56.602 INFO:teuthology.orchestra.run.smithi203.stdout: "segmjr": 2, 2024-04-25T16:52:56.602 INFO:teuthology.orchestra.run.smithi203.stdout: "segexg": 0, 2024-04-25T16:52:56.602 INFO:teuthology.orchestra.run.smithi203.stdout: "segexd": 0, 2024-04-25T16:52:56.602 INFO:teuthology.orchestra.run.smithi203.stdout: "expos": 4194304, 2024-04-25T16:52:56.602 INFO:teuthology.orchestra.run.smithi203.stdout: "wrpos": 12557344, 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: "rdpos": 12556770, 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: "jlat": { 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: "replayed": 4065 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.603 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_mem": { 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "ino": 3428, 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "ino+": 3460, 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "ino-": 32, 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "dir": 440, 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "dir+": 490, 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "dir-": 50, 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "dn": 3450, 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "dn+": 3469, 2024-04-25T16:52:56.604 INFO:teuthology.orchestra.run.smithi203.stdout: "dn-": 19, 2024-04-25T16:52:56.605 INFO:teuthology.orchestra.run.smithi203.stdout: "cap": 0, 2024-04-25T16:52:56.605 INFO:teuthology.orchestra.run.smithi203.stdout: "cap+": 0, 2024-04-25T16:52:56.605 INFO:teuthology.orchestra.run.smithi203.stdout: "cap-": 0, 2024-04-25T16:52:56.605 INFO:teuthology.orchestra.run.smithi203.stdout: "rss": 130148, 2024-04-25T16:52:56.605 INFO:teuthology.orchestra.run.smithi203.stdout: "heap": 215440 2024-04-25T16:52:56.605 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.605 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_server": { 2024-04-25T16:52:56.605 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_client_request": 0, 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: "dispatch_server_request": 0, 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_request": 0, 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_client_session": 0, 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: "handle_peer_request": 0, 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: "req_create_latency": { 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.606 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.607 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getattr_latency": { 2024-04-25T16:52:56.607 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.607 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.607 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.607 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.607 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getfilelock_latency": { 2024-04-25T16:52:56.607 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.607 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: "req_link_latency": { 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookup_latency": { 2024-04-25T16:52:56.608 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.609 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.609 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.609 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookuphash_latency": { 2024-04-25T16:52:56.609 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.609 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.609 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.609 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.610 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupino_latency": { 2024-04-25T16:52:56.610 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.610 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.610 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.610 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.610 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupname_latency": { 2024-04-25T16:52:56.610 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.610 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.611 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.611 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupparent_latency": { 2024-04-25T16:52:56.611 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.611 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.611 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.611 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.611 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lookupsnap_latency": { 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: "req_lssnap_latency": { 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.612 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.613 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mkdir_latency": { 2024-04-25T16:52:56.613 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.614 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.614 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.614 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.614 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mknod_latency": { 2024-04-25T16:52:56.614 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.614 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.614 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.614 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: "req_mksnap_latency": { 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: "req_open_latency": { 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.615 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: "req_readdir_latency": { 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rename_latency": { 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.616 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.617 INFO:teuthology.orchestra.run.smithi203.stdout: "req_renamesnap_latency": { 2024-04-25T16:52:56.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.617 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.617 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.617 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.617 INFO:teuthology.orchestra.run.smithi203.stdout: "req_snapdiff_latency": { 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmdir_latency": { 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.618 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmsnap_latency": { 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: "req_rmxattr_latency": { 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.619 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.620 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setattr_latency": { 2024-04-25T16:52:56.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.620 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.620 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.620 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setdirlayout_latency": { 2024-04-25T16:52:56.620 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setfilelock_latency": { 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.621 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setlayout_latency": { 2024-04-25T16:52:56.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.622 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.622 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.622 INFO:teuthology.orchestra.run.smithi203.stdout: "req_setxattr_latency": { 2024-04-25T16:52:56.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.622 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.622 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: "req_symlink_latency": { 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: "req_unlink_latency": { 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.623 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: "req_getvxattr_latency": { 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.624 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_sessions": { 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: "session_count": 1, 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: "session_add": 2, 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: "session_remove": 1, 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_open": 1, 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: "sessions_stale": 0, 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: "total_load": 0, 2024-04-25T16:52:56.625 INFO:teuthology.orchestra.run.smithi203.stdout: "average_load": 0, 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: "avg_session_uptime": 2327, 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: "mdthresh_evicted": 0 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: "mempool": { 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: "bloom_filter_items": 0, 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:52:56.626 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:52:56.627 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:52:56.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:52:56.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_extent_items": 0, 2024-04-25T16:52:56.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:52:56.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_blob_items": 0, 2024-04-25T16:52:56.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:52:56.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:52:56.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:52:56.628 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:52:56.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:52:56.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:52:56.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:52:56.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_txc_items": 0, 2024-04-25T16:52:56.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:52:56.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:52:56.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:52:56.629 INFO:teuthology.orchestra.run.smithi203.stdout: "bluestore_writing_items": 0, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_bytes": 0, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_items": 0, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_bytes": 205560, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_anon_items": 59, 2024-04-25T16:52:56.630 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:52:56.631 INFO:teuthology.orchestra.run.smithi203.stdout: "buffer_meta_items": 0, 2024-04-25T16:52:56.631 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_bytes": 0, 2024-04-25T16:52:56.631 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_items": 0, 2024-04-25T16:52:56.631 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:52:56.631 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_mapbl_items": 0, 2024-04-25T16:52:56.631 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:52:56.631 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_pglog_items": 0, 2024-04-25T16:52:56.631 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_bytes": 12168, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_items": 276, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_bytes": 0, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "pgmap_items": 0, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_bytes": 10294120, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_co_items": 54376, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_bytes": 0, 2024-04-25T16:52:56.632 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_1_items": 0, 2024-04-25T16:52:56.633 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_bytes": 0, 2024-04-25T16:52:56.633 INFO:teuthology.orchestra.run.smithi203.stdout: "unittest_2_items": 0 2024-04-25T16:52:56.633 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.633 INFO:teuthology.orchestra.run.smithi203.stdout: "objecter": { 2024-04-25T16:52:56.633 INFO:teuthology.orchestra.run.smithi203.stdout: "op_active": 0, 2024-04-25T16:52:56.633 INFO:teuthology.orchestra.run.smithi203.stdout: "op_laggy": 0, 2024-04-25T16:52:56.633 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send": 1632, 2024-04-25T16:52:56.633 INFO:teuthology.orchestra.run.smithi203.stdout: "op_send_bytes": 21, 2024-04-25T16:52:56.634 INFO:teuthology.orchestra.run.smithi203.stdout: "op_resend": 0, 2024-04-25T16:52:56.634 INFO:teuthology.orchestra.run.smithi203.stdout: "op_reply": 1632, 2024-04-25T16:52:56.634 INFO:teuthology.orchestra.run.smithi203.stdout: "op_latency": { 2024-04-25T16:52:56.634 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1632, 2024-04-25T16:52:56.634 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 2.169948868, 2024-04-25T16:52:56.634 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.001329625 2024-04-25T16:52:56.634 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.634 INFO:teuthology.orchestra.run.smithi203.stdout: "op_inflight": 0, 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: "oplen_avg": { 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 1632, 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 1633 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: "op": 1632, 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: "op_r": 1632, 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: "op_w": 0, 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: "op_rmw": 0, 2024-04-25T16:52:56.635 INFO:teuthology.orchestra.run.smithi203.stdout: "op_pg": 0, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_stat": 1054, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_create": 0, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_read": 576, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_write": 0, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writefull": 0, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_writesame": 0, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_append": 0, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_zero": 0, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_truncate": 0, 2024-04-25T16:52:56.636 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_delete": 0, 2024-04-25T16:52:56.637 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_mapext": 0, 2024-04-25T16:52:56.637 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_sparse_read": 0, 2024-04-25T16:52:56.637 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_clonerange": 0, 2024-04-25T16:52:56.637 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_getxattr": 1, 2024-04-25T16:52:56.637 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_setxattr": 0, 2024-04-25T16:52:56.637 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:52:56.637 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_rmxattr": 0, 2024-04-25T16:52:56.637 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:52:56.638 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_call": 0, 2024-04-25T16:52:56.638 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_watch": 0, 2024-04-25T16:52:56.638 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_notify": 0, 2024-04-25T16:52:56.638 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:52:56.638 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls": 0, 2024-04-25T16:52:56.638 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:52:56.638 INFO:teuthology.orchestra.run.smithi203.stdout: "osdop_other": 0, 2024-04-25T16:52:56.638 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_active": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_send": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_resend": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "linger_ping": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_active": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_send": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "poolop_resend": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_active": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_send": 0, 2024-04-25T16:52:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: "poolstat_resend": 0, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_active": 0, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_send": 0, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "statfs_resend": 0, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "command_active": 0, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "command_send": 0, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "command_resend": 0, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "map_epoch": 106, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "map_full": 0, 2024-04-25T16:52:56.640 INFO:teuthology.orchestra.run.smithi203.stdout: "map_inc": 3, 2024-04-25T16:52:56.641 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_sessions": 7, 2024-04-25T16:52:56.641 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_open": 7, 2024-04-25T16:52:56.641 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_session_close": 0, 2024-04-25T16:52:56.641 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_laggy": 0, 2024-04-25T16:52:56.641 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_wr": 0, 2024-04-25T16:52:56.641 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_rd": 2, 2024-04-25T16:52:56.641 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_del": 0 2024-04-25T16:52:56.641 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: "oft": { 2024-04-25T16:52:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_objs": 0, 2024-04-25T16:52:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:52:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_updates": 0, 2024-04-25T16:52:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: "omap_total_removes": 0 2024-04-25T16:52:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: "osd-slow-ops": { 2024-04-25T16:52:56.642 INFO:teuthology.orchestra.run.smithi203.stdout: "slow_ops_count": 0 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: "purge_queue": { 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops": 0, 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing": 0, 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executing_high_water": 0, 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed_ops": 0, 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_executed": 0, 2024-04-25T16:52:56.643 INFO:teuthology.orchestra.run.smithi203.stdout: "pq_item_in_journal": 0 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 1805, 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 8983124, 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:56.644 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 1805, 2024-04-25T16:52:56.645 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:56.645 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:56.645 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1805, 2024-04-25T16:52:56.645 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 8983124, 2024-04-25T16:52:56.645 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:56.645 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.645 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.645 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_bytes": { 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 104857600, 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:56.646 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:56.647 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:56.647 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1632, 2024-04-25T16:52:56.647 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 8363045, 2024-04-25T16:52:56.647 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1632, 2024-04-25T16:52:56.647 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 8363045, 2024-04-25T16:52:56.647 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:56.647 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.647 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-objecter_ops": { 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 1024, 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:56.648 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:56.649 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:56.649 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:56.649 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 1632, 2024-04-25T16:52:56.649 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 1632, 2024-04-25T16:52:56.649 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 1632, 2024-04-25T16:52:56.649 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 1632, 2024-04-25T16:52:56.649 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:56.649 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:52:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T16:52:56.650 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T16:52:56.651 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:56.652 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.652 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.652 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.652 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.652 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2024-04-25T16:52:56.652 INFO:teuthology.orchestra.run.smithi203.stdout: "throttle-write_buf_throttle-0x55ba6a626c30": { 2024-04-25T16:52:56.652 INFO:teuthology.orchestra.run.smithi203.stdout: "val": 0, 2024-04-25T16:52:56.652 INFO:teuthology.orchestra.run.smithi203.stdout: "max": 3758096384, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "get_started": 0, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "get": 0, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "get_sum": 0, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_fail": 0, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "get_or_fail_success": 0, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "take": 0, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "take_sum": 0, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "put": 0, 2024-04-25T16:52:56.653 INFO:teuthology.orchestra.run.smithi203.stdout: "put_sum": 0, 2024-04-25T16:52:56.654 INFO:teuthology.orchestra.run.smithi203.stdout: "wait": { 2024-04-25T16:52:56.654 INFO:teuthology.orchestra.run.smithi203.stdout: "avgcount": 0, 2024-04-25T16:52:56.654 INFO:teuthology.orchestra.run.smithi203.stdout: "sum": 0.000000000, 2024-04-25T16:52:56.654 INFO:teuthology.orchestra.run.smithi203.stdout: "avgtime": 0.000000000 2024-04-25T16:52:56.654 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.654 INFO:teuthology.orchestra.run.smithi203.stdout: } 2024-04-25T16:52:56.654 INFO:teuthology.orchestra.run.smithi203.stdout:} 2024-04-25T16:52:57.323 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-25T16:52:57.324 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-25T16:52:57.324 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-04-25T16:52:57.324 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-25T16:52:57.334 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-25T16:52:57.335 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:52:57.335 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:52:57.369 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:52:57.369 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:52:57.428 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:52:57.586 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.588+0000 7fc1410aa640 1 -- 172.21.15.98:0/722714019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c1050a0 msgr2=0x7fc13c1054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:57.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.588+0000 7fc1410aa640 1 --2- 172.21.15.98:0/722714019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c1050a0 0x7fc13c1054b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fc12c009900 tx=0x7fc12c02a410 comp rx=0 tx=0).stop 2024-04-25T16:52:57.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.589+0000 7fc1410aa640 1 -- 172.21.15.98:0/722714019 shutdown_connections 2024-04-25T16:52:57.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.589+0000 7fc1410aa640 1 --2- 172.21.15.98:0/722714019 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc13c102250 0x7fc13c10bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.589+0000 7fc1410aa640 1 --2- 172.21.15.98:0/722714019 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc13c101750 0x7fc13c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.589+0000 7fc1410aa640 1 --2- 172.21.15.98:0/722714019 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c1050a0 0x7fc13c1054b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.587 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.589+0000 7fc1410aa640 1 -- 172.21.15.98:0/722714019 >> 172.21.15.98:0/722714019 conn(0x7fc13c0fd370 msgr2=0x7fc13c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.589+0000 7fc1410aa640 1 -- 172.21.15.98:0/722714019 shutdown_connections 2024-04-25T16:52:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.589+0000 7fc1410aa640 1 -- 172.21.15.98:0/722714019 wait complete. 2024-04-25T16:52:57.588 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.589+0000 7fc1410aa640 1 Processor -- start 2024-04-25T16:52:57.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc1410aa640 1 -- start start 2024-04-25T16:52:57.590 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc1410aa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c101750 0x7fc13c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc1410aa640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc13c102250 0x7fc13c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc1410aa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc13c1050a0 0x7fc13c1a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc1410aa640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c10f4c0 con 0x7fc13c101750 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc1410aa640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c10f340 con 0x7fc13c102250 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc1410aa640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc13c10f640 con 0x7fc13c1050a0 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c101750 0x7fc13c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c101750 0x7fc13c19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48038/0 (socket says 172.21.15.98:48038) 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 -- 172.21.15.98:0/818007588 learned_addr learned my addr 172.21.15.98:0/818007588 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:57.591 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13a59c640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc13c102250 0x7fc13c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13b59e640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc13c1050a0 0x7fc13c1a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 -- 172.21.15.98:0/818007588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc13c1050a0 msgr2=0x7fc13c1a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc13c1050a0 0x7fc13c1a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 -- 172.21.15.98:0/818007588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc13c102250 msgr2=0x7fc13c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc13c102250 0x7fc13c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 -- 172.21.15.98:0/818007588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc13c1a1ca0 con 0x7fc13c101750 2024-04-25T16:52:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.590+0000 7fc13ad9d640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c101750 0x7fc13c19a160 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc12c005760 tx=0x7fc12c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:57.592 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.591+0000 7fc123fff640 1 -- 172.21.15.98:0/818007588 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c03f030 con 0x7fc13c101750 2024-04-25T16:52:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.591+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc13c1a1f30 con 0x7fc13c101750 2024-04-25T16:52:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.591+0000 7fc123fff640 1 -- 172.21.15.98:0/818007588 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc12c003a80 con 0x7fc13c101750 2024-04-25T16:52:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.591+0000 7fc123fff640 1 -- 172.21.15.98:0/818007588 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc12c035070 con 0x7fc13c101750 2024-04-25T16:52:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.591+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc13c1a2390 con 0x7fc13c101750 2024-04-25T16:52:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.592+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc13c108a30 con 0x7fc13c101750 2024-04-25T16:52:57.593 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.593+0000 7fc123fff640 1 -- 172.21.15.98:0/818007588 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fc12c042030 con 0x7fc13c101750 2024-04-25T16:52:57.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.594+0000 7fc123fff640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1240afaf0 0x7fc1240b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:57.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.594+0000 7fc13a59c640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1240afaf0 0x7fc1240b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:57.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.594+0000 7fc123fff640 1 -- 172.21.15.98:0/818007588 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7fc12c100230 con 0x7fc13c101750 2024-04-25T16:52:57.594 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.594+0000 7fc13a59c640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1240afaf0 0x7fc1240b2000 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc1300058c0 tx=0x7fc13000a250 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:57.595 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.597+0000 7fc123fff640 1 -- 172.21.15.98:0/818007588 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc12c0c8350 con 0x7fc13c101750 2024-04-25T16:52:57.685 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:57 smithi203 ceph-mon[33115]: pgmap v990: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T16:52:57.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:57 smithi098 ceph-mon[29162]: pgmap v990: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T16:52:57.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:57 smithi173 ceph-mon[32556]: pgmap v990: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-25T16:52:57.773 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.776+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc13c02dda0 con 0x7fc13c101750 2024-04-25T16:52:57.775 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.777+0000 7fc123fff640 1 -- 172.21.15.98:0/818007588 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+27569 (secure 0 0 0) 0x7fc12c0d03d0 con 0x7fc13c101750 2024-04-25T16:52:57.775 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:52:57.776 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":106,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:51:36.553341+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6839/2689051587":"2024-04-26T16:44:26.579258+0000","172.21.15.98:6838/2689051587":"2024-04-26T16:44:26.579258+0000","172.21.15.173:6833/2439698545":"2024-04-26T16:44:03.604569+0000","172.21.15.173:6832/2439698545":"2024-04-26T16:44:03.604569+0000","172.21.15.203:6832/2590142748":"2024-04-26T16:44:03.513876+0000","172.21.15.203:6837/3458796538":"2024-04-26T16:43:39.806154+0000","172.21.15.203:6839/3267653510":"2024-04-26T16:43:37.634911+0000","172.21.15.203:6835/4038856286":"2024-04-26T16:43:37.105127+0000","172.21.15.173:6838/3538054402":"2024-04-26T16:35:43.083983+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.173:6838/2873375163":"2024-04-26T16:40:54.470086+0000","172.21.15.173:6836/3031685940":"2024-04-26T16:44:26.346994+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6832/1945729206":"2024-04-26T16:43:15.031214+0000","172.21.15.203:6834/4038856286":"2024-04-26T16:43:37.105127+0000","172.21.15.173:6838/2050448663":"2024-04-26T16:35:09.397460+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.173:6839/3538054402":"2024-04-26T16:35:43.083983+0000","172.21.15.203:6834/3629133920":"2024-04-26T16:51:36.553281+0000","172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6836/3458796538":"2024-04-26T16:43:39.806154+0000","172.21.15.203:6838/3267653510":"2024-04-26T16:43:37.634911+0000","172.21.15.203:6834/1151278804":"2024-04-26T16:40:28.525704+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6836/998442662":"2024-04-26T16:37:09.422158+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.173:6837/3031685940":"2024-04-26T16:44:26.346994+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6835/3629133920":"2024-04-26T16:51:36.553281+0000","172.21.15.173:6836/311453164":"2024-04-26T16:40:25.599313+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6839/2982027693":"2024-04-26T16:39:30.428647+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6834/890593273":"2024-04-26T16:35:58.723078+0000","172.21.15.173:6839/2050448663":"2024-04-26T16:35:09.397460+0000","172.21.15.203:6835/890593273":"2024-04-26T16:35:58.723078+0000","172.21.15.173:6838/3664329633":"2024-04-26T16:36:17.370724+0000","172.21.15.173:6839/3664329633":"2024-04-26T16:36:17.370724+0000","172.21.15.203:6833/2590142748":"2024-04-26T16:44:03.513876+0000","172.21.15.203:6837/998442662":"2024-04-26T16:37:09.422158+0000","172.21.15.203:6838/2982027693":"2024-04-26T16:39:30.428647+0000","172.21.15.203:6835/2230353021":"2024-04-26T16:35:26.239567+0000","172.21.15.173:6837/311453164":"2024-04-26T16:40:25.599313+0000","172.21.15.203:6833/1945729206":"2024-04-26T16:43:15.031214+0000","172.21.15.203:6834/2230353021":"2024-04-26T16:35:26.239567+0000","172.21.15.203:6835/1151278804":"2024-04-26T16:40:28.525704+0000","172.21.15.173:6832/3037734321":"2024-04-26T16:43:15.040785+0000","172.21.15.173:6839/2873375163":"2024-04-26T16:40:54.470086+0000","172.21.15.98:6838/3308601909":"2024-04-26T16:41:27.323250+0000","172.21.15.98:6839/3308601909":"2024-04-26T16:41:27.323250+0000","172.21.15.173:6833/3037734321":"2024-04-26T16:43:15.040785+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:52:57.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1240afaf0 msgr2=0x7fc1240b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:57.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1240afaf0 0x7fc1240b2000 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc1300058c0 tx=0x7fc13000a250 comp rx=0 tx=0).stop 2024-04-25T16:52:57.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c101750 msgr2=0x7fc13c19a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:57.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c101750 0x7fc13c19a160 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc12c005760 tx=0x7fc12c004270 comp rx=0 tx=0).stop 2024-04-25T16:52:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 shutdown_connections 2024-04-25T16:52:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc1240afaf0 0x7fc1240b2000 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc13c1050a0 0x7fc13c1a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc13c102250 0x7fc13c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.779+0000 7fc1410aa640 1 --2- 172.21.15.98:0/818007588 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc13c101750 0x7fc13c19a160 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.780+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 >> 172.21.15.98:0/818007588 conn(0x7fc13c0fd370 msgr2=0x7fc13c105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.780+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 shutdown_connections 2024-04-25T16:52:57.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.780+0000 7fc1410aa640 1 -- 172.21.15.98:0/818007588 wait complete. 2024-04-25T16:52:57.789 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:52:57.789 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:52:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 -- 172.21.15.98:0/4121073553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 msgr2=0x7f2cc010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 --2- 172.21.15.98:0/4121073553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 0x7f2cc010ad40 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009900 tx=0x7f2cb802a410 comp rx=0 tx=0).stop 2024-04-25T16:52:57.902 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 -- 172.21.15.98:0/4121073553 shutdown_connections 2024-04-25T16:52:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 --2- 172.21.15.98:0/4121073553 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 0x7f2cc010ad40 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 --2- 172.21.15.98:0/4121073553 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cc0103c20 0x7f2cc0104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 --2- 172.21.15.98:0/4121073553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cc01029e0 0x7f2cc01036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 -- 172.21.15.98:0/4121073553 >> 172.21.15.98:0/4121073553 conn(0x7f2cc00fdbd0 msgr2=0x7f2cc0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 -- 172.21.15.98:0/4121073553 shutdown_connections 2024-04-25T16:52:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 -- 172.21.15.98:0/4121073553 wait complete. 2024-04-25T16:52:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 Processor -- start 2024-04-25T16:52:57.903 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.903+0000 7f2cc594d640 1 -- start start 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc594d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cc01029e0 0x7f2cc01992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc594d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cc0103c20 0x7f2cc0199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc594d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 0x7f2cc01a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc594d640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc010e3a0 con 0x7f2cc0108920 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc594d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc010e220 con 0x7f2cc01029e0 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc594d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc010e520 con 0x7f2cc0103c20 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 0x7f2cc01a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 0x7f2cc01a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48058/0 (socket says 172.21.15.98:48058) 2024-04-25T16:52:57.904 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 -- 172.21.15.98:0/1533969456 learned_addr learned my addr 172.21.15.98:0/1533969456 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbeffd640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cc01029e0 0x7f2cc01992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbe7fc640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cc0103c20 0x7f2cc0199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 -- 172.21.15.98:0/1533969456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cc0103c20 msgr2=0x7f2cc0199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cc0103c20 0x7f2cc0199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 -- 172.21.15.98:0/1533969456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cc01029e0 msgr2=0x7f2cc01992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cc01029e0 0x7f2cc01992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 -- 172.21.15.98:0/1533969456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc01a0d80 con 0x7f2cc0108920 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cbf7fe640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 0x7f2cc01a07c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8005ab0 tx=0x7f2cb80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:57.905 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc494b640 1 -- 172.21.15.98:0/1533969456 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb803f030 con 0x7f2cc0108920 2024-04-25T16:52:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc01a1010 con 0x7f2cc0108920 2024-04-25T16:52:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc494b640 1 -- 172.21.15.98:0/1533969456 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2cb8003c90 con 0x7f2cc0108920 2024-04-25T16:52:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc494b640 1 -- 172.21.15.98:0/1533969456 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8035070 con 0x7f2cc0108920 2024-04-25T16:52:57.906 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.904+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc01a1550 con 0x7f2cc0108920 2024-04-25T16:52:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.905+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc0107730 con 0x7f2cc0108920 2024-04-25T16:52:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.907+0000 7f2cc494b640 1 -- 172.21.15.98:0/1533969456 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f2cb8042030 con 0x7f2cc0108920 2024-04-25T16:52:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.908+0000 7f2cc494b640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ca80afaf0 0x7f2ca80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.908+0000 7f2cbeffd640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ca80afaf0 0x7f2ca80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.908+0000 7f2cc494b640 1 -- 172.21.15.98:0/1533969456 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f2cb8100160 con 0x7f2cc0108920 2024-04-25T16:52:57.907 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.908+0000 7f2cbeffd640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ca80afaf0 0x7f2ca80b2000 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2cc01035e0 tx=0x7f2cb0007600 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:57.909 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:57.911+0000 7f2cc494b640 1 -- 172.21.15.98:0/1533969456 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2cb80c8280 con 0x7f2cc0108920 2024-04-25T16:52:58.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.094+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2cc002dd90 con 0x7f2cc0108920 2024-04-25T16:52:58.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.095+0000 7f2cc494b640 1 -- 172.21.15.98:0/1533969456 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f2cb80c7b80 con 0x7f2cc0108920 2024-04-25T16:52:58.092 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:52:58.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.096+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ca80afaf0 msgr2=0x7f2ca80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ca80afaf0 0x7f2ca80b2000 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2cc01035e0 tx=0x7f2cb0007600 comp rx=0 tx=0).stop 2024-04-25T16:52:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 msgr2=0x7f2cc01a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 0x7f2cc01a07c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8005ab0 tx=0x7f2cb80042f0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 shutdown_connections 2024-04-25T16:52:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ca80afaf0 0x7f2ca80b2000 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2cc0108920 0x7f2cc01a07c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2cc0103c20 0x7f2cc0199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 --2- 172.21.15.98:0/1533969456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cc01029e0 0x7f2cc01992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 >> 172.21.15.98:0/1533969456 conn(0x7f2cc00fdbd0 msgr2=0x7f2cc0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 shutdown_connections 2024-04-25T16:52:58.096 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.097+0000 7f2cc594d640 1 -- 172.21.15.98:0/1533969456 wait complete. 2024-04-25T16:52:58.106 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:52:58.217 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 -- 172.21.15.98:0/1128486783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d0103f00 msgr2=0x7fb5d01027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1128486783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d0103f00 0x7fb5d01027e0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009900 tx=0x7fb5c402a410 comp rx=0 tx=0).stop 2024-04-25T16:52:58.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 -- 172.21.15.98:0/1128486783 shutdown_connections 2024-04-25T16:52:58.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1128486783 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0108a30 0x7fb5d010ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1128486783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5d01043e0 0x7fb5d0102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1128486783 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d0103f00 0x7fb5d01027e0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 -- 172.21.15.98:0/1128486783 >> 172.21.15.98:0/1128486783 conn(0x7fb5d00fd380 msgr2=0x7fb5d00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:58.218 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 -- 172.21.15.98:0/1128486783 shutdown_connections 2024-04-25T16:52:58.219 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 -- 172.21.15.98:0/1128486783 wait complete. 2024-04-25T16:52:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.220+0000 7fb5d516c640 1 Processor -- start 2024-04-25T16:52:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5d516c640 1 -- start start 2024-04-25T16:52:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5d516c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0103f00 0x7fb5d0198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5d516c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5d0108a30 0x7fb5d01991a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5d516c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d019e0e0 0x7fb5d01a0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5d516c640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d010e3a0 con 0x7fb5d019e0e0 2024-04-25T16:52:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5d516c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d010e220 con 0x7fb5d0108a30 2024-04-25T16:52:58.223 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5d516c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fb5d010e520 con 0x7fb5d0103f00 2024-04-25T16:52:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5cf59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d019e0e0 0x7fb5d01a0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5cf59e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d019e0e0 0x7fb5d01a0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48090/0 (socket says 172.21.15.98:48090) 2024-04-25T16:52:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5cf59e640 1 -- 172.21.15.98:0/1811504532 learned_addr learned my addr 172.21.15.98:0/1811504532 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5ce59c640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5d0108a30 0x7fb5d01991a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5ced9d640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0103f00 0x7fb5d0198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.224 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5cf59e640 1 -- 172.21.15.98:0/1811504532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0103f00 msgr2=0x7fb5d0198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5cf59e640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0103f00 0x7fb5d0198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5cf59e640 1 -- 172.21.15.98:0/1811504532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5d0108a30 msgr2=0x7fb5d01991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5cf59e640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5d0108a30 0x7fb5d01991a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.225 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5cf59e640 1 -- 172.21.15.98:0/1811504532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5d01a0b70 con 0x7fb5d019e0e0 2024-04-25T16:52:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.221+0000 7fb5ced9d640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0103f00 0x7fb5d0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:52:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.222+0000 7fb5cf59e640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d019e0e0 0x7fb5d01a0500 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800eb10 tx=0x7fb5c800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.222+0000 7fb5b7fff640 1 -- 172.21.15.98:0/1811504532 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8004b20 con 0x7fb5d019e0e0 2024-04-25T16:52:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.222+0000 7fb5b7fff640 1 -- 172.21.15.98:0/1811504532 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb5c8004cc0 con 0x7fb5d019e0e0 2024-04-25T16:52:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.222+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5d01a0e60 con 0x7fb5d019e0e0 2024-04-25T16:52:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.222+0000 7fb5b7fff640 1 -- 172.21.15.98:0/1811504532 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8010690 con 0x7fb5d019e0e0 2024-04-25T16:52:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.222+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5d01a12d0 con 0x7fb5d019e0e0 2024-04-25T16:52:58.226 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.223+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5d0107130 con 0x7fb5d019e0e0 2024-04-25T16:52:58.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.224+0000 7fb5b7fff640 1 -- 172.21.15.98:0/1811504532 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fb5c8010830 con 0x7fb5d019e0e0 2024-04-25T16:52:58.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.225+0000 7fb5b7fff640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5b80afba0 0x7fb5b80b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.225+0000 7fb5ced9d640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5b80afba0 0x7fb5b80b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.225+0000 7fb5b7fff640 1 -- 172.21.15.98:0/1811504532 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7fb5c80dfef0 con 0x7fb5d019e0e0 2024-04-25T16:52:58.227 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.225+0000 7fb5ced9d640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5b80afba0 0x7fb5b80b20b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4000c00 tx=0x7fb5c4005720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:58.228 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.228+0000 7fb5b7fff640 1 -- 172.21.15.98:0/1811504532 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb5c8053ff0 con 0x7fb5d019e0e0 2024-04-25T16:52:58.402 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.405+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb5d002dda0 con 0x7fb5d019e0e0 2024-04-25T16:52:58.403 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.406+0000 7fb5b7fff640 1 -- 172.21.15.98:0/1811504532 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7fb5c80a8010 con 0x7fb5d019e0e0 2024-04-25T16:52:58.403 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:52:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5b80afba0 msgr2=0x7fb5b80b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.405 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5b80afba0 0x7fb5b80b20b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4000c00 tx=0x7fb5c4005720 comp rx=0 tx=0).stop 2024-04-25T16:52:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d019e0e0 msgr2=0x7fb5d01a0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d019e0e0 0x7fb5d01a0500 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800eb10 tx=0x7fb5c800c2d0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 shutdown_connections 2024-04-25T16:52:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fb5b80afba0 0x7fb5b80b20b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fb5d019e0e0 0x7fb5d01a0500 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5d0108a30 0x7fb5d01991a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.406 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 --2- 172.21.15.98:0/1811504532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb5d0103f00 0x7fb5d0198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 >> 172.21.15.98:0/1811504532 conn(0x7fb5d00fd380 msgr2=0x7fb5d00ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 shutdown_connections 2024-04-25T16:52:58.407 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.408+0000 7fb5d516c640 1 -- 172.21.15.98:0/1811504532 wait complete. 2024-04-25T16:52:58.417 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:52:58.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 -- 172.21.15.98:0/3243019158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3750108920 msgr2=0x7f375010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 --2- 172.21.15.98:0/3243019158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3750108920 0x7f375010ad40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f3748009ca0 tx=0x7f374802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 -- 172.21.15.98:0/3243019158 shutdown_connections 2024-04-25T16:52:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 --2- 172.21.15.98:0/3243019158 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f3750108920 0x7f375010ad40 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 --2- 172.21.15.98:0/3243019158 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3750103c20 0x7f3750104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 --2- 172.21.15.98:0/3243019158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37501029e0 0x7f37501036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 -- 172.21.15.98:0/3243019158 >> 172.21.15.98:0/3243019158 conn(0x7f37500fdbd0 msgr2=0x7f3750100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 -- 172.21.15.98:0/3243019158 shutdown_connections 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 -- 172.21.15.98:0/3243019158 wait complete. 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.530+0000 7f37551c7640 1 Processor -- start 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f37551c7640 1 -- start start 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f37551c7640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37501029e0 0x7f375019a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f37551c7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3750103c20 0x7f375019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f37551c7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3750108920 0x7f37501a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f37551c7640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f375010e3f0 con 0x7f37501029e0 2024-04-25T16:52:58.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f37551c7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f375010e270 con 0x7f3750108920 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f37551c7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f375010e570 con 0x7f3750103c20 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37501029e0 0x7f375019a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37501029e0 0x7f375019a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48112/0 (socket says 172.21.15.98:48112) 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 -- 172.21.15.98:0/3742707696 learned_addr learned my addr 172.21.15.98:0/3742707696 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374f59e640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3750108920 0x7f37501a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374e59c640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3750103c20 0x7f375019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 -- 172.21.15.98:0/3742707696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3750103c20 msgr2=0x7f375019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3750103c20 0x7f375019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 -- 172.21.15.98:0/3742707696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3750108920 msgr2=0x7f37501a18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3750108920 0x7f37501a18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 -- 172.21.15.98:0/3742707696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37501a1f50 con 0x7f37501029e0 2024-04-25T16:52:58.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374f59e640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3750108920 0x7f37501a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:52:58.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.531+0000 7f374ed9d640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37501029e0 0x7f375019a410 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f374400c970 tx=0x7f374400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:58.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.532+0000 7f3737fff640 1 -- 172.21.15.98:0/3742707696 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3744004220 con 0x7f37501029e0 2024-04-25T16:52:58.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.532+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37501a2240 con 0x7f37501029e0 2024-04-25T16:52:58.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.532+0000 7f3737fff640 1 -- 172.21.15.98:0/3742707696 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3744007e40 con 0x7f37501029e0 2024-04-25T16:52:58.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.532+0000 7f3737fff640 1 -- 172.21.15.98:0/3742707696 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374400f450 con 0x7f37501029e0 2024-04-25T16:52:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.532+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37501a27e0 con 0x7f37501029e0 2024-04-25T16:52:58.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.533+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f375006c2a0 con 0x7f37501029e0 2024-04-25T16:52:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.534+0000 7f3737fff640 1 -- 172.21.15.98:0/3742707696 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f37440079b0 con 0x7f37501029e0 2024-04-25T16:52:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.535+0000 7f3737fff640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f37380afb70 0x7f37380b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.535+0000 7f374e59c640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f37380afb70 0x7f37380b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.535+0000 7f3737fff640 1 -- 172.21.15.98:0/3742707696 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f37440df780 con 0x7f37501029e0 2024-04-25T16:52:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.535+0000 7f374e59c640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f37380afb70 0x7f37380b2080 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f375019b4b0 tx=0x7f374000b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:58.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.538+0000 7f3737fff640 1 -- 172.21.15.98:0/3742707696 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3744014030 con 0x7f37501029e0 2024-04-25T16:52:58.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.718+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f37501a2aa0 con 0x7f37501029e0 2024-04-25T16:52:58.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.718+0000 7f3737fff640 1 -- 172.21.15.98:0/3742707696 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f37440a7920 con 0x7f37501029e0 2024-04-25T16:52:58.716 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:52:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f37380afb70 msgr2=0x7f37380b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f37380afb70 0x7f37380b2080 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f375019b4b0 tx=0x7f374000b5b0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37501029e0 msgr2=0x7f375019a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37501029e0 0x7f375019a410 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f374400c970 tx=0x7f374400ce30 comp rx=0 tx=0).stop 2024-04-25T16:52:58.718 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 shutdown_connections 2024-04-25T16:52:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f37380afb70 0x7f37380b2080 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3750108920 0x7f37501a18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3750103c20 0x7f375019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 --2- 172.21.15.98:0/3742707696 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f37501029e0 0x7f375019a410 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 >> 172.21.15.98:0/3742707696 conn(0x7f37500fdbd0 msgr2=0x7f3750069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 shutdown_connections 2024-04-25T16:52:58.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.720+0000 7f37551c7640 1 -- 172.21.15.98:0/3742707696 wait complete. 2024-04-25T16:52:58.728 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:58 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/818007588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:52:58.728 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:58 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1533969456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:52:58.728 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:58 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1811504532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:52:58.729 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:52:58.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:58 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/818007588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:52:58.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:58 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1533969456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:52:58.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:58 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1811504532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:52:58.842 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.842+0000 7f7a02814640 1 -- 172.21.15.98:0/1586179919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 msgr2=0x7f79fc101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.842+0000 7f7a02814640 1 --2- 172.21.15.98:0/1586179919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 0x7f79fc101bd0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f79f0009900 tx=0x7f79f002a410 comp rx=0 tx=0).stop 2024-04-25T16:52:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.842+0000 7f7a02814640 1 -- 172.21.15.98:0/1586179919 shutdown_connections 2024-04-25T16:52:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.842+0000 7f7a02814640 1 --2- 172.21.15.98:0/1586179919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc102260 0x7f79fc10bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.842+0000 7f7a02814640 1 --2- 172.21.15.98:0/1586179919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 0x7f79fc101bd0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.842+0000 7f7a02814640 1 --2- 172.21.15.98:0/1586179919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc1050b0 0x7f79fc1054c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.842+0000 7f7a02814640 1 -- 172.21.15.98:0/1586179919 >> 172.21.15.98:0/1586179919 conn(0x7f79fc0fd380 msgr2=0x7f79fc0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.843+0000 7f7a02814640 1 -- 172.21.15.98:0/1586179919 shutdown_connections 2024-04-25T16:52:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.843+0000 7f7a02814640 1 -- 172.21.15.98:0/1586179919 wait complete. 2024-04-25T16:52:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.843+0000 7f7a02814640 1 Processor -- start 2024-04-25T16:52:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.843+0000 7f7a02814640 1 -- start start 2024-04-25T16:52:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.843+0000 7f7a02814640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 0x7f79fc19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.843+0000 7f7a02814640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc102260 0x7f79fc19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.843+0000 7f7a02814640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc1050b0 0x7f79fc1a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.843+0000 7f7a02814640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc10f4f0 con 0x7f79fc101760 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f7a02814640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc10f370 con 0x7f79fc102260 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f7a02814640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f79fc10f670 con 0x7f79fc1050b0 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 0x7f79fc19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 0x7f79fc19a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48130/0 (socket says 172.21.15.98:48130) 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 -- 172.21.15.98:0/3052018535 learned_addr learned my addr 172.21.15.98:0/3052018535 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f7a00db1640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc1050b0 0x7f79fc1a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.845 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fb7fe640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc102260 0x7f79fc19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 -- 172.21.15.98:0/3052018535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc1050b0 msgr2=0x7f79fc1a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc1050b0 0x7f79fc1a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 -- 172.21.15.98:0/3052018535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc102260 msgr2=0x7f79fc19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc102260 0x7f79fc19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 -- 172.21.15.98:0/3052018535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79fc1a1cd0 con 0x7f79fc101760 2024-04-25T16:52:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79fbfff640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 0x7f79fc19a190 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00c970 tx=0x7f79ec00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79f97fa640 1 -- 172.21.15.98:0/3052018535 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec004220 con 0x7f79fc101760 2024-04-25T16:52:58.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79f97fa640 1 -- 172.21.15.98:0/3052018535 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f79ec007e40 con 0x7f79fc101760 2024-04-25T16:52:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79fc1a1fc0 con 0x7f79fc101760 2024-04-25T16:52:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f79f97fa640 1 -- 172.21.15.98:0/3052018535 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00f450 con 0x7f79fc101760 2024-04-25T16:52:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.844+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79fc1a2420 con 0x7f79fc101760 2024-04-25T16:52:58.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.845+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79fc06a160 con 0x7f79fc101760 2024-04-25T16:52:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.847+0000 7f79f97fa640 1 -- 172.21.15.98:0/3052018535 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f79ec0079b0 con 0x7f79fc101760 2024-04-25T16:52:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.847+0000 7f79f97fa640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f79e40afaf0 0x7f79e40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.847+0000 7f79fb7fe640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f79e40afaf0 0x7f79e40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.847+0000 7f79f97fa640 1 -- 172.21.15.98:0/3052018535 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f79ec0df830 con 0x7f79fc101760 2024-04-25T16:52:58.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.848+0000 7f79fb7fe640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f79e40afaf0 0x7f79e40b2000 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f79f0006010 tx=0x7f79f0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:58.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:58.851+0000 7f79f97fa640 1 -- 172.21.15.98:0/3052018535 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79ec014030 con 0x7f79fc101760 2024-04-25T16:52:58.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:58 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/818007588' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:52:58.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:58 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1533969456' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:52:58.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:58 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1811504532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:52:59.027 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.030+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f79fc02dbd0 con 0x7f79fc101760 2024-04-25T16:52:59.028 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.030+0000 7f79f97fa640 1 -- 172.21.15.98:0/3052018535 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v106) ==== 103+0+11 (secure 0 0 0) 0x7f79ec0a7950 con 0x7f79fc101760 2024-04-25T16:52:59.028 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:52:59.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f79e40afaf0 msgr2=0x7f79e40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f79e40afaf0 0x7f79e40b2000 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f79f0006010 tx=0x7f79f0000bc0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 msgr2=0x7f79fc19a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 0x7f79fc19a190 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00c970 tx=0x7f79ec00ce30 comp rx=0 tx=0).stop 2024-04-25T16:52:59.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 shutdown_connections 2024-04-25T16:52:59.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f79e40afaf0 0x7f79e40b2000 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.030 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f79fc1050b0 0x7f79fc1a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79fc102260 0x7f79fc19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 --2- 172.21.15.98:0/3052018535 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f79fc101760 0x7f79fc19a190 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 >> 172.21.15.98:0/3052018535 conn(0x7f79fc0fd380 msgr2=0x7f79fc105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:59.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 shutdown_connections 2024-04-25T16:52:59.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.032+0000 7f7a02814640 1 -- 172.21.15.98:0/3052018535 wait complete. 2024-04-25T16:52:59.041 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T16:52:59.151 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 -- 172.21.15.98:0/2939070103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 msgr2=0x7fc028105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/2939070103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 0x7fc028105140 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fc018009900 tx=0x7fc01802a410 comp rx=0 tx=0).stop 2024-04-25T16:52:59.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 -- 172.21.15.98:0/2939070103 shutdown_connections 2024-04-25T16:52:59.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/2939070103 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc028109a30 0x7fc02810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/2939070103 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 0x7fc028105140 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/2939070103 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028103af0 0x7fc0281047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 -- 172.21.15.98:0/2939070103 >> 172.21.15.98:0/2939070103 conn(0x7fc0280fece0 msgr2=0x7fc028101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:59.152 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 -- 172.21.15.98:0/2939070103 shutdown_connections 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.151+0000 7fc02e7f3640 1 -- 172.21.15.98:0/2939070103 wait complete. 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.152+0000 7fc02e7f3640 1 Processor -- start 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.152+0000 7fc02e7f3640 1 -- start start 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.152+0000 7fc02e7f3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028103af0 0x7fc02819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.152+0000 7fc02e7f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 0x7fc02819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.152+0000 7fc02e7f3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc028109a30 0x7fc0281a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.152+0000 7fc02e7f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc02810f4c0 con 0x7fc028104d30 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.152+0000 7fc02e7f3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc02810f340 con 0x7fc028109a30 2024-04-25T16:52:59.153 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.152+0000 7fc02e7f3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc02810f640 con 0x7fc028103af0 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 0x7fc02819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 0x7fc02819a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48138/0 (socket says 172.21.15.98:48138) 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 -- 172.21.15.98:0/3382396859 learned_addr learned my addr 172.21.15.98:0/3382396859 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc027fff640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028103af0 0x7fc02819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc02cd90640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc028109a30 0x7fc0281a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 -- 172.21.15.98:0/3382396859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028103af0 msgr2=0x7fc02819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028103af0 0x7fc02819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 -- 172.21.15.98:0/3382396859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc028109a30 msgr2=0x7fc0281a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.154 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc028109a30 0x7fc0281a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 -- 172.21.15.98:0/3382396859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0281a1ca0 con 0x7fc028104d30 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc02cd90640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc028109a30 0x7fc0281a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0277fe640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 0x7fc02819a6a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc018009230 tx=0x7fc0180040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0257fa640 1 -- 172.21.15.98:0/3382396859 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01803f030 con 0x7fc028104d30 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0257fa640 1 -- 172.21.15.98:0/3382396859 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc0180044c0 con 0x7fc028104d30 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0281a1f30 con 0x7fc028104d30 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc0257fa640 1 -- 172.21.15.98:0/3382396859 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc018035070 con 0x7fc028104d30 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.153+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0281a2390 con 0x7fc028104d30 2024-04-25T16:52:59.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.154+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc028108840 con 0x7fc028104d30 2024-04-25T16:52:59.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.156+0000 7fc0257fa640 1 -- 172.21.15.98:0/3382396859 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fc018042030 con 0x7fc028104d30 2024-04-25T16:52:59.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.156+0000 7fc0257fa640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc0100afb70 0x7fc0100b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.157+0000 7fc027fff640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc0100afb70 0x7fc0100b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.157+0000 7fc0257fa640 1 -- 172.21.15.98:0/3382396859 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7fc0180fffb0 con 0x7fc028104d30 2024-04-25T16:52:59.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.157+0000 7fc027fff640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc0100afb70 0x7fc0100b2080 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0058c0 tx=0x7fc01c00a2d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:59.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.160+0000 7fc0257fa640 1 -- 172.21.15.98:0/3382396859 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0180c80d0 con 0x7fc028104d30 2024-04-25T16:52:59.348 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.351+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc02802db50 con 0x7fc028104d30 2024-04-25T16:52:59.349 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.352+0000 7fc0257fa640 1 -- 172.21.15.98:0/3382396859 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 52 entries v106) ==== 81+0+3183 (secure 0 0 0) 0x7fc0180d0150 con 0x7fc028104d30 2024-04-25T16:52:59.349 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6839/2689051587 2024-04-26T16:44:26.579258+0000 2024-04-25T16:52:59.350 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6838/2689051587 2024-04-26T16:44:26.579258+0000 2024-04-25T16:52:59.350 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6833/2439698545 2024-04-26T16:44:03.604569+0000 2024-04-25T16:52:59.350 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6832/2439698545 2024-04-26T16:44:03.604569+0000 2024-04-25T16:52:59.350 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6832/2590142748 2024-04-26T16:44:03.513876+0000 2024-04-25T16:52:59.350 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6837/3458796538 2024-04-26T16:43:39.806154+0000 2024-04-25T16:52:59.350 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6839/3267653510 2024-04-26T16:43:37.634911+0000 2024-04-25T16:52:59.350 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/4038856286 2024-04-26T16:43:37.105127+0000 2024-04-25T16:52:59.351 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6838/3538054402 2024-04-26T16:35:43.083983+0000 2024-04-25T16:52:59.351 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2573630616 2024-04-26T16:23:07.830236+0000 2024-04-25T16:52:59.351 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6838/2873375163 2024-04-26T16:40:54.470086+0000 2024-04-25T16:52:59.351 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6836/3031685940 2024-04-26T16:44:26.346994+0000 2024-04-25T16:52:59.351 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/4161723298 2024-04-26T16:22:41.266215+0000 2024-04-25T16:52:59.351 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6832/1945729206 2024-04-26T16:43:15.031214+0000 2024-04-25T16:52:59.351 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/4038856286 2024-04-26T16:43:37.105127+0000 2024-04-25T16:52:59.351 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6838/2050448663 2024-04-26T16:35:09.397460+0000 2024-04-25T16:52:59.352 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1429360204 2024-04-26T16:22:41.266215+0000 2024-04-25T16:52:59.352 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6839/3538054402 2024-04-26T16:35:43.083983+0000 2024-04-25T16:52:59.352 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/3629133920 2024-04-26T16:51:36.553281+0000 2024-04-25T16:52:59.352 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1883089564 2024-04-26T16:23:07.830236+0000 2024-04-25T16:52:59.352 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6836/3458796538 2024-04-26T16:43:39.806154+0000 2024-04-25T16:52:59.352 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6838/3267653510 2024-04-26T16:43:37.634911+0000 2024-04-25T16:52:59.352 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/1151278804 2024-04-26T16:40:28.525704+0000 2024-04-25T16:52:59.352 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/119631250 2024-04-26T16:23:07.830236+0000 2024-04-25T16:52:59.353 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6836/998442662 2024-04-26T16:37:09.422158+0000 2024-04-25T16:52:59.353 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/217035597 2024-04-26T16:23:07.830236+0000 2024-04-25T16:52:59.353 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3681298222 2024-04-26T16:22:41.266215+0000 2024-04-25T16:52:59.353 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6837/3031685940 2024-04-26T16:44:26.346994+0000 2024-04-25T16:52:59.353 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/4161723298 2024-04-26T16:22:41.266215+0000 2024-04-25T16:52:59.353 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/3629133920 2024-04-26T16:51:36.553281+0000 2024-04-25T16:52:59.353 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6836/311453164 2024-04-26T16:40:25.599313+0000 2024-04-25T16:52:59.353 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3090764993 2024-04-26T16:22:41.266215+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6839/2982027693 2024-04-26T16:39:30.428647+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/217035597 2024-04-26T16:23:07.830236+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/890593273 2024-04-26T16:35:58.723078+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6839/2050448663 2024-04-26T16:35:09.397460+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/890593273 2024-04-26T16:35:58.723078+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6838/3664329633 2024-04-26T16:36:17.370724+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6839/3664329633 2024-04-26T16:36:17.370724+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6833/2590142748 2024-04-26T16:44:03.513876+0000 2024-04-25T16:52:59.354 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6837/998442662 2024-04-26T16:37:09.422158+0000 2024-04-25T16:52:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6838/2982027693 2024-04-26T16:39:30.428647+0000 2024-04-25T16:52:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/2230353021 2024-04-26T16:35:26.239567+0000 2024-04-25T16:52:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6837/311453164 2024-04-26T16:40:25.599313+0000 2024-04-25T16:52:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6833/1945729206 2024-04-26T16:43:15.031214+0000 2024-04-25T16:52:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/2230353021 2024-04-26T16:35:26.239567+0000 2024-04-25T16:52:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/1151278804 2024-04-26T16:40:28.525704+0000 2024-04-25T16:52:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6832/3037734321 2024-04-26T16:43:15.040785+0000 2024-04-25T16:52:59.355 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6839/2873375163 2024-04-26T16:40:54.470086+0000 2024-04-25T16:52:59.356 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6838/3308601909 2024-04-26T16:41:27.323250+0000 2024-04-25T16:52:59.356 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6839/3308601909 2024-04-26T16:41:27.323250+0000 2024-04-25T16:52:59.356 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6833/3037734321 2024-04-26T16:43:15.040785+0000 2024-04-25T16:52:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:listed 52 entries 2024-04-25T16:52:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc0100afb70 msgr2=0x7fc0100b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc0100afb70 0x7fc0100b2080 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc01c0058c0 tx=0x7fc01c00a2d0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 msgr2=0x7fc02819a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 0x7fc02819a6a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fc018009230 tx=0x7fc0180040c0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 shutdown_connections 2024-04-25T16:52:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc0100afb70 0x7fc0100b2080 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.357 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc028109a30 0x7fc0281a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc028104d30 0x7fc02819a6a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 --2- 172.21.15.98:0/3382396859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc028103af0 0x7fc02819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 >> 172.21.15.98:0/3382396859 conn(0x7fc0280fece0 msgr2=0x7fc028105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 shutdown_connections 2024-04-25T16:52:59.358 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.354+0000 7fc02e7f3640 1 -- 172.21.15.98:0/3382396859 wait complete. 2024-04-25T16:52:59.363 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-04-25T16:52:59.363 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:52:59.379 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:52:59.529 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 -- 172.21.15.98:0/1500875046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98581029c0 msgr2=0x7f98581036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 --2- 172.21.15.98:0/1500875046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98581029c0 0x7f98581036c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f9848009900 tx=0x7f984802a410 comp rx=0 tx=0).stop 2024-04-25T16:52:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 -- 172.21.15.98:0/1500875046 shutdown_connections 2024-04-25T16:52:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 --2- 172.21.15.98:0/1500875046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9858108900 0x7f985810ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 --2- 172.21.15.98:0/1500875046 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9858103c00 0x7f9858104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 --2- 172.21.15.98:0/1500875046 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f98581029c0 0x7f98581036c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.530 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 -- 172.21.15.98:0/1500875046 >> 172.21.15.98:0/1500875046 conn(0x7f98580fdbd0 msgr2=0x7f9858100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 -- 172.21.15.98:0/1500875046 shutdown_connections 2024-04-25T16:52:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 -- 172.21.15.98:0/1500875046 wait complete. 2024-04-25T16:52:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 Processor -- start 2024-04-25T16:52:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.530+0000 7f985f5f3640 1 -- start start 2024-04-25T16:52:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985f5f3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98581029c0 0x7f985806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985f5f3640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9858103c00 0x7f98581a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.531 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985f5f3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9858108900 0x7f98581a3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985f5f3640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f985810e350 con 0x7f9858103c00 2024-04-25T16:52:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985f5f3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f985810e1d0 con 0x7f9858108900 2024-04-25T16:52:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985f5f3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f985810e4d0 con 0x7f98581029c0 2024-04-25T16:52:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9858103c00 0x7f98581a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9858103c00 0x7f98581a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48158/0 (socket says 172.21.15.98:48158) 2024-04-25T16:52:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 -- 172.21.15.98:0/2520465882 learned_addr learned my addr 172.21.15.98:0/2520465882 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985d38f640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98581029c0 0x7f985806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.532 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985db90640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9858108900 0x7f98581a3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 -- 172.21.15.98:0/2520465882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98581029c0 msgr2=0x7f985806a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98581029c0 0x7f985806a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 -- 172.21.15.98:0/2520465882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9858108900 msgr2=0x7f98581a3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9858108900 0x7f98581a3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 -- 172.21.15.98:0/2520465882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98581a3e80 con 0x7f9858103c00 2024-04-25T16:52:59.533 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985cb8e640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9858103c00 0x7f98581a12f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f985400d950 tx=0x7f985400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:59.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f984e7fc640 1 -- 172.21.15.98:0/2520465882 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9854017070 con 0x7f9858103c00 2024-04-25T16:52:59.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98581a4170 con 0x7f9858103c00 2024-04-25T16:52:59.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f984e7fc640 1 -- 172.21.15.98:0/2520465882 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9854009c30 con 0x7f9858103c00 2024-04-25T16:52:59.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f984e7fc640 1 -- 172.21.15.98:0/2520465882 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985400f620 con 0x7f9858103c00 2024-04-25T16:52:59.534 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.531+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98581a46c0 con 0x7f9858103c00 2024-04-25T16:52:59.535 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.533+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9858107710 con 0x7f9858103c00 2024-04-25T16:52:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.534+0000 7f984e7fc640 1 -- 172.21.15.98:0/2520465882 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f9854010460 con 0x7f9858103c00 2024-04-25T16:52:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.534+0000 7f984e7fc640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f98440afaf0 0x7f98440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.535+0000 7f985d38f640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f98440afaf0 0x7f98440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.535+0000 7f984e7fc640 1 -- 172.21.15.98:0/2520465882 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f98540df340 con 0x7f9858103c00 2024-04-25T16:52:59.536 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.535+0000 7f985d38f640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f98440afaf0 0x7f98440b2000 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9848005760 tx=0x7f9848000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:59.537 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.539+0000 7f984e7fc640 1 -- 172.21.15.98:0/2520465882 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98540a7460 con 0x7f9858103c00 2024-04-25T16:52:59.715 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.717+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f985802db50 con 0x7f9858103c00 2024-04-25T16:52:59.716 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.719+0000 7f984e7fc640 1 -- 172.21.15.98:0/2520465882 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+27569 (secure 0 0 0) 0x7f985802db50 con 0x7f9858103c00 2024-04-25T16:52:59.717 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-04-25T16:52:59.717 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":106,"fsid":"c520eaa8-031f-11ef-bc93-c7b262605968","created":"2024-04-25T16:22:09.449249+0000","modified":"2024-04-25T16:51:36.553341+0000","last_up_change":"2024-04-25T16:29:20.977316+0000","last_in_change":"2024-04-25T16:29:01.006691+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T16:25:33.926482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T16:30:07.019946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T16:30:08.292692+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T16:30:11.984469+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"738a5e3f-5992-4c5f-befd-a6130241c120","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6803","nonce":2011959581}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6805","nonce":2011959581}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6809","nonce":2011959581}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":2011959581},{"type":"v1","addr":"172.21.15.98:6807","nonce":2011959581}]},"public_addr":"172.21.15.98:6803/2011959581","cluster_addr":"172.21.15.98:6805/2011959581","heartbeat_back_addr":"172.21.15.98:6809/2011959581","heartbeat_front_addr":"172.21.15.98:6807/2011959581","state":["exists","up"]},{"osd":1,"uuid":"1d963fea-ee1e-4f05-af8e-31a28d82c49c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6801","nonce":2144972305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6803","nonce":2144972305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6807","nonce":2144972305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":2144972305},{"type":"v1","addr":"172.21.15.173:6805","nonce":2144972305}]},"public_addr":"172.21.15.173:6801/2144972305","cluster_addr":"172.21.15.173:6803/2144972305","heartbeat_back_addr":"172.21.15.173:6807/2144972305","heartbeat_front_addr":"172.21.15.173:6805/2144972305","state":["exists","up"]},{"osd":2,"uuid":"ea05c891-ec27-45e1-9226-2f3d5752a73f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6801","nonce":3173578158}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6803","nonce":3173578158}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6807","nonce":3173578158}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":3173578158},{"type":"v1","addr":"172.21.15.203:6805","nonce":3173578158}]},"public_addr":"172.21.15.203:6801/3173578158","cluster_addr":"172.21.15.203:6803/3173578158","heartbeat_back_addr":"172.21.15.203:6807/3173578158","heartbeat_front_addr":"172.21.15.203:6805/3173578158","state":["exists","up"]},{"osd":3,"uuid":"fb6d74f9-c730-4a72-a11f-141d71e4ba71","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6811","nonce":214861667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6813","nonce":214861667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6817","nonce":214861667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":214861667},{"type":"v1","addr":"172.21.15.98:6815","nonce":214861667}]},"public_addr":"172.21.15.98:6811/214861667","cluster_addr":"172.21.15.98:6813/214861667","heartbeat_back_addr":"172.21.15.98:6817/214861667","heartbeat_front_addr":"172.21.15.98:6815/214861667","state":["exists","up"]},{"osd":4,"uuid":"7375471d-1a21-428b-855f-8d9cd5f5e5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6809","nonce":1101244004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6811","nonce":1101244004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6815","nonce":1101244004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":1101244004},{"type":"v1","addr":"172.21.15.173:6813","nonce":1101244004}]},"public_addr":"172.21.15.173:6809/1101244004","cluster_addr":"172.21.15.173:6811/1101244004","heartbeat_back_addr":"172.21.15.173:6815/1101244004","heartbeat_front_addr":"172.21.15.173:6813/1101244004","state":["exists","up"]},{"osd":5,"uuid":"61fbd3e8-1e5f-46b8-ae2f-8ee3df02455f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6809","nonce":4007044064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6811","nonce":4007044064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6815","nonce":4007044064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":4007044064},{"type":"v1","addr":"172.21.15.203:6813","nonce":4007044064}]},"public_addr":"172.21.15.203:6809/4007044064","cluster_addr":"172.21.15.203:6811/4007044064","heartbeat_back_addr":"172.21.15.203:6815/4007044064","heartbeat_front_addr":"172.21.15.203:6813/4007044064","state":["exists","up"]},{"osd":6,"uuid":"89aa247d-c76d-4fa9-a368-bc45954b8b3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6819","nonce":3921819389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6821","nonce":3921819389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6825","nonce":3921819389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3921819389},{"type":"v1","addr":"172.21.15.98:6823","nonce":3921819389}]},"public_addr":"172.21.15.98:6819/3921819389","cluster_addr":"172.21.15.98:6821/3921819389","heartbeat_back_addr":"172.21.15.98:6825/3921819389","heartbeat_front_addr":"172.21.15.98:6823/3921819389","state":["exists","up"]},{"osd":7,"uuid":"c4a90448-6dcb-4dd1-aec5-cc423090bdf0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6817","nonce":442659395}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6819","nonce":442659395}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6823","nonce":442659395}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":442659395},{"type":"v1","addr":"172.21.15.173:6821","nonce":442659395}]},"public_addr":"172.21.15.173:6817/442659395","cluster_addr":"172.21.15.173:6819/442659395","heartbeat_back_addr":"172.21.15.173:6823/442659395","heartbeat_front_addr":"172.21.15.173:6821/442659395","state":["exists","up"]},{"osd":8,"uuid":"921fce6a-8fac-4547-b46a-b177454e3b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6816","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6817","nonce":3847547758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6819","nonce":3847547758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6823","nonce":3847547758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6820","nonce":3847547758},{"type":"v1","addr":"172.21.15.203:6821","nonce":3847547758}]},"public_addr":"172.21.15.203:6817/3847547758","cluster_addr":"172.21.15.203:6819/3847547758","heartbeat_back_addr":"172.21.15.203:6823/3847547758","heartbeat_front_addr":"172.21.15.203:6821/3847547758","state":["exists","up"]},{"osd":9,"uuid":"53d9c2d6-bded-41da-a30b-7b4d2c44b55a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6827","nonce":3292339123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6829","nonce":3292339123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6833","nonce":3292339123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3292339123},{"type":"v1","addr":"172.21.15.98:6831","nonce":3292339123}]},"public_addr":"172.21.15.98:6827/3292339123","cluster_addr":"172.21.15.98:6829/3292339123","heartbeat_back_addr":"172.21.15.98:6833/3292339123","heartbeat_front_addr":"172.21.15.98:6831/3292339123","state":["exists","up"]},{"osd":10,"uuid":"18d5642e-ef6c-41dd-8aa4-9fd5190d03f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6825","nonce":3946840322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6826","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6827","nonce":3946840322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6831","nonce":3946840322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":3946840322},{"type":"v1","addr":"172.21.15.173:6829","nonce":3946840322}]},"public_addr":"172.21.15.173:6825/3946840322","cluster_addr":"172.21.15.173:6827/3946840322","heartbeat_back_addr":"172.21.15.173:6831/3946840322","heartbeat_front_addr":"172.21.15.173:6829/3946840322","state":["exists","up"]},{"osd":11,"uuid":"09b2102c-57de-477e-9a48-008e84d76db9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6825","nonce":1060778865}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6827","nonce":1060778865}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6831","nonce":1060778865}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1060778865},{"type":"v1","addr":"172.21.15.203:6829","nonce":1060778865}]},"public_addr":"172.21.15.203:6825/1060778865","cluster_addr":"172.21.15.203:6827/1060778865","heartbeat_back_addr":"172.21.15.203:6831/1060778865","heartbeat_front_addr":"172.21.15.203:6829/1060778865","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:24:41.399396+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:06.049188+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:29.994612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:25:54.429163+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:19.944780+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:26:45.134166+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:10.783623+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:27:36.041796+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:01.328324+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:27.192460+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:28:53.219700+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T16:29:19.701974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.e","mappings":[{"from":2,"to":1}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.12","mappings":[{"from":3,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":6}]},{"pgid":"3.3c","mappings":[{"from":4,"to":9}]},{"pgid":"4.5","mappings":[{"from":5,"to":10}]},{"pgid":"4.6","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6839/2689051587":"2024-04-26T16:44:26.579258+0000","172.21.15.98:6838/2689051587":"2024-04-26T16:44:26.579258+0000","172.21.15.173:6833/2439698545":"2024-04-26T16:44:03.604569+0000","172.21.15.173:6832/2439698545":"2024-04-26T16:44:03.604569+0000","172.21.15.203:6832/2590142748":"2024-04-26T16:44:03.513876+0000","172.21.15.203:6837/3458796538":"2024-04-26T16:43:39.806154+0000","172.21.15.203:6839/3267653510":"2024-04-26T16:43:37.634911+0000","172.21.15.203:6835/4038856286":"2024-04-26T16:43:37.105127+0000","172.21.15.173:6838/3538054402":"2024-04-26T16:35:43.083983+0000","172.21.15.98:0/2573630616":"2024-04-26T16:23:07.830236+0000","172.21.15.173:6838/2873375163":"2024-04-26T16:40:54.470086+0000","172.21.15.173:6836/3031685940":"2024-04-26T16:44:26.346994+0000","172.21.15.98:6800/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6832/1945729206":"2024-04-26T16:43:15.031214+0000","172.21.15.203:6834/4038856286":"2024-04-26T16:43:37.105127+0000","172.21.15.173:6838/2050448663":"2024-04-26T16:35:09.397460+0000","172.21.15.98:0/1429360204":"2024-04-26T16:22:41.266215+0000","172.21.15.173:6839/3538054402":"2024-04-26T16:35:43.083983+0000","172.21.15.203:6834/3629133920":"2024-04-26T16:51:36.553281+0000","172.21.15.98:0/1883089564":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6836/3458796538":"2024-04-26T16:43:39.806154+0000","172.21.15.203:6838/3267653510":"2024-04-26T16:43:37.634911+0000","172.21.15.203:6834/1151278804":"2024-04-26T16:40:28.525704+0000","172.21.15.98:0/119631250":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6836/998442662":"2024-04-26T16:37:09.422158+0000","172.21.15.98:6801/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.98:0/3681298222":"2024-04-26T16:22:41.266215+0000","172.21.15.173:6837/3031685940":"2024-04-26T16:44:26.346994+0000","172.21.15.98:6801/4161723298":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6835/3629133920":"2024-04-26T16:51:36.553281+0000","172.21.15.173:6836/311453164":"2024-04-26T16:40:25.599313+0000","172.21.15.98:0/3090764993":"2024-04-26T16:22:41.266215+0000","172.21.15.203:6839/2982027693":"2024-04-26T16:39:30.428647+0000","172.21.15.98:6800/217035597":"2024-04-26T16:23:07.830236+0000","172.21.15.203:6834/890593273":"2024-04-26T16:35:58.723078+0000","172.21.15.173:6839/2050448663":"2024-04-26T16:35:09.397460+0000","172.21.15.203:6835/890593273":"2024-04-26T16:35:58.723078+0000","172.21.15.173:6838/3664329633":"2024-04-26T16:36:17.370724+0000","172.21.15.173:6839/3664329633":"2024-04-26T16:36:17.370724+0000","172.21.15.203:6833/2590142748":"2024-04-26T16:44:03.513876+0000","172.21.15.203:6837/998442662":"2024-04-26T16:37:09.422158+0000","172.21.15.203:6838/2982027693":"2024-04-26T16:39:30.428647+0000","172.21.15.203:6835/2230353021":"2024-04-26T16:35:26.239567+0000","172.21.15.173:6837/311453164":"2024-04-26T16:40:25.599313+0000","172.21.15.203:6833/1945729206":"2024-04-26T16:43:15.031214+0000","172.21.15.203:6834/2230353021":"2024-04-26T16:35:26.239567+0000","172.21.15.203:6835/1151278804":"2024-04-26T16:40:28.525704+0000","172.21.15.173:6832/3037734321":"2024-04-26T16:43:15.040785+0000","172.21.15.173:6839/2873375163":"2024-04-26T16:40:54.470086+0000","172.21.15.98:6838/3308601909":"2024-04-26T16:41:27.323250+0000","172.21.15.98:6839/3308601909":"2024-04-26T16:41:27.323250+0000","172.21.15.173:6833/3037734321":"2024-04-26T16:43:15.040785+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:52:59.719 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.721+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f98440afaf0 msgr2=0x7f98440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.721+0000 7f985f5f3640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f98440afaf0 0x7f98440b2000 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f9848005760 tx=0x7f9848000bc0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9858103c00 msgr2=0x7f98581a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9858103c00 0x7f98581a12f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f985400d950 tx=0x7f985400de10 comp rx=0 tx=0).stop 2024-04-25T16:52:59.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 shutdown_connections 2024-04-25T16:52:59.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f98440afaf0 0x7f98440b2000 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9858108900 0x7f98581a3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f9858103c00 0x7f98581a12f0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.720 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 --2- 172.21.15.98:0/2520465882 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98581029c0 0x7f985806a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 >> 172.21.15.98:0/2520465882 conn(0x7f98580fdbd0 msgr2=0x7f98581048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:59.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 shutdown_connections 2024-04-25T16:52:59.721 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.722+0000 7f985f5f3640 1 -- 172.21.15.98:0/2520465882 wait complete. 2024-04-25T16:52:59.729 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:59 smithi098 ceph-mon[29162]: pgmap v991: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T16:52:59.730 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:59 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3742707696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:52:59.730 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:59 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3052018535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:52:59.730 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:52:59 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3382396859' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:52:59.731 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T16:25:33.926482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T16:30:07.019946+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T16:30:08.292692+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T16:30:11.984469+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T16:52:59.732 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:52:59.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:59 smithi173 ceph-mon[32556]: pgmap v991: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T16:52:59.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:59 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3742707696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:52:59.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:59 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3052018535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:52:59.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:52:59 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3382396859' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:52:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 -- 172.21.15.98:0/2114876688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 msgr2=0x7fc324104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 --2- 172.21.15.98:0/2114876688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 0x7fc324104800 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fc314009900 tx=0x7fc31402a410 comp rx=0 tx=0).stop 2024-04-25T16:52:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 -- 172.21.15.98:0/2114876688 shutdown_connections 2024-04-25T16:52:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 --2- 172.21.15.98:0/2114876688 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc324109a40 0x7fc32410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 --2- 172.21.15.98:0/2114876688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc324104d40 0x7fc324105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 --2- 172.21.15.98:0/2114876688 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 0x7fc324104800 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.843 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 -- 172.21.15.98:0/2114876688 >> 172.21.15.98:0/2114876688 conn(0x7fc3240fecf0 msgr2=0x7fc3241011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:52:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 -- 172.21.15.98:0/2114876688 shutdown_connections 2024-04-25T16:52:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 -- 172.21.15.98:0/2114876688 wait complete. 2024-04-25T16:52:59.844 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.845+0000 7fc32a084640 1 Processor -- start 2024-04-25T16:52:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc32a084640 1 -- start start 2024-04-25T16:52:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc32a084640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 0x7fc32419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc32a084640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc324104d40 0x7fc32419a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc32a084640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc324109a40 0x7fc3241a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.846 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc32a084640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7fc32410f4b0 con 0x7fc324103b00 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc32a084640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7fc32410f330 con 0x7fc324104d40 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc32a084640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7fc32410f630 con 0x7fc324109a40 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc3237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 0x7fc32419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc3237fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 0x7fc32419a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48186/0 (socket says 172.21.15.98:48186) 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc3237fe640 1 -- 172.21.15.98:0/4138944500 learned_addr learned my addr 172.21.15.98:0/4138944500 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc323fff640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc324109a40 0x7fc3241a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc322ffd640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc324104d40 0x7fc32419a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc3237fe640 1 -- 172.21.15.98:0/4138944500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc324109a40 msgr2=0x7fc3241a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.847 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc3237fe640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc324109a40 0x7fc3241a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc3237fe640 1 -- 172.21.15.98:0/4138944500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc324104d40 msgr2=0x7fc32419a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc3237fe640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc324104d40 0x7fc32419a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc3237fe640 1 -- 172.21.15.98:0/4138944500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3241a1cb0 con 0x7fc324103b00 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.846+0000 7fc322ffd640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc324104d40 0x7fc32419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.847+0000 7fc3237fe640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 0x7fc32419a170 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc314005760 tx=0x7fc314004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.847+0000 7fc320ff9640 1 -- 172.21.15.98:0/4138944500 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc31403f030 con 0x7fc324103b00 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.847+0000 7fc320ff9640 1 -- 172.21.15.98:0/4138944500 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc314003a80 con 0x7fc324103b00 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.847+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3241a1f40 con 0x7fc324103b00 2024-04-25T16:52:59.848 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.847+0000 7fc320ff9640 1 -- 172.21.15.98:0/4138944500 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc314035070 con 0x7fc324103b00 2024-04-25T16:52:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.847+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3241a2340 con 0x7fc324103b00 2024-04-25T16:52:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.848+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc324108850 con 0x7fc324103b00 2024-04-25T16:52:59.849 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.849+0000 7fc320ff9640 1 -- 172.21.15.98:0/4138944500 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fc314042030 con 0x7fc324103b00 2024-04-25T16:52:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.850+0000 7fc320ff9640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc30c0afaf0 0x7fc30c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:52:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.850+0000 7fc322ffd640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc30c0afaf0 0x7fc30c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:52:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.850+0000 7fc320ff9640 1 -- 172.21.15.98:0/4138944500 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7fc3141001e0 con 0x7fc324103b00 2024-04-25T16:52:59.850 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.850+0000 7fc322ffd640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc30c0afaf0 0x7fc30c0b2000 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc318005fd0 tx=0x7fc318005850 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:52:59.851 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:52:59.854+0000 7fc320ff9640 1 -- 172.21.15.98:0/4138944500 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3140c8300 con 0x7fc324103b00 2024-04-25T16:52:59.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:59 smithi203 ceph-mon[33115]: pgmap v991: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-25T16:52:59.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:59 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3742707696' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:52:59.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:59 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3052018535' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:52:59.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:52:59 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3382396859' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:53:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.031+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc32402dce0 con 0x7fc324103b00 2024-04-25T16:53:00.029 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.032+0000 7fc320ff9640 1 -- 172.21.15.98:0/4138944500 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7fc3140d0380 con 0x7fc324103b00 2024-04-25T16:53:00.029 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-04-25T16:53:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.033+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc30c0afaf0 msgr2=0x7fc30c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.031 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc30c0afaf0 0x7fc30c0b2000 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc318005fd0 tx=0x7fc318005850 comp rx=0 tx=0).stop 2024-04-25T16:53:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 msgr2=0x7fc32419a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 0x7fc32419a170 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fc314005760 tx=0x7fc314004270 comp rx=0 tx=0).stop 2024-04-25T16:53:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 shutdown_connections 2024-04-25T16:53:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7fc30c0afaf0 0x7fc30c0b2000 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc324109a40 0x7fc3241a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc324104d40 0x7fc32419a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 --2- 172.21.15.98:0/4138944500 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7fc324103b00 0x7fc32419a170 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.032 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 >> 172.21.15.98:0/4138944500 conn(0x7fc3240fecf0 msgr2=0x7fc324105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 shutdown_connections 2024-04-25T16:53:00.033 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.034+0000 7fc32a084640 1 -- 172.21.15.98:0/4138944500 wait complete. 2024-04-25T16:53:00.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:53:00.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 -- 172.21.15.98:0/1331796300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384104460 msgr2=0x7f438410abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 --2- 172.21.15.98:0/1331796300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384104460 0x7f438410abb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f437c009900 tx=0x7f437c02a410 comp rx=0 tx=0).stop 2024-04-25T16:53:00.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 -- 172.21.15.98:0/1331796300 shutdown_connections 2024-04-25T16:53:00.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 --2- 172.21.15.98:0/1331796300 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384104460 0x7f438410abb0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 --2- 172.21.15.98:0/1331796300 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4384103940 0x7f4384103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.155 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 --2- 172.21.15.98:0/1331796300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4384102f60 0x7f4384103370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 -- 172.21.15.98:0/1331796300 >> 172.21.15.98:0/1331796300 conn(0x7f43840fd340 msgr2=0x7f43840ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:00.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 -- 172.21.15.98:0/1331796300 shutdown_connections 2024-04-25T16:53:00.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 -- 172.21.15.98:0/1331796300 wait complete. 2024-04-25T16:53:00.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 Processor -- start 2024-04-25T16:53:00.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.155+0000 7f438c6b1640 1 -- start start 2024-04-25T16:53:00.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438c6b1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4384102f60 0x7f4384102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438c6b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384103940 0x7f438419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.156 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438c6b1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4384104460 0x7f43841a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438c6b1640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f438410e3c0 con 0x7f4384103940 2024-04-25T16:53:00.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438c6b1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f438410e240 con 0x7f4384102f60 2024-04-25T16:53:00.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438c6b1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f438410e540 con 0x7f4384104460 2024-04-25T16:53:00.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384103940 0x7f438419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384103940 0x7f438419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48208/0 (socket says 172.21.15.98:48208) 2024-04-25T16:53:00.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 -- 172.21.15.98:0/3621320869 learned_addr learned my addr 172.21.15.98:0/3621320869 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:53:00.157 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438a44d640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4384102f60 0x7f4384102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 -- 172.21.15.98:0/3621320869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4384104460 msgr2=0x7f43841a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438ac4e640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4384104460 0x7f43841a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4384104460 0x7f43841a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 -- 172.21.15.98:0/3621320869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4384102f60 msgr2=0x7f4384102200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4384102f60 0x7f4384102200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 -- 172.21.15.98:0/3621320869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43841a1bb0 con 0x7f4384103940 2024-04-25T16:53:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438ac4e640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4384104460 0x7f43841a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:53:00.158 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438a44d640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4384102f60 0x7f4384102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:53:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f4389c4c640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384103940 0x7f438419ee40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f437400d950 tx=0x7f437400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:53:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f437b7fe640 1 -- 172.21.15.98:0/3621320869 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374017070 con 0x7f4384103940 2024-04-25T16:53:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f437b7fe640 1 -- 172.21.15.98:0/3621320869 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4374009c30 con 0x7f4384103940 2024-04-25T16:53:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f437b7fe640 1 -- 172.21.15.98:0/3621320869 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f437400f620 con 0x7f4384103940 2024-04-25T16:53:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43841a1ea0 con 0x7f4384103940 2024-04-25T16:53:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.156+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43841a2360 con 0x7f4384103940 2024-04-25T16:53:00.159 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.157+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4384109910 con 0x7f4384103940 2024-04-25T16:53:00.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.159+0000 7f437b7fe640 1 -- 172.21.15.98:0/3621320869 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f4374010460 con 0x7f4384103940 2024-04-25T16:53:00.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.160+0000 7f437b7fe640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43700afb70 0x7f43700b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.160+0000 7f438a44d640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43700afb70 0x7f43700b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.160 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.160+0000 7f437b7fe640 1 -- 172.21.15.98:0/3621320869 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f43740df320 con 0x7f4384103940 2024-04-25T16:53:00.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.160+0000 7f438a44d640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43700afb70 0x7f43700b2080 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4380005c40 tx=0x7f438000a430 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:53:00.161 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.163+0000 7f437b7fe640 1 -- 172.21.15.98:0/3621320869 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43740a74c0 con 0x7f4384103940 2024-04-25T16:53:00.338 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.340+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f438402dce0 con 0x7f4384103940 2024-04-25T16:53:00.339 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.341+0000 7f437b7fe640 1 -- 172.21.15.98:0/3621320869 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f4374009de0 con 0x7f4384103940 2024-04-25T16:53:00.339 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:53:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43700afb70 msgr2=0x7f43700b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43700afb70 0x7f43700b2080 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4380005c40 tx=0x7f438000a430 comp rx=0 tx=0).stop 2024-04-25T16:53:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384103940 msgr2=0x7f438419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384103940 0x7f438419ee40 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f437400d950 tx=0x7f437400de10 comp rx=0 tx=0).stop 2024-04-25T16:53:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 shutdown_connections 2024-04-25T16:53:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f43700afb70 0x7f43700b2080 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4384104460 0x7f43841a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.341 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f4384103940 0x7f438419ee40 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 --2- 172.21.15.98:0/3621320869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4384102f60 0x7f4384102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 >> 172.21.15.98:0/3621320869 conn(0x7f43840fd340 msgr2=0x7f4384106ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 shutdown_connections 2024-04-25T16:53:00.342 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.343+0000 7f438c6b1640 1 -- 172.21.15.98:0/3621320869 wait complete. 2024-04-25T16:53:00.352 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:53:00.465 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.465+0000 7f0bd7815640 1 -- 172.21.15.98:0/1165825919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd0103c30 msgr2=0x7f0bd0104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.465+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1165825919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd0103c30 0x7f0bd0104040 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009900 tx=0x7f0bc002a410 comp rx=0 tx=0).stop 2024-04-25T16:53:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 -- 172.21.15.98:0/1165825919 shutdown_connections 2024-04-25T16:53:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1165825919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bd0108930 0x7f0bd010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1165825919 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd0103c30 0x7f0bd0104040 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.466 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1165825919 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bd01029f0 0x7f0bd01036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 -- 172.21.15.98:0/1165825919 >> 172.21.15.98:0/1165825919 conn(0x7f0bd00fdbc0 msgr2=0x7f0bd0100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 -- 172.21.15.98:0/1165825919 shutdown_connections 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 -- 172.21.15.98:0/1165825919 wait complete. 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 Processor -- start 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 -- start start 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd01029f0 0x7f0bd0075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bd0103c30 0x7f0bd019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bd0108930 0x7f0bd01a1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.467 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd010e3a0 con 0x7f0bd01029f0 2024-04-25T16:53:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd010e220 con 0x7f0bd0103c30 2024-04-25T16:53:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.466+0000 7f0bd7815640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd010e520 con 0x7f0bd0108930 2024-04-25T16:53:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd01029f0 0x7f0bd0075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd01029f0 0x7f0bd0075860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48222/0 (socket says 172.21.15.98:48222) 2024-04-25T16:53:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 -- 172.21.15.98:0/1370320202 learned_addr learned my addr 172.21.15.98:0/1370320202 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:53:00.468 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd5db2640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bd0108930 0x7f0bd01a1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 -- 172.21.15.98:0/1370320202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bd0108930 msgr2=0x7f0bd01a1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bd0108930 0x7f0bd01a1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 -- 172.21.15.98:0/1370320202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bd0103c30 msgr2=0x7f0bd019eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd4db0640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bd0103c30 0x7f0bd019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bd0103c30 0x7f0bd019eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 -- 172.21.15.98:0/1370320202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd01a1c70 con 0x7f0bd01029f0 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd4db0640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bd0103c30 0x7f0bd019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd55b1640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd01029f0 0x7f0bd0075860 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc005fd0 tx=0x7f0bcc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:53:00.469 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bc67fc640 1 -- 172.21.15.98:0/1370320202 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc017070 con 0x7f0bd01029f0 2024-04-25T16:53:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bc67fc640 1 -- 172.21.15.98:0/1370320202 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0bcc009c30 con 0x7f0bd01029f0 2024-04-25T16:53:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd01a1f60 con 0x7f0bd01029f0 2024-04-25T16:53:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bc67fc640 1 -- 172.21.15.98:0/1370320202 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc003c90 con 0x7f0bd01029f0 2024-04-25T16:53:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.467+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd01a23c0 con 0x7f0bd01029f0 2024-04-25T16:53:00.470 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.468+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd006a190 con 0x7f0bd01029f0 2024-04-25T16:53:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.470+0000 7f0bc67fc640 1 -- 172.21.15.98:0/1370320202 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f0bcc003e30 con 0x7f0bd01029f0 2024-04-25T16:53:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.470+0000 7f0bc67fc640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0bbc0afaf0 0x7f0bbc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.470+0000 7f0bd4db0640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0bbc0afaf0 0x7f0bbc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.470+0000 7f0bc67fc640 1 -- 172.21.15.98:0/1370320202 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f0bcc0dfc40 con 0x7f0bd01029f0 2024-04-25T16:53:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.471+0000 7f0bd4db0640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0bbc0afaf0 0x7f0bbc0b2000 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0005f70 tx=0x7f0bc0005720 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:53:00.472 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.474+0000 7f0bc67fc640 1 -- 172.21.15.98:0/1370320202 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0bd006a190 con 0x7f0bd01029f0 2024-04-25T16:53:00.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.653+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0bd002dbf0 con 0x7f0bd01029f0 2024-04-25T16:53:00.651 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.653+0000 7f0bc67fc640 1 -- 172.21.15.98:0/1370320202 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f0bcc020030 con 0x7f0bd01029f0 2024-04-25T16:53:00.651 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 64 2024-04-25T16:53:00.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0bbc0afaf0 msgr2=0x7f0bbc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0bbc0afaf0 0x7f0bbc0b2000 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0005f70 tx=0x7f0bc0005720 comp rx=0 tx=0).stop 2024-04-25T16:53:00.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd01029f0 msgr2=0x7f0bd0075860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.653 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd01029f0 0x7f0bd0075860 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc005fd0 tx=0x7f0bcc00dde0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 shutdown_connections 2024-04-25T16:53:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f0bbc0afaf0 0x7f0bbc0b2000 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0bd0108930 0x7f0bd01a1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bd0103c30 0x7f0bd019eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 --2- 172.21.15.98:0/1370320202 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f0bd01029f0 0x7f0bd0075860 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 >> 172.21.15.98:0/1370320202 conn(0x7f0bd00fdbc0 msgr2=0x7f0bd0104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.655+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 shutdown_connections 2024-04-25T16:53:00.654 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.656+0000 7f0bd7815640 1 -- 172.21.15.98:0/1370320202 wait complete. 2024-04-25T16:53:00.663 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:00 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/2520465882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:53:00.663 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:00 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/4138944500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:53:00.663 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:00 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3621320869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:53:00.664 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T16:53:00.717 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:00 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/2520465882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:53:00.717 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:00 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/4138944500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:53:00.717 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:00 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3621320869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:53:00.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:00 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/2520465882' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:53:00.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:00 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/4138944500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:53:00.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:00 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3621320869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:53:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 -- 172.21.15.98:0/3564973181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 msgr2=0x7f2c38104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 --2- 172.21.15.98:0/3564973181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 0x7f2c38104030 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009900 tx=0x7f2c2802a410 comp rx=0 tx=0).stop 2024-04-25T16:53:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 -- 172.21.15.98:0/3564973181 shutdown_connections 2024-04-25T16:53:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 --2- 172.21.15.98:0/3564973181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c38108920 0x7f2c3810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 --2- 172.21.15.98:0/3564973181 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 0x7f2c38104030 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.777 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 --2- 172.21.15.98:0/3564973181 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c381029e0 0x7f2c381036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 -- 172.21.15.98:0/3564973181 >> 172.21.15.98:0/3564973181 conn(0x7f2c380fdbd0 msgr2=0x7f2c38100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 -- 172.21.15.98:0/3564973181 shutdown_connections 2024-04-25T16:53:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 -- 172.21.15.98:0/3564973181 wait complete. 2024-04-25T16:53:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 Processor -- start 2024-04-25T16:53:00.778 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.779+0000 7f2c3e292640 1 -- start start 2024-04-25T16:53:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c3e292640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c381029e0 0x7f2c3819a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.780 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c3e292640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 0x7f2c3819a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c3e292640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c38108920 0x7f2c381a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c3e292640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810e410 con 0x7f2c38103c20 2024-04-25T16:53:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c3e292640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810e290 con 0x7f2c38108920 2024-04-25T16:53:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c3e292640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2c3810e590 con 0x7f2c381029e0 2024-04-25T16:53:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 0x7f2c3819a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 0x7f2c3819a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48244/0 (socket says 172.21.15.98:48244) 2024-04-25T16:53:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 -- 172.21.15.98:0/1625473870 learned_addr learned my addr 172.21.15.98:0/1625473870 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:53:00.781 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c37fff640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c381029e0 0x7f2c3819a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c3c82f640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c38108920 0x7f2c381a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 -- 172.21.15.98:0/1625473870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c381029e0 msgr2=0x7f2c3819a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c381029e0 0x7f2c3819a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 -- 172.21.15.98:0/1625473870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c38108920 msgr2=0x7f2c381a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c38108920 0x7f2c381a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 -- 172.21.15.98:0/1625473870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c381a1ff0 con 0x7f2c38103c20 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c3c82f640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c38108920 0x7f2c381a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.780+0000 7f2c377fe640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 0x7f2c3819a9f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009230 tx=0x7f2c280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:53:00.782 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.781+0000 7f2c357fa640 1 -- 172.21.15.98:0/1625473870 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c2803f030 con 0x7f2c38103c20 2024-04-25T16:53:00.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.781+0000 7f2c357fa640 1 -- 172.21.15.98:0/1625473870 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c28003c90 con 0x7f2c38103c20 2024-04-25T16:53:00.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.781+0000 7f2c357fa640 1 -- 172.21.15.98:0/1625473870 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c28035070 con 0x7f2c38103c20 2024-04-25T16:53:00.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.781+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c381a2280 con 0x7f2c38103c20 2024-04-25T16:53:00.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.781+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c381a2770 con 0x7f2c38103c20 2024-04-25T16:53:00.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.782+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c3806c2a0 con 0x7f2c38103c20 2024-04-25T16:53:00.783 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.783+0000 7f2c357fa640 1 -- 172.21.15.98:0/1625473870 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f2c28042030 con 0x7f2c38103c20 2024-04-25T16:53:00.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.784+0000 7f2c357fa640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c200afaf0 0x7f2c200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:00.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.784+0000 7f2c37fff640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c200afaf0 0x7f2c200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:00.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.784+0000 7f2c357fa640 1 -- 172.21.15.98:0/1625473870 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f2c280ffcb0 con 0x7f2c38103c20 2024-04-25T16:53:00.784 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.784+0000 7f2c37fff640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c200afaf0 0x7f2c200b2000 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c005fd0 tx=0x7f2c2c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:53:00.785 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.788+0000 7f2c357fa640 1 -- 172.21.15.98:0/1625473870 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c280c7dd0 con 0x7f2c38103c20 2024-04-25T16:53:00.961 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.964+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2c381a29d0 con 0x7f2c38103c20 2024-04-25T16:53:00.962 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.964+0000 7f2c357fa640 1 -- 172.21.15.98:0/1625473870 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v106) ==== 103+0+11 (secure 0 0 0) 0x7f2c280c76d0 con 0x7f2c38103c20 2024-04-25T16:53:00.962 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 32 2024-04-25T16:53:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.966+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c200afaf0 msgr2=0x7f2c200b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.966+0000 7f2c3e292640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c200afaf0 0x7f2c200b2000 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2c2c005fd0 tx=0x7f2c2c00b410 comp rx=0 tx=0).stop 2024-04-25T16:53:00.964 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.966+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 msgr2=0x7f2c3819a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.966+0000 7f2c3e292640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 0x7f2c3819a9f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2c28009230 tx=0x7f2c280042f0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.967+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 shutdown_connections 2024-04-25T16:53:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.967+0000 7f2c3e292640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2c200afaf0 0x7f2c200b2000 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.967+0000 7f2c3e292640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2c38108920 0x7f2c381a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.967+0000 7f2c3e292640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2c38103c20 0x7f2c3819a9f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.967+0000 7f2c3e292640 1 --2- 172.21.15.98:0/1625473870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2c381029e0 0x7f2c3819a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.967+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 >> 172.21.15.98:0/1625473870 conn(0x7f2c380fdbd0 msgr2=0x7f2c38069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:00.965 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.967+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 shutdown_connections 2024-04-25T16:53:00.966 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:00.967+0000 7f2c3e292640 1 -- 172.21.15.98:0/1625473870 wait complete. 2024-04-25T16:53:00.976 DEBUG:teuthology.orchestra.run.smithi098:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T16:53:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 -- 172.21.15.98:0/4119434295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8103b00 msgr2=0x7f2ee8104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 --2- 172.21.15.98:0/4119434295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8103b00 0x7f2ee8104800 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0009900 tx=0x7f2ee002a410 comp rx=0 tx=0).stop 2024-04-25T16:53:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 -- 172.21.15.98:0/4119434295 shutdown_connections 2024-04-25T16:53:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 --2- 172.21.15.98:0/4119434295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8109a40 0x7f2ee810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 --2- 172.21.15.98:0/4119434295 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8104d40 0x7f2ee8105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.088 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 --2- 172.21.15.98:0/4119434295 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8103b00 0x7f2ee8104800 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 -- 172.21.15.98:0/4119434295 >> 172.21.15.98:0/4119434295 conn(0x7f2ee80fecf0 msgr2=0x7f2ee81011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 -- 172.21.15.98:0/4119434295 shutdown_connections 2024-04-25T16:53:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 -- 172.21.15.98:0/4119434295 wait complete. 2024-04-25T16:53:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 Processor -- start 2024-04-25T16:53:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 -- start start 2024-04-25T16:53:01.089 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8103b00 0x7f2ee8076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.088+0000 7f2ef055f640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8104d40 0x7f2ee8074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2ef055f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8109a40 0x7f2ee8075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2ef055f640 1 -- --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee810f490 con 0x7f2ee8104d40 2024-04-25T16:53:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2ef055f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee810f310 con 0x7f2ee8103b00 2024-04-25T16:53:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2ef055f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee810f610 con 0x7f2ee8109a40 2024-04-25T16:53:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8104d40 0x7f2ee8074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:01.090 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 --2- >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8104d40 0x7f2ee8074ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.98:3300/0 says I am v2:172.21.15.98:48270/0 (socket says 172.21.15.98:48270) 2024-04-25T16:53:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 -- 172.21.15.98:0/3152352641 learned_addr learned my addr 172.21.15.98:0/3152352641 (peer_addr_for_me v2:172.21.15.98:0/0) 2024-04-25T16:53:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eee2fb640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8103b00 0x7f2ee8076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eeeafc640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8109a40 0x7f2ee8075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 -- 172.21.15.98:0/3152352641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8109a40 msgr2=0x7f2ee8075430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:01.091 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8109a40 0x7f2ee8075430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 -- 172.21.15.98:0/3152352641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8103b00 msgr2=0x7f2ee8076850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8103b00 0x7f2ee8076850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 -- 172.21.15.98:0/3152352641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee8075b60 con 0x7f2ee8104d40 2024-04-25T16:53:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2eedafa640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8104d40 0x7f2ee8074ef0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800d950 tx=0x7f2ed800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:53:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2edf7fe640 1 -- 172.21.15.98:0/3152352641 <== mon.0 v2:172.21.15.98:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8017030 con 0x7f2ee8104d40 2024-04-25T16:53:01.092 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2edf7fe640 1 -- 172.21.15.98:0/3152352641 <== mon.0 v2:172.21.15.98:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2ed8004020 con 0x7f2ee8104d40 2024-04-25T16:53:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee81a4240 con 0x7f2ee8104d40 2024-04-25T16:53:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2edf7fe640 1 -- 172.21.15.98:0/3152352641 <== mon.0 v2:172.21.15.98:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed80055d0 con 0x7f2ee8104d40 2024-04-25T16:53:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.089+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee81a4700 con 0x7f2ee8104d40 2024-04-25T16:53:01.093 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.090+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee8108850 con 0x7f2ee8104d40 2024-04-25T16:53:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.092+0000 7f2edf7fe640 1 -- 172.21.15.98:0/3152352641 <== mon.0 v2:172.21.15.98:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f2ed8009a30 con 0x7f2ee8104d40 2024-04-25T16:53:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.092+0000 7f2edf7fe640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ed40afaf0 0x7f2ed40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:53:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.092+0000 7f2eee2fb640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ed40afaf0 0x7f2ed40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:53:01.094 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.092+0000 7f2edf7fe640 1 -- 172.21.15.98:0/3152352641 <== mon.0 v2:172.21.15.98:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 10985+0+0 (secure 0 0 0) 0x7f2ed80df700 con 0x7f2ee8104d40 2024-04-25T16:53:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.093+0000 7f2eee2fb640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ed40afaf0 0x7f2ed40b2000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0005760 tx=0x7f2ee0000bc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:53:01.095 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.096+0000 7f2edf7fe640 1 -- 172.21.15.98:0/3152352641 <== mon.0 v2:172.21.15.98:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ed80a7820 con 0x7f2ee8104d40 2024-04-25T16:53:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.286+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 --> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2ee802db50 con 0x7f2ee8104d40 2024-04-25T16:53:01.284 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.287+0000 7f2edf7fe640 1 -- 172.21.15.98:0/3152352641 <== mon.0 v2:172.21.15.98:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 52 entries v106) ==== 81+0+3183 (secure 0 0 0) 0x7f2ed80a7120 con 0x7f2ee8104d40 2024-04-25T16:53:01.285 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6839/2689051587 2024-04-26T16:44:26.579258+0000 2024-04-25T16:53:01.285 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6838/2689051587 2024-04-26T16:44:26.579258+0000 2024-04-25T16:53:01.285 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6833/2439698545 2024-04-26T16:44:03.604569+0000 2024-04-25T16:53:01.285 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6832/2439698545 2024-04-26T16:44:03.604569+0000 2024-04-25T16:53:01.285 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6832/2590142748 2024-04-26T16:44:03.513876+0000 2024-04-25T16:53:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6837/3458796538 2024-04-26T16:43:39.806154+0000 2024-04-25T16:53:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6839/3267653510 2024-04-26T16:43:37.634911+0000 2024-04-25T16:53:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/4038856286 2024-04-26T16:43:37.105127+0000 2024-04-25T16:53:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6838/3538054402 2024-04-26T16:35:43.083983+0000 2024-04-25T16:53:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/2573630616 2024-04-26T16:23:07.830236+0000 2024-04-25T16:53:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6838/2873375163 2024-04-26T16:40:54.470086+0000 2024-04-25T16:53:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6836/3031685940 2024-04-26T16:44:26.346994+0000 2024-04-25T16:53:01.286 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/4161723298 2024-04-26T16:22:41.266215+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6832/1945729206 2024-04-26T16:43:15.031214+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/4038856286 2024-04-26T16:43:37.105127+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6838/2050448663 2024-04-26T16:35:09.397460+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1429360204 2024-04-26T16:22:41.266215+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6839/3538054402 2024-04-26T16:35:43.083983+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/3629133920 2024-04-26T16:51:36.553281+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/1883089564 2024-04-26T16:23:07.830236+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6836/3458796538 2024-04-26T16:43:39.806154+0000 2024-04-25T16:53:01.287 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6838/3267653510 2024-04-26T16:43:37.634911+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/1151278804 2024-04-26T16:40:28.525704+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/119631250 2024-04-26T16:23:07.830236+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6836/998442662 2024-04-26T16:37:09.422158+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/217035597 2024-04-26T16:23:07.830236+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3681298222 2024-04-26T16:22:41.266215+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6837/3031685940 2024-04-26T16:44:26.346994+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6801/4161723298 2024-04-26T16:22:41.266215+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/3629133920 2024-04-26T16:51:36.553281+0000 2024-04-25T16:53:01.288 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6836/311453164 2024-04-26T16:40:25.599313+0000 2024-04-25T16:53:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:0/3090764993 2024-04-26T16:22:41.266215+0000 2024-04-25T16:53:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6839/2982027693 2024-04-26T16:39:30.428647+0000 2024-04-25T16:53:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6800/217035597 2024-04-26T16:23:07.830236+0000 2024-04-25T16:53:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/890593273 2024-04-26T16:35:58.723078+0000 2024-04-25T16:53:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6839/2050448663 2024-04-26T16:35:09.397460+0000 2024-04-25T16:53:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/890593273 2024-04-26T16:35:58.723078+0000 2024-04-25T16:53:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6838/3664329633 2024-04-26T16:36:17.370724+0000 2024-04-25T16:53:01.289 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6839/3664329633 2024-04-26T16:36:17.370724+0000 2024-04-25T16:53:01.290 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6833/2590142748 2024-04-26T16:44:03.513876+0000 2024-04-25T16:53:01.290 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6837/998442662 2024-04-26T16:37:09.422158+0000 2024-04-25T16:53:01.290 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6838/2982027693 2024-04-26T16:39:30.428647+0000 2024-04-25T16:53:01.290 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/2230353021 2024-04-26T16:35:26.239567+0000 2024-04-25T16:53:01.290 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6837/311453164 2024-04-26T16:40:25.599313+0000 2024-04-25T16:53:01.290 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6833/1945729206 2024-04-26T16:43:15.031214+0000 2024-04-25T16:53:01.290 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6834/2230353021 2024-04-26T16:35:26.239567+0000 2024-04-25T16:53:01.291 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.203:6835/1151278804 2024-04-26T16:40:28.525704+0000 2024-04-25T16:53:01.291 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6832/3037734321 2024-04-26T16:43:15.040785+0000 2024-04-25T16:53:01.291 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6839/2873375163 2024-04-26T16:40:54.470086+0000 2024-04-25T16:53:01.291 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6838/3308601909 2024-04-26T16:41:27.323250+0000 2024-04-25T16:53:01.291 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.98:6839/3308601909 2024-04-26T16:41:27.323250+0000 2024-04-25T16:53:01.291 INFO:teuthology.orchestra.run.smithi098.stdout:172.21.15.173:6833/3037734321 2024-04-26T16:43:15.040785+0000 2024-04-25T16:53:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:listed 52 entries 2024-04-25T16:53:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ed40afaf0 msgr2=0x7f2ed40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ed40afaf0 0x7f2ed40b2000 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0005760 tx=0x7f2ee0000bc0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8104d40 msgr2=0x7f2ee8074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:53:01.293 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8104d40 0x7f2ee8074ef0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2ed800d950 tx=0x7f2ed800de10 comp rx=0 tx=0).stop 2024-04-25T16:53:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 shutdown_connections 2024-04-25T16:53:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:6800/4283487583,v1:172.21.15.98:6801/4283487583] conn(0x7f2ed40afaf0 0x7f2ed40b2000 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ee8109a40 0x7f2ee8075430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] conn(0x7f2ee8104d40 0x7f2ee8074ef0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 --2- 172.21.15.98:0/3152352641 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ee8103b00 0x7f2ee8076850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:53:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 >> 172.21.15.98:0/3152352641 conn(0x7f2ee80fecf0 msgr2=0x7f2ee8105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:53:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 shutdown_connections 2024-04-25T16:53:01.294 INFO:teuthology.orchestra.run.smithi098.stderr:2024-04-25T16:53:01.289+0000 7f2ef055f640 1 -- 172.21.15.98:0/3152352641 wait complete. 2024-04-25T16:53:01.298 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi098.front.sepia.ceph.com... 2024-04-25T16:53:01.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:53:01.298 DEBUG:teuthology.orchestra.run.smithi098:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T16:53:01.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi098.stderr:ceph-fuse[83058]: fuse finished with error 0 and tester_r 0 2024-04-25T16:53:01.349 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T16:53:01.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:01 smithi098 ceph-mon[29162]: pgmap v992: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T16:53:01.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:01 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1370320202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:53:01.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:01 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/1625473870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:53:01.751 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:01 smithi098 ceph-mon[29162]: from='client.? 172.21.15.98:0/3152352641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:53:01.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:01 smithi173 ceph-mon[32556]: pgmap v992: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T16:53:01.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:01 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1370320202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:53:01.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:01 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/1625473870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:53:01.763 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:01 smithi173 ceph-mon[32556]: from='client.? 172.21.15.98:0/3152352641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:53:01.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:01 smithi203 ceph-mon[33115]: pgmap v992: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 10 op/s 2024-04-25T16:53:01.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:01 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1370320202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:53:01.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:01 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/1625473870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-25T16:53:01.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:01 smithi203 ceph-mon[33115]: from='client.? 172.21.15.98:0/3152352641' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:53:03.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:03 smithi098 ceph-mon[29162]: pgmap v993: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:53:03.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:03 smithi173 ceph-mon[32556]: pgmap v993: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:53:03.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:03 smithi203 ceph-mon[33115]: pgmap v993: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:53:05.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:05 smithi098 ceph-mon[29162]: pgmap v994: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-25T16:53:05.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:05 smithi173 ceph-mon[32556]: pgmap v994: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-25T16:53:05.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:05 smithi203 ceph-mon[33115]: pgmap v994: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-25T16:53:07.451 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T16:53:07.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:53:07.451 DEBUG:teuthology.orchestra.run.smithi098:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T16:53:07.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:53:07.486 DEBUG:teuthology.orchestra.run.smithi098:> ip netns list 2024-04-25T16:53:07.551 INFO:teuthology.orchestra.run.smithi098.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T16:53:07.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:53:07.551 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:53:07.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:53:07.621 DEBUG:teuthology.orchestra.run.smithi098:> sudo ip link delete ceph-brx 2024-04-25T16:53:07.712 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T16:53:07.730 INFO:tasks.cephadm:Teardown begin 2024-04-25T16:53:07.731 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/cephadm.py", line 2246, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: 2024-04-25T16:53:07.735 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:53:07.737 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:07 smithi098 ceph-mon[29162]: pgmap v995: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T16:53:07.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:07 smithi173 ceph-mon[32556]: pgmap v995: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T16:53:07.763 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:53:07.797 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:53:07.825 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:07 smithi203 ceph-mon[33115]: pgmap v995: 129 pgs: 129 active+clean; 1.7 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-04-25T16:53:07.831 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T16:53:07.831 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T16:53:07.854 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T16:53:07.872 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T16:53:07.890 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T16:53:07.890 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T16:53:07.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.a 2024-04-25T16:53:08.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:07 smithi098 systemd[1]: Stopping Ceph mon.a for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:08.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a[29157]: 2024-04-25T16:53:08.222+0000 7f613981c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:08.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a[29157]: 2024-04-25T16:53:08.222+0000 7f613981c700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T16:53:08.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:08 smithi098 podman[127032]: 2024-04-25 16:53:08.497122406 +0000 UTC m=+0.395849650 container died 14b5f450ea62fdb680835601015e2c12ecb96813a73cae46fe511f2f48e4e3bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-25T16:53:09.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:08 smithi098 podman[127032]: 2024-04-25 16:53:08.856578395 +0000 UTC m=+0.755305688 container cleanup 14b5f450ea62fdb680835601015e2c12ecb96813a73cae46fe511f2f48e4e3bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:53:09.171 INFO:journalctl@ceph.mon.a.smithi098.stdout:Apr 25 16:53:08 smithi098 bash[127032]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-a 2024-04-25T16:53:09.360 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T16:53:09.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:09.420 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T16:53:09.420 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T16:53:09.420 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.b 2024-04-25T16:53:09.762 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:09 smithi173 systemd[1]: Stopping Ceph mon.b for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:10.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-b[32552]: 2024-04-25T16:53:09.807+0000 7f59ee4ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:10.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-b[32552]: 2024-04-25T16:53:09.807+0000 7f59ee4ab700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T16:53:10.187 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:09 smithi173 podman[70406]: 2024-04-25 16:53:09.89018084 +0000 UTC m=+0.189957907 container died 8af9d65e7b935c6eb496e1d3e29c15322b86868dd1ea6e54ec8f8a23e95d6997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T16:53:10.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:10 smithi173 podman[70406]: 2024-04-25 16:53:10.29854957 +0000 UTC m=+0.598326642 container cleanup 8af9d65e7b935c6eb496e1d3e29c15322b86868dd1ea6e54ec8f8a23e95d6997 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-b, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-25T16:53:10.512 INFO:journalctl@ceph.mon.b.smithi173.stdout:Apr 25 16:53:10 smithi173 bash[70406]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-b 2024-04-25T16:53:10.730 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T16:53:10.777 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:10.778 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T16:53:10.778 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T16:53:10.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.c 2024-04-25T16:53:11.138 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:10 smithi203 systemd[1]: Stopping Ceph mon.c for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:11.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-c[33100]: 2024-04-25T16:53:11.201+0000 7f193ee67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:11.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-c[33100]: 2024-04-25T16:53:11.201+0000 7f193ee67700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T16:53:11.436 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:11 smithi203 podman[72542]: 2024-04-25 16:53:11.334170428 +0000 UTC m=+0.363994077 container died bb42c2be863d93bbe7431674f330a01df9b20038e114653a45fa081c108f7979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T16:53:11.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:11 smithi203 podman[72542]: 2024-04-25 16:53:11.719251727 +0000 UTC m=+0.749075372 container cleanup bb42c2be863d93bbe7431674f330a01df9b20038e114653a45fa081c108f7979 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-25T16:53:11.936 INFO:journalctl@ceph.mon.c.smithi203.stdout:Apr 25 16:53:11 smithi203 bash[72542]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mon-c 2024-04-25T16:53:12.101 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T16:53:12.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:12.146 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T16:53:12.146 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T16:53:12.147 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.x 2024-04-25T16:53:12.486 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:53:12 smithi098 systemd[1]: Stopping Ceph mgr.x for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:12.750 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:53:12 smithi098 podman[127504]: 2024-04-25 16:53:12.489996992 +0000 UTC m=+0.144216707 container died 4f5165254f85e8fd31f3f8c3d5df2a681712521f098331520f07cef392da5c26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:53:13.250 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:53:12 smithi098 podman[127504]: 2024-04-25 16:53:12.885500554 +0000 UTC m=+0.539720265 container cleanup 4f5165254f85e8fd31f3f8c3d5df2a681712521f098331520f07cef392da5c26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424) 2024-04-25T16:53:13.250 INFO:journalctl@ceph.mgr.x.smithi098.stdout:Apr 25 16:53:12 smithi098 bash[127504]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-x 2024-04-25T16:53:13.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T16:53:13.436 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:13.437 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T16:53:13.437 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T16:53:13.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.y 2024-04-25T16:53:13.762 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:53:13 smithi173 systemd[1]: Stopping Ceph mgr.y for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:14.061 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:53:13 smithi173 podman[70641]: 2024-04-25 16:53:13.800098106 +0000 UTC m=+0.163562847 container died 4390a4506f4a0d308eb3e2cfd2087e42b25924eeb60843a33501e11f3fc378a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T16:53:14.437 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:53:14 smithi173 podman[70641]: 2024-04-25 16:53:14.180626624 +0000 UTC m=+0.544091358 container cleanup 4390a4506f4a0d308eb3e2cfd2087e42b25924eeb60843a33501e11f3fc378a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T16:53:14.437 INFO:journalctl@ceph.mgr.y.smithi173.stdout:Apr 25 16:53:14 smithi173 bash[70641]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-y 2024-04-25T16:53:14.597 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T16:53:14.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:14.631 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T16:53:14.632 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T16:53:14.632 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.z 2024-04-25T16:53:14.936 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:53:14 smithi203 systemd[1]: Stopping Ceph mgr.z for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:15.539 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:53:15 smithi203 podman[72664]: 2024-04-25 16:53:15.26240772 +0000 UTC m=+0.437968445 container died 8c8b25a83d7d5225493c53ec5fbb34359a86c315b3e4bc02ad929d348263de9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T16:53:15.936 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:53:15 smithi203 podman[72664]: 2024-04-25 16:53:15.668006606 +0000 UTC m=+0.843567344 container cleanup 8c8b25a83d7d5225493c53ec5fbb34359a86c315b3e4bc02ad929d348263de9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T16:53:15.936 INFO:journalctl@ceph.mgr.z.smithi203.stdout:Apr 25 16:53:15 smithi203 bash[72664]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mgr-z 2024-04-25T16:53:16.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T16:53:16.117 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:16.118 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T16:53:16.118 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T16:53:16.118 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.0 2024-04-25T16:53:16.421 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:16 smithi098 systemd[1]: Stopping Ceph osd.0 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:16.750 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0[43971]: 2024-04-25T16:53:16.423+0000 7f7e22fa4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:16.750 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0[43971]: 2024-04-25T16:53:16.423+0000 7f7e22fa4700 -1 osd.0 106 *** Got signal Terminated *** 2024-04-25T16:53:16.750 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0[43971]: 2024-04-25T16:53:16.423+0000 7f7e22fa4700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:53:22.106 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:21 smithi098 podman[127627]: 2024-04-25 16:53:21.821032341 +0000 UTC m=+5.514400118 container died 0e442e05dd3f413c7f0a23668c715704f9562b69bf188bdf833762785afabc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:53:22.465 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:22 smithi098 podman[127627]: 2024-04-25 16:53:22.207986443 +0000 UTC m=+5.901354221 container cleanup 0e442e05dd3f413c7f0a23668c715704f9562b69bf188bdf833762785afabc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:53:22.465 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:22 smithi098 bash[127627]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0 2024-04-25T16:53:22.750 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:22 smithi098 podman[127646]: 2024-04-25 16:53:22.468195793 +0000 UTC m=+0.639257682 container remove 0e442e05dd3f413c7f0a23668c715704f9562b69bf188bdf833762785afabc83 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-25T16:53:23.136 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:22 smithi098 podman[127744]: 2024-04-25 16:53:22.733338233 +0000 UTC m=+0.018212906 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:53:23.136 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:22 smithi098 podman[127744]: 2024-04-25 16:53:22.894683251 +0000 UTC m=+0.179557910 container create a0c8a9e216f55fb01a1b790bc2c80a0f9711b30e1d69d5a806bca0fcf931664e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T16:53:23.500 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:23 smithi098 podman[127744]: 2024-04-25 16:53:23.139269995 +0000 UTC m=+0.424144652 container init a0c8a9e216f55fb01a1b790bc2c80a0f9711b30e1d69d5a806bca0fcf931664e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:53:23.501 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:23 smithi098 podman[127744]: 2024-04-25 16:53:23.142568369 +0000 UTC m=+0.427443036 container start a0c8a9e216f55fb01a1b790bc2c80a0f9711b30e1d69d5a806bca0fcf931664e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T16:53:23.501 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:23 smithi098 podman[127744]: 2024-04-25 16:53:23.256015108 +0000 UTC m=+0.540889769 container attach a0c8a9e216f55fb01a1b790bc2c80a0f9711b30e1d69d5a806bca0fcf931664e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True) 2024-04-25T16:53:24.001 INFO:journalctl@ceph.osd.0.smithi098.stdout:Apr 25 16:53:23 smithi098 podman[127744]: 2024-04-25 16:53:23.684547506 +0000 UTC m=+0.969422172 container died a0c8a9e216f55fb01a1b790bc2c80a0f9711b30e1d69d5a806bca0fcf931664e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:53:24.357 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T16:53:24.395 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:24.395 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T16:53:24.395 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T16:53:24.395 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.1 2024-04-25T16:53:24.682 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:24 smithi173 systemd[1]: Stopping Ceph osd.1 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:25.012 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1[37167]: 2024-04-25T16:53:24.684+0000 7f5df81cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:25.012 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1[37167]: 2024-04-25T16:53:24.684+0000 7f5df81cd700 -1 osd.1 106 *** Got signal Terminated *** 2024-04-25T16:53:25.012 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1[37167]: 2024-04-25T16:53:24.684+0000 7f5df81cd700 -1 osd.1 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:53:30.407 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:30 smithi173 podman[70765]: 2024-04-25 16:53:30.101820175 +0000 UTC m=+5.515481414 container died 9df6fa099810604eaf35d5cf8c4cfb84fc5c1c14a24107a624ea885b4222da2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T16:53:30.667 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:30 smithi173 podman[70765]: 2024-04-25 16:53:30.410325875 +0000 UTC m=+5.823987116 container cleanup 9df6fa099810604eaf35d5cf8c4cfb84fc5c1c14a24107a624ea885b4222da2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:53:30.668 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:30 smithi173 bash[70765]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1 2024-04-25T16:53:31.012 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:30 smithi173 podman[70782]: 2024-04-25 16:53:30.670141689 +0000 UTC m=+0.569202185 container remove 9df6fa099810604eaf35d5cf8c4cfb84fc5c1c14a24107a624ea885b4222da2f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:53:31.354 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:31 smithi173 podman[70876]: 2024-04-25 16:53:30.923999332 +0000 UTC m=+0.018003873 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:53:31.354 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:31 smithi173 podman[70876]: 2024-04-25 16:53:31.046215927 +0000 UTC m=+0.140220435 container create d7cad20fbf6f8c24549569937a2c6d5011da6248effe045cdca4684db17a845e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T16:53:31.354 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:31 smithi173 podman[70876]: 2024-04-25 16:53:31.265621087 +0000 UTC m=+0.359625602 container init d7cad20fbf6f8c24549569937a2c6d5011da6248effe045cdca4684db17a845e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-25T16:53:31.354 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:31 smithi173 podman[70876]: 2024-04-25 16:53:31.268752882 +0000 UTC m=+0.362757392 container start d7cad20fbf6f8c24549569937a2c6d5011da6248effe045cdca4684db17a845e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:53:31.355 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:31 smithi173 podman[70876]: 2024-04-25 16:53:31.357410646 +0000 UTC m=+0.451415155 container attach d7cad20fbf6f8c24549569937a2c6d5011da6248effe045cdca4684db17a845e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:53:31.762 INFO:journalctl@ceph.osd.1.smithi173.stdout:Apr 25 16:53:31 smithi173 podman[70876]: 2024-04-25 16:53:31.561327959 +0000 UTC m=+0.655332489 container died d7cad20fbf6f8c24549569937a2c6d5011da6248effe045cdca4684db17a845e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T16:53:32.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T16:53:32.225 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:32.226 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T16:53:32.226 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T16:53:32.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.2 2024-04-25T16:53:32.545 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:32 smithi203 systemd[1]: Stopping Ceph osd.2 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:32.805 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2[38294]: 2024-04-25T16:53:32.546+0000 7ff342884700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:32.806 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2[38294]: 2024-04-25T16:53:32.546+0000 7ff342884700 -1 osd.2 106 *** Got signal Terminated *** 2024-04-25T16:53:32.806 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2[38294]: 2024-04-25T16:53:32.546+0000 7ff342884700 -1 osd.2 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:53:38.306 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:38 smithi203 podman[72788]: 2024-04-25 16:53:38.001077293 +0000 UTC m=+5.576647017 container died d3661c45945a69e06399336d79b900c78ae9befae37d16eda738ba7b45278dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-25T16:53:38.664 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:38 smithi203 podman[72788]: 2024-04-25 16:53:38.407385165 +0000 UTC m=+5.982954917 container cleanup d3661c45945a69e06399336d79b900c78ae9befae37d16eda738ba7b45278dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:53:38.664 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:38 smithi203 bash[72788]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2 2024-04-25T16:53:38.936 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:38 smithi203 podman[72819]: 2024-04-25 16:53:38.667286317 +0000 UTC m=+0.661534815 container remove d3661c45945a69e06399336d79b900c78ae9befae37d16eda738ba7b45278dd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-25T16:53:39.302 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:38 smithi203 podman[72908]: 2024-04-25 16:53:38.885534868 +0000 UTC m=+0.018315705 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:53:39.302 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:38 smithi203 podman[72908]: 2024-04-25 16:53:38.993620246 +0000 UTC m=+0.126401054 container create b07a76b054286c6f426d827e17e4caa00c2fe516435741b0c663bde169d7d4e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T16:53:39.302 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:39 smithi203 podman[72908]: 2024-04-25 16:53:39.213342548 +0000 UTC m=+0.346123358 container init b07a76b054286c6f426d827e17e4caa00c2fe516435741b0c663bde169d7d4e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:53:39.302 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:39 smithi203 podman[72908]: 2024-04-25 16:53:39.216552187 +0000 UTC m=+0.349332997 container start b07a76b054286c6f426d827e17e4caa00c2fe516435741b0c663bde169d7d4e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T16:53:39.302 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:39 smithi203 podman[72908]: 2024-04-25 16:53:39.305106917 +0000 UTC m=+0.437887742 container attach b07a76b054286c6f426d827e17e4caa00c2fe516435741b0c663bde169d7d4e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:53:39.931 INFO:journalctl@ceph.osd.2.smithi203.stdout:Apr 25 16:53:39 smithi203 podman[72908]: 2024-04-25 16:53:39.574320454 +0000 UTC m=+0.707101280 container died b07a76b054286c6f426d827e17e4caa00c2fe516435741b0c663bde169d7d4e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T16:53:40.234 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T16:53:40.275 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:40.276 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T16:53:40.276 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T16:53:40.276 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.3 2024-04-25T16:53:40.586 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:40 smithi098 systemd[1]: Stopping Ceph osd.3 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:40.847 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:40 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3[50507]: 2024-04-25T16:53:40.588+0000 7f2fbb212700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:40.848 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:40 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3[50507]: 2024-04-25T16:53:40.588+0000 7f2fbb212700 -1 osd.3 106 *** Got signal Terminated *** 2024-04-25T16:53:40.848 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:40 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3[50507]: 2024-04-25T16:53:40.588+0000 7f2fbb212700 -1 osd.3 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:53:43.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:42 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:42.919+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:44.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:43.898+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:44.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:43.782+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:45.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:44.810+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:45.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:44.918+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:45.481 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:45 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:45.060+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:45.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:45 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:45.483+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:46.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:45.814+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:46.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:45.890+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:46.371 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:46.108+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:46.371 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:46 smithi098 podman[127849]: 2024-04-25 16:53:46.058890384 +0000 UTC m=+5.586280171 container died 793d66d8c16507ee8dd755bb9e9b73ea2b6d95acd6a8727c62de34ff87fc38a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:53:46.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:46.339+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:46.729 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:46 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:46.490+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:46.730 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:46 smithi098 podman[127849]: 2024-04-25 16:53:46.481017554 +0000 UTC m=+6.008407351 container cleanup 793d66d8c16507ee8dd755bb9e9b73ea2b6d95acd6a8727c62de34ff87fc38a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T16:53:46.730 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:46 smithi098 bash[127849]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3 2024-04-25T16:53:47.000 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:46 smithi098 podman[127878]: 2024-04-25 16:53:46.732748174 +0000 UTC m=+0.674954702 container remove 793d66d8c16507ee8dd755bb9e9b73ea2b6d95acd6a8727c62de34ff87fc38a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-25T16:53:47.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:46.785+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:47.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:46.911+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:47.408 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:47 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:47.069+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:47.408 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:47 smithi098 podman[127972]: 2024-04-25 16:53:47.049178087 +0000 UTC m=+0.018298480 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:53:47.408 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:47 smithi098 podman[127972]: 2024-04-25 16:53:47.167256514 +0000 UTC m=+0.136376894 container create 8c5ab32b4bad19286aee66bd6f02a150b7096bcec6b9f27df9ab6108cae743fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T16:53:47.408 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:47 smithi098 podman[127972]: 2024-04-25 16:53:47.411478026 +0000 UTC m=+0.380598406 container init 8c5ab32b4bad19286aee66bd6f02a150b7096bcec6b9f27df9ab6108cae743fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T16:53:47.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:47 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:47.052+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:47.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:47.321+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:47.706 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:47 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:47.516+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:47.706 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:47 smithi098 podman[127972]: 2024-04-25 16:53:47.414599737 +0000 UTC m=+0.383720126 container start 8c5ab32b4bad19286aee66bd6f02a150b7096bcec6b9f27df9ab6108cae743fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T16:53:47.706 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:47 smithi098 podman[127972]: 2024-04-25 16:53:47.503419421 +0000 UTC m=+0.472539815 container attach 8c5ab32b4bad19286aee66bd6f02a150b7096bcec6b9f27df9ab6108cae743fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T16:53:48.000 INFO:journalctl@ceph.osd.3.smithi098.stdout:Apr 25 16:53:47 smithi098 podman[127972]: 2024-04-25 16:53:47.70975176 +0000 UTC m=+0.678872138 container died 8c5ab32b4bad19286aee66bd6f02a150b7096bcec6b9f27df9ab6108cae743fa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T16:53:48.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:47.866+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:48.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:47.804+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:48.396 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:48 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:48.066+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:48.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T16:53:48.471 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:48.471 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T16:53:48.472 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T16:53:48.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.4 2024-04-25T16:53:48.499 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:48 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:48.015+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:48.685 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:48.352+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:48.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:48 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:48.536+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:48.762 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:48 smithi173 systemd[1]: Stopping Ceph osd.4 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:49.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:48.887+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:49.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:48.787+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:49.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:48 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:48.967+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:49.262 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:48 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4[44106]: 2024-04-25T16:53:48.765+0000 7f5038579700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:49.263 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:48 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4[44106]: 2024-04-25T16:53:48.765+0000 7f5038579700 -1 osd.4 106 *** Got signal Terminated *** 2024-04-25T16:53:49.263 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:48 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4[44106]: 2024-04-25T16:53:48.765+0000 7f5038579700 -1 osd.4 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:53:49.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:49 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:49.100+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:49.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:49.362+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:50.001 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:49 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:49.533+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:50.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:49.798+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:50.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:49.903+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:50.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:49 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:49.942+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:50.484 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:50 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:50.090+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:50.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:50 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:50.342+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:50.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:50 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:50.487+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:51.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:50 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:50.941+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:51.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:50 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:50.754+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:51.232 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:50 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:50.897+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:51.446 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:51 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:51.137+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:51.512 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4[44106]: 2024-04-25T16:53:51.234+0000 7f5030b27700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.146032+0000 front 2024-04-25T16:53:28.145996+0000 (oldest deadline 2024-04-25T16:53:50.446166+0000) 2024-04-25T16:53:51.685 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:51.326+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:51.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:51 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:51.449+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:51.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:51 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:51.449+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:52.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:51.797+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:52.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:51.961+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:52.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:51 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:51.961+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:53:52.221 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:51 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:51.885+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:52.443 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:52 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:52.098+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:52.443 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:52 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:52.098+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:53:52.512 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:52 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4[44106]: 2024-04-25T16:53:52.223+0000 7f5030b27700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.146032+0000 front 2024-04-25T16:53:28.145996+0000 (oldest deadline 2024-04-25T16:53:50.446166+0000) 2024-04-25T16:53:52.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:52 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:52.336+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:52.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:52 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:52.446+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:52.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:52 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:52.446+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:53.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:52 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:52.787+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:53.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:52 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:52.914+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:53.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:52 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:52.914+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:53:53.259 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:52 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:52.913+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:53.259 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:52 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:52.913+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:53:53.471 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:53 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:53.121+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:53.472 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:53 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:53.121+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:53:53.512 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:53 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4[44106]: 2024-04-25T16:53:53.261+0000 7f5030b27700 -1 osd.4 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.146032+0000 front 2024-04-25T16:53:28.145996+0000 (oldest deadline 2024-04-25T16:53:50.446166+0000) 2024-04-25T16:53:53.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:53 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:53.386+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:53.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:53 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:53.474+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:53.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:53 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:53.474+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:54.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:53 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:53.836+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:54.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:53 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:53.836+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:53:54.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:53 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:53.953+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:54.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:53 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:53.953+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:53:54.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:53:53 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:53:53.883+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:53:54.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:53 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:53.903+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:54.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:53 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:53.903+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:53:54.479 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:54 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:54.126+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:54.479 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:54 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:54.126+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:53:54.546 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:54 smithi173 podman[70993]: 2024-04-25 16:53:54.276016705 +0000 UTC m=+5.605853933 container died 60155c5098f36127c70ceb2dddf6c1903d4a6373c154267842bd082f3e20f419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:53:54.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:54 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:54.377+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:54.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:54 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:54.377+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.653622+0000 front 2024-04-25T16:53:28.653663+0000 (oldest deadline 2024-04-25T16:53:53.953579+0000) 2024-04-25T16:53:54.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:54 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:54.481+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:54.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:54 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:54.481+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:54.915 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:54 smithi173 podman[70993]: 2024-04-25 16:53:54.648625558 +0000 UTC m=+5.978462840 container cleanup 60155c5098f36127c70ceb2dddf6c1903d4a6373c154267842bd082f3e20f419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T16:53:54.915 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:54 smithi173 bash[70993]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4 2024-04-25T16:53:55.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:54 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:54.862+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:55.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:54 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:54.862+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:53:55.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:54 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:54.977+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:55.187 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:54 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:54.977+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:53:55.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:53:54 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:53:54.917+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:53:55.263 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:54 smithi173 podman[71009]: 2024-04-25 16:53:54.966789773 +0000 UTC m=+0.693373763 container remove 60155c5098f36127c70ceb2dddf6c1903d4a6373c154267842bd082f3e20f419 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T16:53:55.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:54 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:54.953+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:55.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:54 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:54.953+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:53:55.444 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:55 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:55.148+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:55.445 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:55 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:55.148+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:53:55.625 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:55 smithi173 podman[71101]: 2024-04-25 16:53:55.212786788 +0000 UTC m=+0.017859340 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:53:55.625 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:55 smithi173 podman[71101]: 2024-04-25 16:53:55.334438377 +0000 UTC m=+0.139510912 container create 408ecc6ce83554985e90ff4f73b6c69d7307217b3334679a3299f47f6926bb0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-25T16:53:55.625 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:55 smithi173 podman[71101]: 2024-04-25 16:53:55.545331278 +0000 UTC m=+0.350403816 container init 408ecc6ce83554985e90ff4f73b6c69d7307217b3334679a3299f47f6926bb0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-25T16:53:55.626 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:55 smithi173 podman[71101]: 2024-04-25 16:53:55.548780066 +0000 UTC m=+0.353852601 container start 408ecc6ce83554985e90ff4f73b6c69d7307217b3334679a3299f47f6926bb0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:53:55.626 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:55 smithi173 podman[71101]: 2024-04-25 16:53:55.628477228 +0000 UTC m=+0.433549766 container attach 408ecc6ce83554985e90ff4f73b6c69d7307217b3334679a3299f47f6926bb0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:53:55.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:55.376+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:55.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:55.376+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.653622+0000 front 2024-04-25T16:53:28.653663+0000 (oldest deadline 2024-04-25T16:53:53.953579+0000) 2024-04-25T16:53:55.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:55 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:55.447+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:55.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:55 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:55.447+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:55.908 INFO:journalctl@ceph.osd.4.smithi173.stdout:Apr 25 16:53:55 smithi173 podman[71101]: 2024-04-25 16:53:55.840813947 +0000 UTC m=+0.645886481 container died 408ecc6ce83554985e90ff4f73b6c69d7307217b3334679a3299f47f6926bb0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T16:53:55.908 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:53:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:53:55.909+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:53:56.185 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:56.018+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:56.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:56.018+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:53:56.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:55.886+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:56.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:55 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:55.886+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:53:56.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:55.950+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:56.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:55 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:55.950+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:53:56.427 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:56 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:56.110+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:56.428 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:56 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:56.110+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:53:56.498 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T16:53:56.576 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:53:56.576 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T16:53:56.576 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T16:53:56.576 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.5 2024-04-25T16:53:56.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:56.392+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:56.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:56.392+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.653622+0000 front 2024-04-25T16:53:28.653663+0000 (oldest deadline 2024-04-25T16:53:53.953579+0000) 2024-04-25T16:53:56.687 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:56 smithi203 systemd[1]: Stopping Ceph osd.5 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:53:56.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:56 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:56.430+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:56.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:56 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:56.430+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:57.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:57 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:57.053+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:57.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:57 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:57.053+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:53:57.187 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:56.904+0000 7fa4442a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:53:57.187 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:56.904+0000 7fa4442a0700 -1 osd.5 106 *** Got signal Terminated *** 2024-04-25T16:53:57.187 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:56.904+0000 7fa4442a0700 -1 osd.5 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:53:57.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:56.873+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:57.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:56 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:56.873+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:53:57.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:53:56 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:53:56.866+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:53:57.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:56 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:56.917+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:57.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:56 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:56.917+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:53:57.422 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:57 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:57.103+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:57.422 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:57 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:57.103+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:53:57.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:57 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:57.387+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:57.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:57 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:57.387+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.653622+0000 front 2024-04-25T16:53:28.653663+0000 (oldest deadline 2024-04-25T16:53:53.953579+0000) 2024-04-25T16:53:57.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:57 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:57.425+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:57.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:57 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:57.425+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:58.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:58 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:58.020+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:58.186 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:58 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:58.020+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:53:58.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:57 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:57.904+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:58.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:57 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:57.904+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:53:58.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:53:57 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:53:57.905+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:53:58.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:57 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:57.900+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:58.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:57 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:57.900+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:53:58.426 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:58 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:58.136+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:58.426 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:58 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:58.136+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:53:58.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:58 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:58.370+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:58.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:58 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:58.370+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.653622+0000 front 2024-04-25T16:53:28.653663+0000 (oldest deadline 2024-04-25T16:53:53.953579+0000) 2024-04-25T16:53:58.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:58 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:58.429+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:58.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:58 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:58.429+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:59.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:58 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:58.893+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:53:59.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:58 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:58.893+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:53:59.187 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:59 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:59.068+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:53:59.187 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:53:59 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:53:59.068+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:53:59.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:53:58 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:53:58.885+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:53:59.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:58 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:58.919+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:53:59.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:58 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:58.919+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:53:59.420 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:59 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:59.088+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:53:59.420 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:53:59 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:53:59.088+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:53:59.685 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:59 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:59.380+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:53:59.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:53:59 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:53:59.380+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.653622+0000 front 2024-04-25T16:53:28.653663+0000 (oldest deadline 2024-04-25T16:53:53.953579+0000) 2024-04-25T16:53:59.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:59 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:59.423+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:53:59.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:59 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:59.423+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:53:59.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:53:59 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:53:59.423+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:00.112 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:59 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:59.863+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:00.112 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:59 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:59.863+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:00.112 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:53:59 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:53:59.863+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:00.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:53:59 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:53:59.926+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:00.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:59 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:59.918+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:00.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:53:59 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:53:59.918+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:00.397 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:00.115+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:00.397 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:00.115+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:00.398 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:00.115+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:00.436 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:00 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:00.041+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:00.436 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:00 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:00.041+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:00.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:54:00.400+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:54:00.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:54:00.400+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.653622+0000 front 2024-04-25T16:53:28.653663+0000 (oldest deadline 2024-04-25T16:53:53.953579+0000) 2024-04-25T16:54:00.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:54:00.400+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:34.454493+0000 front 2024-04-25T16:53:34.454520+0000 (oldest deadline 2024-04-25T16:54:00.255415+0000) 2024-04-25T16:54:00.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:00 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:00.438+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:00.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:00 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:00.438+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:00.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:00 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:00.438+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:01.148 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:00.884+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:01.148 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:00.884+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:01.148 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:00 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:00.884+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:01.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:01.000+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:01.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:01.000+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:01.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:00 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:00.969+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:01.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:00 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:00.955+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:01.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:00 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:00.955+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:01.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:00 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:00.955+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:01.406 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:01.150+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:01.406 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:01.150+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:01.406 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:01.150+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:01.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:54:01.408+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.852117+0000 front 2024-04-25T16:53:19.852239+0000 (oldest deadline 2024-04-25T16:53:45.752191+0000) 2024-04-25T16:54:01.686 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:54:01.408+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.653622+0000 front 2024-04-25T16:53:28.653663+0000 (oldest deadline 2024-04-25T16:53:53.953579+0000) 2024-04-25T16:54:01.687 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5[45239]: 2024-04-25T16:54:01.408+0000 7fa43c84e700 -1 osd.5 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:34.454493+0000 front 2024-04-25T16:53:34.454520+0000 (oldest deadline 2024-04-25T16:54:00.255415+0000) 2024-04-25T16:54:01.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:01.399+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:01.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:01.399+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:01.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:01.399+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:02.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:01.904+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:02.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:01.904+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:02.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:01 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:01.904+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:02.187 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:02 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:02.133+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:02.187 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:02 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:02.133+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:02.187 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:02 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:02.133+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:02.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:01.978+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:02.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:01.978+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:02.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:01 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:01.978+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:02.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:02 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:02.006+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:02.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:02 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:02.006+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:02.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:01 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:01.988+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:02.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:01 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:01.988+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:02.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:01 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:01.988+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:02.557 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:02 smithi203 podman[73013]: 2024-04-25 16:54:02.311663365 +0000 UTC m=+5.534953609 container stop b4124898574497ed8fcba54756a153953560728fe254882a942f99f321d450af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:54:02.558 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:02 smithi203 podman[73013]: 2024-04-25 16:54:02.341951864 +0000 UTC m=+5.565242153 container died b4124898574497ed8fcba54756a153953560728fe254882a942f99f321d450af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:54:02.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:02 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:02.392+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:02.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:02 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:02.392+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:02.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:02 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:02.392+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:02.816 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:02 smithi203 podman[73013]: 2024-04-25 16:54:02.714417469 +0000 UTC m=+5.937707712 container cleanup b4124898574497ed8fcba54756a153953560728fe254882a942f99f321d450af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T16:54:02.816 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:02 smithi203 bash[73013]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5 2024-04-25T16:54:03.177 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:02 smithi203 podman[73030]: 2024-04-25 16:54:02.916079746 +0000 UTC m=+0.598059767 container remove b4124898574497ed8fcba54756a153953560728fe254882a942f99f321d450af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-25T16:54:03.178 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:02 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:02.900+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:03.178 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:02 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:02.900+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:03.178 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:02 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:02.900+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:03.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:02 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:02.941+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:03.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:02 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:02.941+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:03.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:02 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:02.941+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:03.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:03.006+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:03.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:03.006+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:03.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:02 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:02.981+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:03.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:02 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:02.981+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:03.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:02 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:02.981+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:03.436 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:03 smithi203 podman[73128]: 2024-04-25 16:54:03.270449105 +0000 UTC m=+0.018670223 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:54:03.436 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:03 smithi203 podman[73128]: 2024-04-25 16:54:03.392359121 +0000 UTC m=+0.140580206 container create 75c84c150219a549f2e2ad7c94d691952a4299de1e97184a8eef84fc23939f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5) 2024-04-25T16:54:03.436 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:03.180+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:03.436 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:03.180+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:03.437 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:03.180+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:03.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:03.423+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:03.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:03.423+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:03.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:03.423+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:03.822 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:03 smithi203 podman[73128]: 2024-04-25 16:54:03.637163868 +0000 UTC m=+0.385384953 container init 75c84c150219a549f2e2ad7c94d691952a4299de1e97184a8eef84fc23939f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T16:54:03.822 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:03 smithi203 podman[73128]: 2024-04-25 16:54:03.640440497 +0000 UTC m=+0.388661592 container start 75c84c150219a549f2e2ad7c94d691952a4299de1e97184a8eef84fc23939f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD) 2024-04-25T16:54:03.823 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:03 smithi203 podman[73128]: 2024-04-25 16:54:03.712263179 +0000 UTC m=+0.460484273 container attach 75c84c150219a549f2e2ad7c94d691952a4299de1e97184a8eef84fc23939f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-25T16:54:04.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:03.904+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:04.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:03.904+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:04.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:03 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:03.904+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:04.186 INFO:journalctl@ceph.osd.5.smithi203.stdout:Apr 25 16:54:03 smithi203 podman[73128]: 2024-04-25 16:54:03.934977906 +0000 UTC m=+0.683199001 container died 75c84c150219a549f2e2ad7c94d691952a4299de1e97184a8eef84fc23939f7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True) 2024-04-25T16:54:04.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:03.983+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:04.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:03.983+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:04.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:03.983+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:04.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:03.987+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:04.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:03.987+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:04.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:03.987+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:04.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:03.990+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:04.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:03 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:03.990+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:04.597 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:04 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:04.213+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:04.597 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:04 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:04.213+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:04.597 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:04 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:04.213+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:04.628 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T16:54:04.666 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:04.666 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T16:54:04.667 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T16:54:04.667 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.6 2024-04-25T16:54:04.693 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:04.399+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:04.693 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:04.399+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:04.694 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:04.399+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:04.950 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:04 smithi098 systemd[1]: Stopping Ceph osd.6 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:05.185 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:04 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:04.862+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:05.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:04 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:04.862+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:05.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:04 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:04.862+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:05.250 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:04.952+0000 7f6f65a36700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:05.250 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:04.952+0000 7f6f65a36700 -1 osd.6 106 *** Got signal Terminated *** 2024-04-25T16:54:05.250 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:04.952+0000 7f6f65a36700 -1 osd.6 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:54:05.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:04.988+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:05.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:04.988+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:05.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:04 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:04.988+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:05.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:04 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:04.983+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:05.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:04 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:04.983+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:05.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:04 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:04.967+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:05.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:04 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:04.967+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:05.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:04 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:04.967+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:05.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:05 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:05.254+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:05.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:05 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:05.254+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:05.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:05 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:05.254+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:05.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:05 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:05.365+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:05.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:05 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:05.365+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:05.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:05 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:05.365+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:06.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:05 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:05.901+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:06.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:05 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:05.901+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:06.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:05 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:05.901+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:06.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:05.988+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:06.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:05.988+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:06.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:05.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:06.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:05.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:06.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:05 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:05.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:06.368 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:06.029+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:06.369 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:06.030+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:06.369 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:06.030+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:06.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:06 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:06.304+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:06.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:06 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:06.304+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:06.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:06 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:06.304+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:06.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:06.371+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:06.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:06.371+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:06.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:06 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:06.371+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:07.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:06 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:06.935+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:07.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:06 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:06.935+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:07.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:06 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:06.935+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:07.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:06 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:06.986+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:07.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:06 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:06.986+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:07.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:06 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:06.986+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:07.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:06 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:06.940+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:07.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:06 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:06.940+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:07.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:06 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:06.940+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:07.411 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:07.070+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:07.411 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:07.070+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:07.411 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:07.070+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:07.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:07 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:07.287+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:07.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:07 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:07.287+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:07.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:07 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:07.287+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:07.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:07.413+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:07.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:07.413+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:07.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:07.413+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:07.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:07 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:07.413+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.504446+0000 front 2024-04-25T16:53:45.504373+0000 (oldest deadline 2024-04-25T16:54:07.204528+0000) 2024-04-25T16:54:08.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:08.024+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:08.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:08.024+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:08.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:08.024+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:08.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:07 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:07.944+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:08.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:07 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:07.944+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:08.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:07 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:07.944+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:08.328 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:07 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:07.941+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:08.329 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:07 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:07.941+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:08.329 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:07 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:07.941+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:08.448 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:08.118+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:08.448 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:08.118+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:08.448 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:08.118+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:08.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:08 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:08.331+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:08.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:08 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:08.331+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:08.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:08 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:08.331+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:08.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:08.450+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:08.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:08.450+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:08.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:08.450+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:08.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:08 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:08.450+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.504446+0000 front 2024-04-25T16:53:45.504373+0000 (oldest deadline 2024-04-25T16:54:07.204528+0000) 2024-04-25T16:54:09.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:08 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:08.930+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:09.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:08 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:08.930+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:09.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:08 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:08.930+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:09.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:08 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:08.930+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:09.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:09.028+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:09.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:09.029+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:09.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:09.029+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:09.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:08.949+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:09.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:08.949+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:09.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:08.949+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:09.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:08.949+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:09.430 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:09 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:09.159+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:09.430 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:09 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:09.159+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:09.430 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:09 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:09.159+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:09.685 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:09 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:09.377+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:09.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:09 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:09.377+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:09.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:09 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:09.377+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:09.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:09 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:09.432+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:18.699916+0000 front 2024-04-25T16:53:18.700134+0000 (oldest deadline 2024-04-25T16:53:44.599873+0000) 2024-04-25T16:54:09.750 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:09 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:09.432+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:24.600398+0000 front 2024-04-25T16:53:24.600323+0000 (oldest deadline 2024-04-25T16:53:50.500360+0000) 2024-04-25T16:54:09.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:09 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:09.432+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.902128+0000 front 2024-04-25T16:53:36.902148+0000 (oldest deadline 2024-04-25T16:53:59.202033+0000) 2024-04-25T16:54:09.751 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:09 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6[57082]: 2024-04-25T16:54:09.432+0000 7f6f61fec700 -1 osd.6 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.504446+0000 front 2024-04-25T16:53:45.504373+0000 (oldest deadline 2024-04-25T16:54:07.204528+0000) 2024-04-25T16:54:10.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:10.042+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:10.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:10.042+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:10.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:10.042+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:10.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:09.931+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:10.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:09.931+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:10.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:09.931+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:10.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:09 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:09.931+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:10.346 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:09 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:09.957+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:10.346 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:09 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:09.957+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:10.347 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:09 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:09.957+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:10.347 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:09 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:09.957+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:10.500 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:10 smithi098 podman[128076]: 2024-04-25 16:54:10.30125015 +0000 UTC m=+5.439701944 container died 8cd85ba1e1a42d3de7a4b5425392d60f71cf14312f5eb944dfc0e7d38d0da14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:54:10.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:10 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:10.110+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:10.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:10 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:10.110+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:10.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:10 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:10.110+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:10.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:10 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:10.110+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:10.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:10 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:10.348+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:10.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:10 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:10.348+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:10.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:10 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:10.348+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:10.926 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:10 smithi098 podman[128076]: 2024-04-25 16:54:10.669720607 +0000 UTC m=+5.808172389 container cleanup 8cd85ba1e1a42d3de7a4b5425392d60f71cf14312f5eb944dfc0e7d38d0da14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T16:54:10.926 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:10 smithi098 bash[128076]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6 2024-04-25T16:54:11.185 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:10 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:10.918+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:11.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:10 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:10.918+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:11.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:10 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:10.918+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:11.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:10 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:10.918+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:11.250 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:10 smithi098 podman[128093]: 2024-04-25 16:54:10.92952359 +0000 UTC m=+0.626616315 container remove 8cd85ba1e1a42d3de7a4b5425392d60f71cf14312f5eb944dfc0e7d38d0da14e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-25T16:54:11.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:11 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:11.071+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:11.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:11 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:11.071+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:11.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:11 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:11.071+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:11.251 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:11 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:11.071+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:11.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:11 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:11.075+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:11.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:11 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:11.075+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:11.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:11 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:11.075+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:11.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:10.917+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:11.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:10.917+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:11.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:10.917+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:11.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:10 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:10.917+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:11.538 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:11 smithi098 podman[128183]: 2024-04-25 16:54:11.190467023 +0000 UTC m=+0.017664587 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:54:11.539 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:11 smithi098 podman[128183]: 2024-04-25 16:54:11.313698437 +0000 UTC m=+0.140895986 container create ef039e7b8b5d9e214e2006259a60d82d3d27475227ea9d69d179ae81d0df5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T16:54:11.539 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:11 smithi098 podman[128183]: 2024-04-25 16:54:11.541875791 +0000 UTC m=+0.369073342 container init ef039e7b8b5d9e214e2006259a60d82d3d27475227ea9d69d179ae81d0df5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T16:54:11.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:11.374+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:11.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:11.374+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:11.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:11.374+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:11.835 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:11 smithi098 podman[128183]: 2024-04-25 16:54:11.54525438 +0000 UTC m=+0.372451931 container start ef039e7b8b5d9e214e2006259a60d82d3d27475227ea9d69d179ae81d0df5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T16:54:11.836 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:11 smithi098 podman[128183]: 2024-04-25 16:54:11.641529217 +0000 UTC m=+0.468726766 container attach ef039e7b8b5d9e214e2006259a60d82d3d27475227ea9d69d179ae81d0df5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T16:54:12.132 INFO:journalctl@ceph.osd.6.smithi098.stdout:Apr 25 16:54:11 smithi098 podman[128183]: 2024-04-25 16:54:11.839190158 +0000 UTC m=+0.666387713 container died ef039e7b8b5d9e214e2006259a60d82d3d27475227ea9d69d179ae81d0df5ca1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T16:54:12.133 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:12 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:12.049+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:12.133 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:12 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:12.049+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:12.133 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:12 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:12.049+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:12.133 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:12 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:12.049+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:12.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:12.079+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:12.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:12.079+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:12.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:12.079+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:12.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:11 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:11.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:12.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:11 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:11.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:12.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:11 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:11.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:12.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:11 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:11.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:12.410 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:11.943+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:12.410 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:11.943+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:12.410 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:11.943+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:12.410 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:11 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:11.943+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:12.462 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T16:54:12.539 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:12.540 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T16:54:12.540 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T16:54:12.540 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.7 2024-04-25T16:54:12.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:12 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:12.412+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:12.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:12 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:12.412+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:12.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:12 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:12.412+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:12.857 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:12 smithi173 systemd[1]: Stopping Ceph osd.7 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:13.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:12.859+0000 7fd1299d2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:13.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:12.859+0000 7fd1299d2700 -1 osd.7 106 *** Got signal Terminated *** 2024-04-25T16:54:13.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:12.859+0000 7fd1299d2700 -1 osd.7 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:54:13.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:13.046+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:13.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:13.046+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:13.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:13.046+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:13.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:12.956+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:13.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:12.956+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:13.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:12.956+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:13.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:12 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:12.956+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:13.315 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:12 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:12.946+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:13.315 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:12 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:12.946+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:13.315 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:12 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:12.946+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:13.315 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:12 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:12.946+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:13.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:13 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:13.065+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:13.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:13 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:13.065+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:13.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:13 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:13.065+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:13.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:13 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:13.065+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:13.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:13 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:13.407+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:13.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:13 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:13.407+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:13.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:13 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:13.407+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:14.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:13 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:13.967+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:14.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:13 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:13.967+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:14.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:13 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:13.967+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:14.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:13 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:13.967+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:14.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:14.023+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:14.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:14.023+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:14.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:14.023+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:14.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:13.975+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:14.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:13.975+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:14.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:13.975+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:14.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:13 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:13.975+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:14.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:14 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:14.108+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:14.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:14 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:14.108+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:14.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:14 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:14.108+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:14.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:14 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:14.108+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:14.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:14.364+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:14.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:14.364+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:14.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:14.364+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:15.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:15 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:15.011+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:15.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:15 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:15.011+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:15.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:15 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:15.011+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:15.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:14.947+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:15.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:14.947+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:15.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:14.947+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:15.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:14 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:14.947+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:15.411 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:14.967+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:15.411 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:14.967+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:15.412 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:14.967+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:15.412 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:14 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:14.967+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:15.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:15.129+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:15.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:15.129+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:15.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:15.129+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:15.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:15 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:15.129+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:15.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:15 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:15.413+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:15.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:15 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:15.413+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:15.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:15 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:15.413+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:16.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:16 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:16.032+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:16.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:16 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:16.032+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:16.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:16 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:16.032+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:16.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:15 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:15.929+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:16.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:15 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:15.929+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:16.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:15 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:15.929+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:16.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:15 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:15.929+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:16.430 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:15 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:15.993+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:16.431 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:15 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:15.993+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:16.431 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:15 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:15.993+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:16.431 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:15 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:15.993+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:16.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:16.133+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:16.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:16.133+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:16.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:16.133+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:16.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:16 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:16.133+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:16.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:16.431+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:16.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:16.431+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:16.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:16.431+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:16.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:16 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:16.431+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:17.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:17.003+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.550179+0000 front 2024-04-25T16:53:29.550150+0000 (oldest deadline 2024-04-25T16:53:53.650387+0000) 2024-04-25T16:54:17.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:17.003+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:37.151522+0000 front 2024-04-25T16:53:37.151647+0000 (oldest deadline 2024-04-25T16:54:01.251375+0000) 2024-04-25T16:54:17.263 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7[50674]: 2024-04-25T16:54:17.003+0000 7fd125f88700 -1 osd.7 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:41.251949+0000 front 2024-04-25T16:53:41.251971+0000 (oldest deadline 2024-04-25T16:54:06.551891+0000) 2024-04-25T16:54:17.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:16 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:16.892+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:17.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:16 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:16.892+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:17.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:16 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:16.892+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:17.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:16 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:16.892+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:17.419 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:17 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.020+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:17.419 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:17 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.020+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:17.419 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:17 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.020+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:17.419 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:17 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.020+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:17.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:17 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:17.166+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:17.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:17 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:17.166+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:17.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:17 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:17.166+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:17.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:17 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:17.166+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:17.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:17 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:17.421+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:17.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:17 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:17.421+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:17.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:17 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:17.421+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:17.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:17 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:17.421+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:18.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:17.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:18.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:17.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:18.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:17.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:18.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:17.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:18.391 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.999+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:18.394 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.999+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:18.394 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.999+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:18.394 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.999+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:18.394 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:17.999+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.687863+0000 front 2024-04-25T16:53:52.687916+0000 (oldest deadline 2024-04-25T16:54:17.387948+0000) 2024-04-25T16:54:18.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:18.192+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:18.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:18.192+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:18.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:18.192+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:18.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:18.192+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:18.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:18 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:18.192+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:18.591 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:18 smithi173 podman[71219]: 2024-04-25 16:54:18.305375421 +0000 UTC m=+5.563333680 container died 8242f02e59a488927b04e0a711ab70efa802f7c35af07a4919a85ef8e8215bef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:54:18.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:18.392+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:18.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:18.392+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:18.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:18.392+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:18.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:18.392+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:18.908 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:18 smithi173 podman[71219]: 2024-04-25 16:54:18.693772425 +0000 UTC m=+5.951730678 container cleanup 8242f02e59a488927b04e0a711ab70efa802f7c35af07a4919a85ef8e8215bef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T16:54:18.908 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:18 smithi173 bash[71219]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7 2024-04-25T16:54:19.262 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:18 smithi173 podman[71236]: 2024-04-25 16:54:18.978818879 +0000 UTC m=+0.676002494 container remove 8242f02e59a488927b04e0a711ab70efa802f7c35af07a4919a85ef8e8215bef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:54:19.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:18 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:18.909+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:19.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:18 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:18.909+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:19.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:18 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:18.909+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:19.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:18 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:18.909+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:19.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:18 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:18.909+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:19.375 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:18.994+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:19.375 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:18.994+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:19.375 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:18.994+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:19.375 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:18.994+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:19.376 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:18 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:18.994+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.687863+0000 front 2024-04-25T16:53:52.687916+0000 (oldest deadline 2024-04-25T16:54:17.387948+0000) 2024-04-25T16:54:19.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:19 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:19.159+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:19.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:19 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:19.159+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:19.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:19 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:19.159+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:19.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:19 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:19.159+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:19.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:19 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:19.159+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:19.612 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:19 smithi173 podman[71326]: 2024-04-25 16:54:19.21511988 +0000 UTC m=+0.017874805 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:54:19.612 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:19 smithi173 podman[71326]: 2024-04-25 16:54:19.338255517 +0000 UTC m=+0.141010407 container create 11d08e96153c26e5b85f82aedd1a1c86d37a2e87b56f005c212084b1b1e13f15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:54:19.612 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:19 smithi173 podman[71326]: 2024-04-25 16:54:19.524044601 +0000 UTC m=+0.326799510 container init 11d08e96153c26e5b85f82aedd1a1c86d37a2e87b56f005c212084b1b1e13f15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T16:54:19.612 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:19 smithi173 podman[71326]: 2024-04-25 16:54:19.52756633 +0000 UTC m=+0.330321223 container start 11d08e96153c26e5b85f82aedd1a1c86d37a2e87b56f005c212084b1b1e13f15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T16:54:19.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:19.376+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:19.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:19.376+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:19.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:19.376+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:19.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:19.376+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:19.943 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:19 smithi173 podman[71326]: 2024-04-25 16:54:19.6153899 +0000 UTC m=+0.418144793 container attach 11d08e96153c26e5b85f82aedd1a1c86d37a2e87b56f005c212084b1b1e13f15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True) 2024-04-25T16:54:19.943 INFO:journalctl@ceph.osd.7.smithi173.stdout:Apr 25 16:54:19 smithi173 podman[71326]: 2024-04-25 16:54:19.824715091 +0000 UTC m=+0.627470006 container died 11d08e96153c26e5b85f82aedd1a1c86d37a2e87b56f005c212084b1b1e13f15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-25T16:54:20.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:19 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:19.946+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:20.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:19 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:19.946+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:20.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:19 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:19.946+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:20.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:19 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:19.946+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:20.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:19 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:19.946+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:20.406 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:19.959+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:20.407 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:19.959+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:20.407 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:19.959+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:20.407 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:19.959+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:20.407 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:19 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:19.959+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.687863+0000 front 2024-04-25T16:53:52.687916+0000 (oldest deadline 2024-04-25T16:54:17.387948+0000) 2024-04-25T16:54:20.497 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T16:54:20.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:20 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:20.186+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:20.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:20 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:20.186+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:20.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:20 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:20.186+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:20.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:20 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:20.186+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:20.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:20 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:20.186+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:20.535 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:20.536 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T16:54:20.536 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T16:54:20.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.8 2024-04-25T16:54:20.686 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 systemd[1]: Stopping Ceph osd.8 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:20.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:20.409+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:20.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:20.409+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:20.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:20.409+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:20.687 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:20.409+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:21.012 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:20 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:20.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:21.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:20 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:20.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:21.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:20 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:20.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:21.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:20 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:20.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:21.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:20 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:20.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:21.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:20.836+0000 7fea30e41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:21.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:20.836+0000 7fea30e41700 -1 osd.8 106 *** Got signal Terminated *** 2024-04-25T16:54:21.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:20.836+0000 7fea30e41700 -1 osd.8 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:54:21.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:20.956+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:21.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:20.956+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:21.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:20.956+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:21.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:20.956+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:21.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:20 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:20.956+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.687863+0000 front 2024-04-25T16:53:52.687916+0000 (oldest deadline 2024-04-25T16:54:17.387948+0000) 2024-04-25T16:54:21.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:21.223+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:21.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:21.223+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:21.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:21.223+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:21.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:21.223+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:21.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:21 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:21.223+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:21.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:21.428+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:21.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:21.428+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:21.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:21.428+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:21.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:21.428+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:22.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:21.922+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:22.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:21.922+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:22.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:21.922+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:22.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:21.922+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:22.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:21 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:21.922+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.687863+0000 front 2024-04-25T16:53:52.687916+0000 (oldest deadline 2024-04-25T16:54:17.387948+0000) 2024-04-25T16:54:22.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:21.955+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:22.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:21.955+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:22.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:21.955+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:22.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:21.955+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:22.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:21 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:21.955+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:22.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:22 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:22.241+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:22.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:22 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:22.241+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:22.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:22 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:22.241+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:22.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:22 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:22.241+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:22.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:22 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:22.241+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:22.685 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:22.400+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:22.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:22.400+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:22.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:22.400+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:22.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:22.400+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:23.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:22 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:22.997+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:23.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:22 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:22.997+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:23.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:22 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:22.997+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:23.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:22 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:22.997+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:23.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:22 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:22.997+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:23.402 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:22.948+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:23.403 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:22.948+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:23.403 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:22.948+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:23.403 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:22.948+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:23.403 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:22 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:22.948+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.687863+0000 front 2024-04-25T16:53:52.687916+0000 (oldest deadline 2024-04-25T16:54:17.387948+0000) 2024-04-25T16:54:23.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:23 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:23.201+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:23.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:23 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:23.201+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:23.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:23 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:23.201+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:23.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:23 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:23.201+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:23.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:23 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:23.201+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:23.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:23 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:23.201+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:23.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:23.405+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:23.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:23.405+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:23.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:23.405+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:23.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:23.405+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:23.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:23.405+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:24.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:23.906+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:24.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:23.906+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:24.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:23.906+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:24.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:23.906+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:24.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:23 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:23.906+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.687863+0000 front 2024-04-25T16:53:52.687916+0000 (oldest deadline 2024-04-25T16:54:17.387948+0000) 2024-04-25T16:54:24.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:24 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:24.246+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:24.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:24 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:24.246+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:24.500 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:24 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:24.246+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:24.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:24 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:24.246+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:24.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:24 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:24.246+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:24.501 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:24 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:24.246+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:24.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:24.028+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:24.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:24.028+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:24.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:24.028+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:24.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:24.028+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:24.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:24 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:24.028+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:24.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:24.375+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:24.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:24.375+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:24.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:24.375+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:24.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:24.375+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:24.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:24.375+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:25.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:24.934+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:21.183193+0000 front 2024-04-25T16:53:21.183215+0000 (oldest deadline 2024-04-25T16:53:42.883121+0000) 2024-04-25T16:54:25.186 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:24.934+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:26.984205+0000 front 2024-04-25T16:53:26.984266+0000 (oldest deadline 2024-04-25T16:53:52.884252+0000) 2024-04-25T16:54:25.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:24.934+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.085773+0000 front 2024-04-25T16:53:35.085925+0000 (oldest deadline 2024-04-25T16:53:59.785838+0000) 2024-04-25T16:54:25.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:24.934+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:44.487154+0000 front 2024-04-25T16:53:44.487171+0000 (oldest deadline 2024-04-25T16:54:08.586958+0000) 2024-04-25T16:54:25.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:24.934+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.687863+0000 front 2024-04-25T16:53:52.687916+0000 (oldest deadline 2024-04-25T16:54:17.387948+0000) 2024-04-25T16:54:25.187 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:24 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8[52226]: 2024-04-25T16:54:24.934+0000 7fea293ef700 -1 osd.8 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:59.688807+0000 front 2024-04-25T16:53:59.688888+0000 (oldest deadline 2024-04-25T16:54:24.388891+0000) 2024-04-25T16:54:25.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:25.008+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:25.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:25.008+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:25.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:25.008+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:25.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:25.008+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:25.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:25.008+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:25.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:25 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:25.008+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:25.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:25 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:25.417+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:25.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:25 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:25.417+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:25.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:25 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:25.417+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:25.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:25 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:25.417+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:25.686 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:25 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:25.417+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:25.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:25 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:25.292+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:25.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:25 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:25.292+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:25.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:25 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:25.292+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:25.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:25 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:25.292+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:25.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:25 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:25.292+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:25.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:25 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:25.292+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:26.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:26.051+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:26.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:26.051+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:26.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:26.051+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:26.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:26.051+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:26.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:26.051+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:26.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:26 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:26.051+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:26.669 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:26 smithi203 podman[73235]: 2024-04-25 16:54:26.400564177 +0000 UTC m=+5.671274947 container died b4cbd7bc44c761857cea67174fa25cb3e0a3ef45c85b1e121e91f6ad62190a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T16:54:26.669 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:26 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:26.443+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:26.669 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:26 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:26.443+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:26.669 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:26 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:26.443+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:26.669 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:26 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:26.443+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:26.670 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:26 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:26.443+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:26.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:26 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:26.317+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:26.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:26 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:26.317+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:26.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:26 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:26.317+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:26.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:26 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:26.317+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:26.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:26 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:26.317+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:26.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:26 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:26.317+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:26.936 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:26 smithi203 podman[73235]: 2024-04-25 16:54:26.729373279 +0000 UTC m=+6.000084055 container cleanup b4cbd7bc44c761857cea67174fa25cb3e0a3ef45c85b1e121e91f6ad62190a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T16:54:26.936 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:26 smithi203 bash[73235]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8 2024-04-25T16:54:27.388 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:26 smithi203 podman[73252]: 2024-04-25 16:54:26.955930638 +0000 UTC m=+0.552794913 container remove b4cbd7bc44c761857cea67174fa25cb3e0a3ef45c85b1e121e91f6ad62190a7d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T16:54:27.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:27.042+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:27.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:27.042+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:27.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:27.042+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:27.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:27.042+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:27.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:27.042+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:27.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:27.042+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:27.648 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:27 smithi203 podman[73342]: 2024-04-25 16:54:27.294532956 +0000 UTC m=+0.017815756 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:54:27.648 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:27 smithi203 podman[73342]: 2024-04-25 16:54:27.407021606 +0000 UTC m=+0.130304392 container create 285bacb8348c6e303ee73f2d0966677af1d8c6080f682fb4e75db1573ff271d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:54:27.649 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:27 smithi203 podman[73342]: 2024-04-25 16:54:27.651772815 +0000 UTC m=+0.375055606 container init 285bacb8348c6e303ee73f2d0966677af1d8c6080f682fb4e75db1573ff271d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T16:54:27.649 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:27 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:27.435+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:27.649 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:27 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:27.435+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:27.649 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:27 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:27.435+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:27.649 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:27 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:27.435+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:27.649 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:27 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:27.435+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:27.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:27.361+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:27.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:27.361+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:27.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:27.361+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:27.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:27.361+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:27.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:27.361+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:27.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:27 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:27.361+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:27.936 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:27 smithi203 podman[73342]: 2024-04-25 16:54:27.655545467 +0000 UTC m=+0.378828262 container start 285bacb8348c6e303ee73f2d0966677af1d8c6080f682fb4e75db1573ff271d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:54:27.936 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:27 smithi203 podman[73342]: 2024-04-25 16:54:27.818905691 +0000 UTC m=+0.542188493 container attach 285bacb8348c6e303ee73f2d0966677af1d8c6080f682fb4e75db1573ff271d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:54:28.302 INFO:journalctl@ceph.osd.8.smithi203.stdout:Apr 25 16:54:27 smithi203 podman[73342]: 2024-04-25 16:54:27.94949819 +0000 UTC m=+0.672780983 container died 285bacb8348c6e303ee73f2d0966677af1d8c6080f682fb4e75db1573ff271d9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:54:28.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:28 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:28.078+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:28.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:28 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:28.078+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:28.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:28 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:28.078+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:28.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:28 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:28.078+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:28.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:28 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:28.078+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:28.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:28 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:28.078+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:28.585 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:28.420+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:28.585 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:28.420+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:28.585 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:28.420+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:28.585 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:28.420+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:28.586 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:28 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:28.420+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:28.612 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T16:54:28.650 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:28.651 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T16:54:28.651 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T16:54:28.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.9 2024-04-25T16:54:28.678 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.395+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:28.678 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.395+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:28.678 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.395+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:28.679 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.395+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:28.679 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.395+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:28.679 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.395+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:28.945 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 systemd[1]: Stopping Ceph osd.9 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:29.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.947+0000 7f2bc4248700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:29.254 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.947+0000 7f2bc4248700 -1 osd.9 106 *** Got signal Terminated *** 2024-04-25T16:54:29.254 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:28 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:28.947+0000 7f2bc4248700 -1 osd.9 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:54:29.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:29 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:29.045+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:29.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:29 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:29.045+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:29.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:29 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:29.045+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:29.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:29 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:29.045+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:29.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:29 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:29.045+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:29.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:29 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:29.045+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:29.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:29 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:29.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:29.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:29 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:29.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:29.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:29 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:29.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:29.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:29 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:29.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:29.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:29 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:29.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:29.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:29 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:29.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:29.792 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:29 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:29.452+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:29.793 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:29 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:29.452+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:29.793 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:29 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:29.452+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:29.793 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:29 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:29.452+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:29.793 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:29 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:29.452+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:30.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:30 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:30.035+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:30.512 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:30 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:30.035+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:30.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:30 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:30.035+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:30.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:30 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:30.035+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:30.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:30 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:30.035+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:30.513 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:30 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:30.035+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:30.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:30 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:30.433+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:30.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:30 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:30.433+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:30.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:30 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:30.433+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:30.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:30 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:30.433+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:30.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:30 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:30.433+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:30.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:30 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:30.433+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:30.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:30 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:30.497+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:30.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:30 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:30.497+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:30.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:30 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:30.497+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:30.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:30 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:30.497+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:30.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:30 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:30.497+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:31.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.002+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:31.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.002+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:31.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.002+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:31.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.002+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:31.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.002+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:31.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.002+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:31.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:31 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:31.413+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:31.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:31 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:31.413+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:31.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:31 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:31.413+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:31.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:31 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:31.413+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:31.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:31 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:31.413+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:31.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:31 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:31.413+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:31.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:31 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:31.471+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:31.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:31 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:31.471+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:31.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:31 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:31.471+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:31.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:31 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:31.471+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:31.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:31 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:31.471+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:31.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:31 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:31.471+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:32.012 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.969+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:32.012 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.969+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:32.012 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.969+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:32.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.969+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:32.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.969+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:32.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.969+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:32.013 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:31 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:31.970+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:32.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:32 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:32.404+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:32.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:32 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:32.404+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:32.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:32 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:32.404+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:32.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:32 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:32.404+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:32.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:32 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:32.404+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:32.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:32 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:32.404+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:32.837 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:32.470+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:32.837 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:32.470+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:32.837 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:32.470+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:32.838 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:32.470+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:32.838 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:32.470+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:32.838 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:32.470+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:33.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:32 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:32.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:33.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:32 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:32.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:33.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:32 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:32.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:33.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:32 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:32.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:33.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:32 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:32.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:33.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:32 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:32.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:33.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:32 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:32.928+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:33.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:33.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:19.860572+0000 front 2024-04-25T16:53:19.860520+0000 (oldest deadline 2024-04-25T16:53:44.560735+0000) 2024-04-25T16:54:33.750 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:33.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.463154+0000 front 2024-04-25T16:53:28.463117+0000 (oldest deadline 2024-04-25T16:53:51.363279+0000) 2024-04-25T16:54:33.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:33.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:35.865582+0000 front 2024-04-25T16:53:35.865707+0000 (oldest deadline 2024-04-25T16:54:01.765532+0000) 2024-04-25T16:54:33.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:33.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:43.466368+0000 front 2024-04-25T16:53:43.466313+0000 (oldest deadline 2024-04-25T16:54:09.366449+0000) 2024-04-25T16:54:33.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:33.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:52.168020+0000 front 2024-04-25T16:53:52.168097+0000 (oldest deadline 2024-04-25T16:54:17.467820+0000) 2024-04-25T16:54:33.751 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:33 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9[63954]: 2024-04-25T16:54:33.393+0000 7f2bc07fe700 -1 osd.9 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:57.468576+0000 front 2024-04-25T16:53:57.468693+0000 (oldest deadline 2024-04-25T16:54:22.768359+0000) 2024-04-25T16:54:33.849 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:33 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:33.488+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:33.849 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:33 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:33.488+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:33.849 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:33 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:33.488+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:33.850 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:33 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:33.488+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:33.850 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:33 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:33.488+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:33.850 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:33 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:33.488+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:34.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:33 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:33.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:34.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:33 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:33.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:34.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:33 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:33.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:34.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:33 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:33.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:34.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:33 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:33.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:34.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:33 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:33.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:34.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:33 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:33.968+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:34.562 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:34 smithi098 podman[128300]: 2024-04-25 16:54:34.314514633 +0000 UTC m=+5.468899478 container died 92514b8edd23a227af910752d11d7ca6a8f82de5f81e9c2a1705c4b6b6f981ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T16:54:34.888 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:34 smithi098 podman[128300]: 2024-04-25 16:54:34.656459365 +0000 UTC m=+5.810844214 container cleanup 92514b8edd23a227af910752d11d7ca6a8f82de5f81e9c2a1705c4b6b6f981ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T16:54:34.888 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:34 smithi098 bash[128300]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9 2024-04-25T16:54:34.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:34 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:34.459+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:34.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:34 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:34.459+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:34.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:34 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:34.459+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:34.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:34 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:34.459+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:34.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:34 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:34.459+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:34.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:34 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:34.459+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:35.250 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:34 smithi098 podman[128318]: 2024-04-25 16:54:34.891180374 +0000 UTC m=+0.575212233 container remove 92514b8edd23a227af910752d11d7ca6a8f82de5f81e9c2a1705c4b6b6f981ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T16:54:35.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:34 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:34.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:35.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:34 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:34.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:35.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:34 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:34.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:35.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:34 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:34.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:35.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:34 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:34.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:35.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:34 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:34.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:35.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:34 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:34.921+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:35.575 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:35 smithi098 podman[128408]: 2024-04-25 16:54:35.216718948 +0000 UTC m=+0.017667887 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:54:35.575 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:35 smithi098 podman[128408]: 2024-04-25 16:54:35.359175365 +0000 UTC m=+0.160124292 container create 4890745bd92d45bb3faaa0d9e586713c90c8a73598e58362bb515554f066d8d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T16:54:35.660 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:35 smithi098 podman[128408]: 2024-04-25 16:54:35.578413202 +0000 UTC m=+0.379362133 container init 4890745bd92d45bb3faaa0d9e586713c90c8a73598e58362bb515554f066d8d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-25T16:54:35.660 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:35 smithi098 podman[128408]: 2024-04-25 16:54:35.582188419 +0000 UTC m=+0.383137346 container start 4890745bd92d45bb3faaa0d9e586713c90c8a73598e58362bb515554f066d8d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-25T16:54:35.661 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:35 smithi098 podman[128408]: 2024-04-25 16:54:35.661861624 +0000 UTC m=+0.462810553 container attach 4890745bd92d45bb3faaa0d9e586713c90c8a73598e58362bb515554f066d8d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T16:54:35.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:35 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:35.447+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:35.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:35 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:35.447+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:35.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:35 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:35.447+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:35.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:35 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:35.447+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:35.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:35 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:35.447+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:35.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:35 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:35.447+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:36.170 INFO:journalctl@ceph.osd.9.smithi098.stdout:Apr 25 16:54:35 smithi098 podman[128408]: 2024-04-25 16:54:35.874621928 +0000 UTC m=+0.675570851 container died 4890745bd92d45bb3faaa0d9e586713c90c8a73598e58362bb515554f066d8d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:54:36.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:35.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:36.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:35.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:36.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:35.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:36.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:35.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:36.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:35.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:36.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:35.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:36.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:35 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:35.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:36.456 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T16:54:36.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:36.545 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T16:54:36.545 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T16:54:36.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.10 2024-04-25T16:54:36.861 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 systemd[1]: Stopping Ceph osd.10 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:36.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:36 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:36.487+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:36.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:36 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:36.487+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:36.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:36 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:36.487+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:36.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:36 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:36.487+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:36.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:36 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:36.487+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:36.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:36 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:36.487+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:37.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.863+0000 7f799a358700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:37.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.863+0000 7f799a358700 -1 osd.10 106 *** Got signal Terminated *** 2024-04-25T16:54:37.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.863+0000 7f799a358700 -1 osd.10 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:54:37.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.924+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:37.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.924+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:37.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.924+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:37.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.924+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:37.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.924+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:37.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.924+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:37.264 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:36 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:36.924+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:37.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:37.465+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:37.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:37.465+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:37.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:37.465+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:37.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:37.465+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:37.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:37.465+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:37.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:37.465+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:38.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:37.898+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:38.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:37.898+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:38.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:37.898+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:38.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:37.898+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:38.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:37.898+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:38.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:37.898+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:38.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:37 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:37.898+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:38.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:38 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:38.442+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:38.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:38 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:38.442+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:38.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:38 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:38.442+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:38.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:38 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:38.442+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:38.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:38 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:38.442+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:38.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:38 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:38.442+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:39.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:38 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:38.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:39.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:38 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:38.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:39.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:38 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:38.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:39.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:38 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:38.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:39.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:38 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:38.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:39.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:38 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:38.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:39.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:38 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:38.943+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:39.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:39 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:39.467+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:39.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:39 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:39.467+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:39.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:39 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:39.467+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:39.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:39 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:39.467+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:39.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:39 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:39.467+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:39.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:39 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:39.467+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:40.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:39.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:40.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:39.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:40.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:39.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:40.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:39.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:40.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:39.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:40.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:39.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:40.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:39 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:39.927+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:40.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:40 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:40.460+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:40.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:40 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:40.460+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:40.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:40 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:40.460+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:40.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:40 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:40.460+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:40.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:40 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:40.460+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:40.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:40 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:40.460+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:41.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:40.894+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:20.240158+0000 front 2024-04-25T16:53:20.240324+0000 (oldest deadline 2024-04-25T16:53:46.140327+0000) 2024-04-25T16:54:41.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:40.894+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:29.542089+0000 front 2024-04-25T16:53:29.542258+0000 (oldest deadline 2024-04-25T16:53:52.442264+0000) 2024-04-25T16:54:41.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:40.894+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:32.442934+0000 front 2024-04-25T16:53:32.442725+0000 (oldest deadline 2024-04-25T16:54:00.643289+0000) 2024-04-25T16:54:41.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:40.894+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6814 osd.3 since back 2024-04-25T16:53:45.145450+0000 front 2024-04-25T16:53:45.145389+0000 (oldest deadline 2024-04-25T16:54:08.045137+0000) 2024-04-25T16:54:41.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:40.894+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:49.746090+0000 front 2024-04-25T16:53:49.746131+0000 (oldest deadline 2024-04-25T16:54:18.546786+0000) 2024-04-25T16:54:41.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:40.894+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.547601+0000 front 2024-04-25T16:53:58.547486+0000 (oldest deadline 2024-04-25T16:54:24.447390+0000) 2024-04-25T16:54:41.263 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:40 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10[57275]: 2024-04-25T16:54:40.894+0000 7f7992906700 -1 osd.10 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:08.449140+0000 front 2024-04-25T16:54:08.448994+0000 (oldest deadline 2024-04-25T16:54:31.348945+0000) 2024-04-25T16:54:41.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:41.503+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:41.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:41.503+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:41.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:41.503+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:41.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:41.503+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:41.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:41.503+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:41.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:41.503+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:41.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:41.503+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:42.552 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:42 smithi173 podman[71455]: 2024-04-25 16:54:42.267811904 +0000 UTC m=+5.521519547 container died 24c5a9729bb707dac917ec8726a4557cded314d9b53543a800e58fe3346d6723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-25T16:54:42.919 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:42 smithi173 podman[71455]: 2024-04-25 16:54:42.654765404 +0000 UTC m=+5.908473042 container cleanup 24c5a9729bb707dac917ec8726a4557cded314d9b53543a800e58fe3346d6723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T16:54:42.919 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:42 smithi173 bash[71455]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10 2024-04-25T16:54:42.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:42 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:42.457+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:42.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:42 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:42.457+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:42.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:42 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:42.457+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:42.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:42 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:42.457+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:42.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:42 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:42.457+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:42.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:42 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:42.457+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:42.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:42 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:42.457+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:43.262 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:42 smithi173 podman[71471]: 2024-04-25 16:54:42.922665754 +0000 UTC m=+0.658447169 container remove 24c5a9729bb707dac917ec8726a4557cded314d9b53543a800e58fe3346d6723 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T16:54:43.556 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:43 smithi173 podman[71560]: 2024-04-25 16:54:43.174584137 +0000 UTC m=+0.017959777 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:54:43.556 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:43 smithi173 podman[71560]: 2024-04-25 16:54:43.340286154 +0000 UTC m=+0.183661848 container create f11b726d10aeb655236d343ed78a1b5ffb99641a4778494f7500d1892641f10b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T16:54:43.556 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:43 smithi173 podman[71560]: 2024-04-25 16:54:43.559471045 +0000 UTC m=+0.402846677 container init f11b726d10aeb655236d343ed78a1b5ffb99641a4778494f7500d1892641f10b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:54:43.849 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:43 smithi173 podman[71560]: 2024-04-25 16:54:43.562709276 +0000 UTC m=+0.406084918 container start f11b726d10aeb655236d343ed78a1b5ffb99641a4778494f7500d1892641f10b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T16:54:43.849 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:43 smithi173 podman[71560]: 2024-04-25 16:54:43.642694427 +0000 UTC m=+0.486070065 container attach f11b726d10aeb655236d343ed78a1b5ffb99641a4778494f7500d1892641f10b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T16:54:43.892 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:43.449+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:43.893 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:43.449+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:43.893 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:43.449+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:43.893 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:43.449+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:43.893 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:43.449+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:43.893 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:43.449+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:43.893 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:43 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:43.449+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:44.142 INFO:journalctl@ceph.osd.10.smithi173.stdout:Apr 25 16:54:43 smithi173 podman[71560]: 2024-04-25 16:54:43.85302483 +0000 UTC m=+0.696400450 container died f11b726d10aeb655236d343ed78a1b5ffb99641a4778494f7500d1892641f10b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-25T16:54:44.465 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T16:54:44.505 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:44.505 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T16:54:44.505 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T16:54:44.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.11 2024-04-25T16:54:44.830 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.450+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:44.830 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.450+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:44.830 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.450+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:44.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.450+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:44.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.450+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:44.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.450+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:44.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.450+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:44.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 systemd[1]: Stopping Ceph osd.11 for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:44.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.832+0000 7f3cd5478700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:44.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.832+0000 7f3cd5478700 -1 osd.11 106 *** Got signal Terminated *** 2024-04-25T16:54:44.831 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:44 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:44.832+0000 7f3cd5478700 -1 osd.11 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:54:45.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:45.440+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:45.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:45.440+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:45.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:45.440+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:45.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:45.440+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:45.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:45.440+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:45.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:45.440+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:45.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:45.440+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:46.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:46.454+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:46.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:46.454+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:46.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:46.454+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:46.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:46.454+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:46.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:46.454+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:46.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:46.454+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:46.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:46 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:46.454+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:47.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:47.479+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:47.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:47.479+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:47.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:47.479+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:47.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:47.479+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:47.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:47.479+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:47.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:47.479+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:47.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:47 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:47.479+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:48.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:48.462+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:48.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:48.462+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:48.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:48.462+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:48.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:48.462+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:48.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:48.462+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:48.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:48.462+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:48.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:48 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:48.462+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:49.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:49.500+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-04-25T16:53:15.579080+0000 front 2024-04-25T16:53:15.578953+0000 (oldest deadline 2024-04-25T16:53:42.579600+0000) 2024-04-25T16:54:49.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:49.500+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6804 osd.1 since back 2024-04-25T16:53:28.882152+0000 front 2024-04-25T16:53:28.881971+0000 (oldest deadline 2024-04-25T16:53:51.782090+0000) 2024-04-25T16:54:49.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:49.500+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6804 osd.2 since back 2024-04-25T16:53:36.383854+0000 front 2024-04-25T16:53:36.383972+0000 (oldest deadline 2024-04-25T16:53:59.283951+0000) 2024-04-25T16:54:49.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:49.500+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6812 osd.4 since back 2024-04-25T16:53:51.487237+0000 front 2024-04-25T16:53:51.487092+0000 (oldest deadline 2024-04-25T16:54:16.186889+0000) 2024-04-25T16:54:49.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:49.500+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.203:6812 osd.5 since back 2024-04-25T16:53:58.487730+0000 front 2024-04-25T16:53:58.487810+0000 (oldest deadline 2024-04-25T16:54:22.587903+0000) 2024-04-25T16:54:49.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:49.500+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.98:6822 osd.6 since back 2024-04-25T16:54:09.490002+0000 front 2024-04-25T16:54:09.490054+0000 (oldest deadline 2024-04-25T16:54:30.589801+0000) 2024-04-25T16:54:49.937 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:49 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11[58845]: 2024-04-25T16:54:49.500+0000 7f3cd1a2e700 -1 osd.11 106 heartbeat_check: no reply from 172.21.15.173:6820 osd.7 since back 2024-04-25T16:54:15.891282+0000 front 2024-04-25T16:54:15.891202+0000 (oldest deadline 2024-04-25T16:54:41.190977+0000) 2024-04-25T16:54:50.638 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:50 smithi203 podman[73463]: 2024-04-25 16:54:50.366351563 +0000 UTC m=+5.665868039 container died 7ab3517cc00eaee3f8e70adbb3af4cd1c32439c0770395c0d5c26700e271f6db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T16:54:50.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:50 smithi203 podman[73463]: 2024-04-25 16:54:50.732961779 +0000 UTC m=+6.032478250 container cleanup 7ab3517cc00eaee3f8e70adbb3af4cd1c32439c0770395c0d5c26700e271f6db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:54:50.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:50 smithi203 bash[73463]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11 2024-04-25T16:54:51.296 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:50 smithi203 podman[73481]: 2024-04-25 16:54:50.967527311 +0000 UTC m=+0.599491611 container remove 7ab3517cc00eaee3f8e70adbb3af4cd1c32439c0770395c0d5c26700e271f6db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T16:54:51.561 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:51 smithi203 podman[73570]: 2024-04-25 16:54:51.2021353 +0000 UTC m=+0.018410442 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:54:51.562 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:51 smithi203 podman[73570]: 2024-04-25 16:54:51.310233694 +0000 UTC m=+0.126508809 container create 57df051989e77731dd3b267efc39958cdad3f9186509168f1587611677f91cc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T16:54:51.562 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:51 smithi203 podman[73570]: 2024-04-25 16:54:51.487779653 +0000 UTC m=+0.304054771 container init 57df051989e77731dd3b267efc39958cdad3f9186509168f1587611677f91cc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:54:51.562 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:51 smithi203 podman[73570]: 2024-04-25 16:54:51.490772085 +0000 UTC m=+0.307047205 container start 57df051989e77731dd3b267efc39958cdad3f9186509168f1587611677f91cc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T16:54:51.562 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:51 smithi203 podman[73570]: 2024-04-25 16:54:51.56272266 +0000 UTC m=+0.378997785 container attach 57df051989e77731dd3b267efc39958cdad3f9186509168f1587611677f91cc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T16:54:51.936 INFO:journalctl@ceph.osd.11.smithi203.stdout:Apr 25 16:54:51 smithi203 podman[73570]: 2024-04-25 16:54:51.790767805 +0000 UTC m=+0.607042934 container died 57df051989e77731dd3b267efc39958cdad3f9186509168f1587611677f91cc6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T16:54:52.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T16:54:52.385 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:52.386 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T16:54:52.386 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T16:54:52.386 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.a 2024-04-25T16:54:52.676 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:54:52 smithi098 systemd[1]: Stopping Ceph mds.a for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:53.000 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:54:52 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a[70762]: 2024-04-25T16:54:52.678+0000 7f2255566700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:53.000 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:54:52 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a[70762]: 2024-04-25T16:54:52.678+0000 7f2255566700 -1 mds.a *** got signal Terminated *** 2024-04-25T16:54:54.276 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:54:54 smithi098 podman[128536]: 2024-04-25 16:54:54.010598363 +0000 UTC m=+1.427749142 container died c85411a4bd2ace02410ba3c35ee07e2422310b6afc9151d669f1eda9fe47f301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T16:54:54.750 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:54:54 smithi098 podman[128536]: 2024-04-25 16:54:54.353313871 +0000 UTC m=+1.770464655 container cleanup c85411a4bd2ace02410ba3c35ee07e2422310b6afc9151d669f1eda9fe47f301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T16:54:54.750 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:54:54 smithi098 bash[128536]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a 2024-04-25T16:54:54.751 INFO:journalctl@ceph.mds.a.smithi098.stdout:Apr 25 16:54:54 smithi098 podman[128551]: 2024-04-25 16:54:54.529551474 +0000 UTC m=+0.509727714 container remove c85411a4bd2ace02410ba3c35ee07e2422310b6afc9151d669f1eda9fe47f301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-04-25T16:54:54.767 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T16:54:54.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:54.801 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T16:54:54.801 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T16:54:54.802 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.d 2024-04-25T16:54:55.138 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:54 smithi098 systemd[1]: Stopping Ceph mds.d for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:55.500 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:55 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d[69803]: 2024-04-25T16:54:55.140+0000 7f0698f57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:55.500 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:55 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d[69803]: 2024-04-25T16:54:55.140+0000 7f0698f57700 -1 mds.d *** got signal Terminated *** 2024-04-25T16:54:58.513 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:58 smithi098 podman[128657]: 2024-04-25 16:54:58.263774803 +0000 UTC m=+3.234521426 container died c6012484385a1a80826e7290b96209abe085c6ecda5bc0bfd1d25634dc8f5c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T16:54:58.805 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:58 smithi098 podman[128657]: 2024-04-25 16:54:58.607268412 +0000 UTC m=+3.578015089 container cleanup c6012484385a1a80826e7290b96209abe085c6ecda5bc0bfd1d25634dc8f5c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:54:58.805 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:58 smithi098 bash[128657]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d 2024-04-25T16:54:59.042 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T16:54:59.062 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:58 smithi098 podman[128671]: 2024-04-25 16:54:58.808660376 +0000 UTC m=+0.543229174 container remove c6012484385a1a80826e7290b96209abe085c6ecda5bc0bfd1d25634dc8f5c96 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T16:54:59.063 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:59 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.d.service: Deactivated successfully. 2024-04-25T16:54:59.063 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:59 smithi098 systemd[1]: Stopped Ceph mds.d for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:54:59.063 INFO:journalctl@ceph.mds.d.smithi098.stdout:Apr 25 16:54:59 smithi098 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.d.service: Consumed 1.179s CPU time. 2024-04-25T16:54:59.073 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:54:59.073 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T16:54:59.074 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T16:54:59.074 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g 2024-04-25T16:54:59.400 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:54:59 smithi098 systemd[1]: Stopping Ceph mds.g for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:54:59.750 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:54:59 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[91203]: 2024-04-25T16:54:59.402+0000 7f1ae3a9f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:54:59.750 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:54:59 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g[91203]: 2024-04-25T16:54:59.402+0000 7f1ae3a9f700 -1 mds.g *** got signal Terminated *** 2024-04-25T16:55:02.358 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:55:02 smithi098 podman[128777]: 2024-04-25 16:55:02.105598267 +0000 UTC m=+2.798925436 container died 6a9cb0b7c61d9462670e9439fddf7100b031df9e56cd9bab086043896edddc2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T16:55:02.708 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:55:02 smithi098 podman[128777]: 2024-04-25 16:55:02.45266099 +0000 UTC m=+3.145988160 container cleanup 6a9cb0b7c61d9462670e9439fddf7100b031df9e56cd9bab086043896edddc2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T16:55:02.709 INFO:journalctl@ceph.mds.g.smithi098.stdout:Apr 25 16:55:02 smithi098 bash[128777]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-g 2024-04-25T16:55:02.880 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T16:55:02.911 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:02.912 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T16:55:02.912 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T16:55:02.912 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.j 2024-04-25T16:55:03.250 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:55:03 smithi098 systemd[1]: Stopping Ceph mds.j for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:03.250 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:55:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j[69436]: 2024-04-25T16:55:03.222+0000 7f7cb0d84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:03.250 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:55:03 smithi098 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j[69436]: 2024-04-25T16:55:03.222+0000 7f7cb0d84700 -1 mds.j *** got signal Terminated *** 2024-04-25T16:55:07.500 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:55:07 smithi098 podman[128896]: 2024-04-25 16:55:07.230219387 +0000 UTC m=+4.092279949 container died 62399e7e22529553536415a18e45342142a13d55463bdb1783ab104af0dc64d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T16:55:07.846 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:55:07 smithi098 podman[128896]: 2024-04-25 16:55:07.614703101 +0000 UTC m=+4.476763663 container cleanup 62399e7e22529553536415a18e45342142a13d55463bdb1783ab104af0dc64d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T16:55:07.846 INFO:journalctl@ceph.mds.j.smithi098.stdout:Apr 25 16:55:07 smithi098 bash[128896]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-j 2024-04-25T16:55:07.959 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T16:55:07.989 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:07.989 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T16:55:07.990 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T16:55:07.990 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b 2024-04-25T16:55:08.268 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:08 smithi173 systemd[1]: Stopping Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:08.605 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: 2024-04-25T16:55:08.270+0000 7f2e2b3cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:08.606 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:08 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b[66646]: 2024-04-25T16:55:08.270+0000 7f2e2b3cc700 -1 mds.b *** got signal Terminated *** 2024-04-25T16:55:16.766 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:16 smithi173 podman[71665]: 2024-04-25 16:55:16.502197625 +0000 UTC m=+8.319672642 container died f14df9f6defc8915e614d874aad5bea811f7baf20669ad19ab6ac1b756033328 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:55:17.185 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:16 smithi173 podman[71665]: 2024-04-25 16:55:16.945915237 +0000 UTC m=+8.763390253 container cleanup f14df9f6defc8915e614d874aad5bea811f7baf20669ad19ab6ac1b756033328 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:55:17.185 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:16 smithi173 bash[71665]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b 2024-04-25T16:55:17.413 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T16:55:17.436 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:17 smithi173 podman[71691]: 2024-04-25 16:55:17.188836318 +0000 UTC m=+0.680651793 container remove f14df9f6defc8915e614d874aad5bea811f7baf20669ad19ab6ac1b756033328 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:55:17.436 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:17 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Deactivated successfully. 2024-04-25T16:55:17.436 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:17 smithi173 systemd[1]: Stopped Ceph mds.b for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:55:17.436 INFO:journalctl@ceph.mds.b.smithi173.stdout:Apr 25 16:55:17 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.b.service: Consumed 1min 45.752s CPU time. 2024-04-25T16:55:17.448 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:17.448 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T16:55:17.449 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T16:55:17.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.e 2024-04-25T16:55:17.762 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:17 smithi173 systemd[1]: Stopping Ceph mds.e for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:18.262 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e[62176]: 2024-04-25T16:55:17.791+0000 7f8dcc2ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:18.262 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:17 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e[62176]: 2024-04-25T16:55:17.791+0000 7f8dcc2ff700 -1 mds.e *** got signal Terminated *** 2024-04-25T16:55:21.714 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:21 smithi173 podman[71799]: 2024-04-25 16:55:21.343836489 +0000 UTC m=+3.679874140 container died ac8af86570cca7515d05eefd46407d8c939dbf17d27f5c8144812f2a153c8350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:55:21.980 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:21 smithi173 podman[71799]: 2024-04-25 16:55:21.782801388 +0000 UTC m=+4.118839027 container cleanup ac8af86570cca7515d05eefd46407d8c939dbf17d27f5c8144812f2a153c8350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:55:21.981 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:21 smithi173 bash[71799]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e 2024-04-25T16:55:22.209 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T16:55:22.230 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:21 smithi173 podman[71814]: 2024-04-25 16:55:21.984001594 +0000 UTC m=+0.642192669 container remove ac8af86570cca7515d05eefd46407d8c939dbf17d27f5c8144812f2a153c8350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T16:55:22.231 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:22 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.e.service: Deactivated successfully. 2024-04-25T16:55:22.231 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:22 smithi173 systemd[1]: Stopped Ceph mds.e for c520eaa8-031f-11ef-bc93-c7b262605968. 2024-04-25T16:55:22.231 INFO:journalctl@ceph.mds.e.smithi173.stdout:Apr 25 16:55:22 smithi173 systemd[1]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.e.service: Consumed 21.126s CPU time. 2024-04-25T16:55:22.240 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:22.241 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T16:55:22.241 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T16:55:22.241 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h 2024-04-25T16:55:22.516 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:55:22 smithi173 systemd[1]: Stopping Ceph mds.h for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:23.012 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:55:22 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[68198]: 2024-04-25T16:55:22.570+0000 7f00f7ecb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:23.012 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:55:22 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h[68198]: 2024-04-25T16:55:22.570+0000 7f00f7ecb700 -1 mds.h *** got signal Terminated *** 2024-04-25T16:55:26.700 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:55:26 smithi173 podman[71922]: 2024-04-25 16:55:26.432697729 +0000 UTC m=+3.972301630 container died ddc980f52dbd50e2d6b55b122d63cdbbf7d68af1a7cac594eb44be4b59eab257 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T16:55:27.012 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:55:26 smithi173 podman[71922]: 2024-04-25 16:55:26.794791319 +0000 UTC m=+4.334395228 container cleanup ddc980f52dbd50e2d6b55b122d63cdbbf7d68af1a7cac594eb44be4b59eab257 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:55:27.012 INFO:journalctl@ceph.mds.h.smithi173.stdout:Apr 25 16:55:26 smithi173 bash[71922]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-h 2024-04-25T16:55:27.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T16:55:27.242 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:27.243 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T16:55:27.243 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T16:55:27.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k 2024-04-25T16:55:27.561 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:55:27 smithi173 systemd[1]: Stopping Ceph mds.k for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:28.012 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:55:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[68030]: 2024-04-25T16:55:27.564+0000 7f885f9d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:28.012 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:55:27 smithi173 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k[68030]: 2024-04-25T16:55:27.564+0000 7f885f9d0700 -1 mds.k *** got signal Terminated *** 2024-04-25T16:55:32.094 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:55:31 smithi173 podman[72041]: 2024-04-25 16:55:31.799945716 +0000 UTC m=+4.342593500 container died a61e5ea3dda99a8de771696af8485fe72d8735f5d481fdad2a06159e96a90511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:55:32.094 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:55:32 smithi173 podman[72041]: 2024-04-25 16:55:32.097874208 +0000 UTC m=+4.640521991 container cleanup a61e5ea3dda99a8de771696af8485fe72d8735f5d481fdad2a06159e96a90511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T16:55:32.439 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:55:32 smithi173 bash[72041]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k 2024-04-25T16:55:32.439 INFO:journalctl@ceph.mds.k.smithi173.stdout:Apr 25 16:55:32 smithi173 podman[72055]: 2024-04-25 16:55:32.32367772 +0000 UTC m=+0.525027534 container remove a61e5ea3dda99a8de771696af8485fe72d8735f5d481fdad2a06159e96a90511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-k, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-04-25T16:55:32.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T16:55:32.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:32.484 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T16:55:32.484 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T16:55:32.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c 2024-04-25T16:55:32.788 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:55:32 smithi203 systemd[1]: Stopping Ceph mds.c for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:33.048 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:55:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[69798]: 2024-04-25T16:55:32.790+0000 7f92ace98700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:33.048 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:55:32 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c[69798]: 2024-04-25T16:55:32.790+0000 7f92ace98700 -1 mds.c *** got signal Terminated *** 2024-04-25T16:55:36.437 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:55:36 smithi203 podman[73676]: 2024-04-25 16:55:36.170919898 +0000 UTC m=+3.489261230 container died 6bdeb6453932c551c48b9a94c6801663a192304d9a4d2e772fe26f8e3e1a3cb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T16:55:36.822 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:55:36 smithi203 podman[73676]: 2024-04-25 16:55:36.574528102 +0000 UTC m=+3.892869442 container cleanup 6bdeb6453932c551c48b9a94c6801663a192304d9a4d2e772fe26f8e3e1a3cb3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-25T16:55:36.822 INFO:journalctl@ceph.mds.c.smithi203.stdout:Apr 25 16:55:36 smithi203 bash[73676]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-c 2024-04-25T16:55:36.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T16:55:36.974 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:36.974 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T16:55:36.974 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T16:55:36.974 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f 2024-04-25T16:55:37.186 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:55:37 smithi203 systemd[1]: Stopping Ceph mds.f for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:37.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:55:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[71337]: 2024-04-25T16:55:37.285+0000 7f1e2156e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:37.686 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:55:37 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f[71337]: 2024-04-25T16:55:37.285+0000 7f1e2156e700 -1 mds.f *** got signal Terminated *** 2024-04-25T16:55:40.908 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:55:40 smithi203 podman[73810]: 2024-04-25 16:55:40.595123854 +0000 UTC m=+3.403779604 container died cddb1fa31d5bda8fcdf312279303a01b1f7e67b5217768e8afcab1776dfd2629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T16:55:40.909 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:55:40 smithi203 podman[73810]: 2024-04-25 16:55:40.912000906 +0000 UTC m=+3.720656655 container cleanup cddb1fa31d5bda8fcdf312279303a01b1f7e67b5217768e8afcab1776dfd2629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T16:55:41.185 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:55:40 smithi203 bash[73810]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f 2024-04-25T16:55:41.186 INFO:journalctl@ceph.mds.f.smithi203.stdout:Apr 25 16:55:41 smithi203 podman[73824]: 2024-04-25 16:55:41.088267583 +0000 UTC m=+0.487175595 container remove cddb1fa31d5bda8fcdf312279303a01b1f7e67b5217768e8afcab1776dfd2629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T16:55:41.211 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T16:55:41.243 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:41.244 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T16:55:41.244 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T16:55:41.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i 2024-04-25T16:55:41.563 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:55:41 smithi203 systemd[1]: Stopping Ceph mds.i for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:41.936 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:55:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[69705]: 2024-04-25T16:55:41.564+0000 7ffb63d16700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:41.936 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:55:41 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i[69705]: 2024-04-25T16:55:41.564+0000 7ffb63d16700 -1 mds.i *** got signal Terminated *** 2024-04-25T16:55:44.334 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:55:44 smithi203 podman[73931]: 2024-04-25 16:55:44.03392612 +0000 UTC m=+2.574477876 container died e7ce0df81dab0f3e856e890e8300912bb73a1092e0f4ed17b99618f23805dd4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T16:55:44.639 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:55:44 smithi203 podman[73931]: 2024-04-25 16:55:44.433332979 +0000 UTC m=+2.973884732 container cleanup e7ce0df81dab0f3e856e890e8300912bb73a1092e0f4ed17b99618f23805dd4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T16:55:44.640 INFO:journalctl@ceph.mds.i.smithi203.stdout:Apr 25 16:55:44 smithi203 bash[73931]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-i 2024-04-25T16:55:44.781 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T16:55:44.812 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:44.812 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T16:55:44.812 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T16:55:44.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l 2024-04-25T16:55:44.935 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:55:44 smithi203 systemd[1]: Stopping Ceph mds.l for c520eaa8-031f-11ef-bc93-c7b262605968... 2024-04-25T16:55:45.436 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:55:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[70199]: 2024-04-25T16:55:45.110+0000 7ff507d73700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:55:45.436 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:55:45 smithi203 ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l[70199]: 2024-04-25T16:55:45.110+0000 7ff507d73700 -1 mds.l *** got signal Terminated *** 2024-04-25T16:55:47.936 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:55:47 smithi203 podman[74053]: 2024-04-25 16:55:47.613976779 +0000 UTC m=+2.587277851 container died 1e88eb729659811f4aa96843952a943faf477ffaad20126e65e5e48f96b08dd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T16:55:48.243 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:55:48 smithi203 podman[74053]: 2024-04-25 16:55:48.06236147 +0000 UTC m=+3.035662592 container cleanup 1e88eb729659811f4aa96843952a943faf477ffaad20126e65e5e48f96b08dd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T16:55:48.244 INFO:journalctl@ceph.mds.l.smithi203.stdout:Apr 25 16:55:48 smithi203 bash[74053]: ceph-c520eaa8-031f-11ef-bc93-c7b262605968-mds-l 2024-04-25T16:55:48.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c520eaa8-031f-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T16:55:48.501 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:55:48.501 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T16:55:48.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c520eaa8-031f-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T16:55:48.692 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:55:51.140 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c520eaa8-031f-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T16:55:51.329 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:55:53.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c520eaa8-031f-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T16:55:53.941 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T16:55:56.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:55:56.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:55:56.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:55:56.435 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T16:55:56.437 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi098/crash 2024-04-25T16:55:56.438 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash -- . 2024-04-25T16:55:56.474 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T16:55:56.474 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-04-25T16:55:56.476 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi173/crash 2024-04-25T16:55:56.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash -- . 2024-04-25T16:55:56.520 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T16:55:56.520 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-04-25T16:55:56.522 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi203/crash 2024-04-25T16:55:56.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash -- . 2024-04-25T16:55:56.552 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T16:55:56.552 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-04-25T16:55:56.553 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T16:55:56.553 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T16:55:56.632 INFO:tasks.cephadm:Compressing logs... 2024-04-25T16:55:56.632 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T16:55:56.635 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T16:55:56.637 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T16:55:56.661 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T16:55:56.661 INFO:teuthology.orchestra.run.smithi203.stderr:: No such file or directory 2024-04-25T16:55:56.661 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log 2024-04-25T16:55:56.661 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T16:55:56.661 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T16:55:56.661 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T16:55:56.671 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T16:55:56.672 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T16:55:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T16:55:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-04-25T16:55:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T16:55:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log 2024-04-25T16:55:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T16:55:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T16:55:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T16:55:56.675 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T16:55:56.676 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T16:55:56.677 INFO:teuthology.orchestra.run.smithi098.stderr: 87.2% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T16:55:56.677 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-25T16:55:56.677 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 94.6% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T16:55:56.678 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T16:55:56.678 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-04-25T16:55:56.678 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T16:55:56.679 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.h.log: 90.5% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T16:55:56.680 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T16:55:56.680 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.10.log: 87.2% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T16:55:56.680 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T16:55:56.680 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T16:55:56.680 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T16:55:56.680 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.e.log: 93.3% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T16:55:56.680 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T16:55:56.686 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T16:55:56.689 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T16:55:56.696 INFO:teuthology.orchestra.run.smithi098.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-25T16:55:56.738 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.l.log: 83.3% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T16:55:56.747 INFO:teuthology.orchestra.run.smithi098.stderr: 83.4% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T16:55:56.817 INFO:teuthology.orchestra.run.smithi098.stderr: 91.1% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T16:55:56.853 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T16:55:56.855 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T16:55:56.916 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T16:55:57.138 INFO:teuthology.orchestra.run.smithi203.stderr: 90.5% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T16:55:57.140 INFO:teuthology.orchestra.run.smithi203.stderr: 93.4% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T16:55:57.376 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.k.log: 92.0% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T16:55:57.570 INFO:teuthology.orchestra.run.smithi203.stderr: 87.2% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T16:55:57.883 INFO:teuthology.orchestra.run.smithi098.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T16:55:57.889 INFO:teuthology.orchestra.run.smithi098.stderr: 94.6% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T16:55:58.568 INFO:teuthology.orchestra.run.smithi173.stderr: 94.1% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T16:55:58.691 INFO:teuthology.orchestra.run.smithi203.stderr: 91.9% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T16:56:00.203 INFO:teuthology.orchestra.run.smithi098.stderr: 90.4% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T16:56:04.744 INFO:teuthology.orchestra.run.smithi173.stderr: 93.8% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T16:56:06.305 INFO:teuthology.orchestra.run.smithi098.stderr: 95.2% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T16:56:16.091 INFO:teuthology.orchestra.run.smithi173.stderr: 91.9% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T16:56:17.187 INFO:teuthology.orchestra.run.smithi173.stderr: 92.0% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T16:56:23.547 INFO:teuthology.orchestra.run.smithi173.stderr: 91.7% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T16:56:31.235 INFO:teuthology.orchestra.run.smithi098.stderr: 90.4% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T16:56:33.269 INFO:teuthology.orchestra.run.smithi173.stderr: 91.6% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T16:56:40.007 INFO:teuthology.orchestra.run.smithi203.stderr: 93.9% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T16:56:42.379 INFO:teuthology.orchestra.run.smithi203.stderr: 94.3% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T16:56:48.224 INFO:teuthology.orchestra.run.smithi203.stderr: 91.5% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T16:56:49.213 INFO:teuthology.orchestra.run.smithi173.stderr: 94.4% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T16:56:51.508 INFO:teuthology.orchestra.run.smithi203.stderr: 91.9% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T16:56:58.019 INFO:teuthology.orchestra.run.smithi173.stderr: 94.0% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T16:56:58.021 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-04-25T16:56:58.022 INFO:teuthology.orchestra.run.smithi173.stderr:real 1m1.370s 2024-04-25T16:56:58.022 INFO:teuthology.orchestra.run.smithi173.stderr:user 3m40.153s 2024-04-25T16:56:58.022 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m6.722s 2024-04-25T16:57:26.360 INFO:teuthology.orchestra.run.smithi203.stderr: 91.7% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T16:57:30.155 INFO:teuthology.orchestra.run.smithi203.stderr: 94.2% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T16:57:55.423 INFO:teuthology.orchestra.run.smithi203.stderr: 91.5% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T16:59:43.415 INFO:teuthology.orchestra.run.smithi098.stderr: 91.8% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T16:59:44.084 INFO:teuthology.orchestra.run.smithi203.stderr: 95.1% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T16:59:44.086 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-25T16:59:44.087 INFO:teuthology.orchestra.run.smithi203.stderr:real 3m47.436s 2024-04-25T16:59:44.087 INFO:teuthology.orchestra.run.smithi203.stderr:user 3m49.424s 2024-04-25T16:59:44.087 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m8.650s 2024-04-25T17:00:02.592 INFO:teuthology.orchestra.run.smithi098.stderr: 91.8% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T17:00:15.509 INFO:teuthology.orchestra.run.smithi098.stderr: 91.5% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T17:00:43.238 INFO:teuthology.orchestra.run.smithi098.stderr: 95.0% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T17:01:28.597 INFO:teuthology.orchestra.run.smithi098.stderr: 91.6% -- replaced with /var/log/ceph/c520eaa8-031f-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T17:05:34.300 INFO:teuthology.orchestra.run.smithi098.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-04-25T17:05:34.302 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-04-25T17:05:34.302 INFO:teuthology.orchestra.run.smithi098.stderr:real 9m37.652s 2024-04-25T17:05:34.302 INFO:teuthology.orchestra.run.smithi098.stderr:user 5m45.770s 2024-04-25T17:05:34.303 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m16.132s 2024-04-25T17:05:34.303 INFO:tasks.cephadm:Archiving logs... 2024-04-25T17:05:34.304 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi098/log 2024-04-25T17:05:34.305 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T17:06:33.560 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi173/log 2024-04-25T17:06:33.561 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T17:07:02.847 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi203/log 2024-04-25T17:07:02.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T17:07:35.559 INFO:tasks.cephadm:Removing cluster... 2024-04-25T17:07:35.560 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c520eaa8-031f-11ef-bc93-c7b262605968 --force 2024-04-25T17:07:36.165 INFO:teuthology.orchestra.run.smithi098.stdout:Deleting cluster with fsid: c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T17:07:36.500 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c520eaa8-031f-11ef-bc93-c7b262605968 --force 2024-04-25T17:07:36.693 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T17:07:37.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c520eaa8-031f-11ef-bc93-c7b262605968 --force 2024-04-25T17:07:37.400 INFO:teuthology.orchestra.run.smithi203.stdout:Deleting cluster with fsid: c520eaa8-031f-11ef-bc93-c7b262605968 2024-04-25T17:07:37.998 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T17:07:37.998 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T17:07:38.017 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T17:07:38.034 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T17:07:38.050 INFO:tasks.cephadm:Teardown complete 2024-04-25T17:07:38.050 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T17:07:38.062 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: 2024-04-25T17:07:38.064 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T17:07:38.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T17:07:38.067 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T17:07:38.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T17:07:38.188 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T17:07:38.198 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T17:07:38.198 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:07:38.201 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:07:38.204 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T17:07:38.214 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:07:38.217 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:07:38.217 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-25T17:07:38.221 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:07:38.221 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-04-25T17:07:38.221 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +840us[ +836us] +/- 50ms 2024-04-25T17:07:38.221 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 -1271us[-1271us] +/- 56ms 2024-04-25T17:07:38.221 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +516us[ +516us] +/- 74ms 2024-04-25T17:07:38.221 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:07:38.227 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:07:38.227 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-25T17:07:38.228 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +820us[ +867us] +/- 51ms 2024-04-25T17:07:38.228 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 3 -1296us[-1249us] +/- 57ms 2024-04-25T17:07:38.228 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +489us[ +525us] +/- 73ms 2024-04-25T17:07:38.228 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:07:38.253 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T17:07:38.253 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-04-25T17:07:38.253 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +806us[ +787us] +/- 51ms 2024-04-25T17:07:38.253 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -1309us[-1309us] +/- 57ms 2024-04-25T17:07:38.253 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +512us[ +512us] +/- 74ms 2024-04-25T17:07:38.253 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T17:07:38.254 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T17:07:38.266 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T17:07:38.267 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T17:07:38.305 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T17:07:38.437 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T17:07:38.506 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T17:07:38.603 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:07:38.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:07:38.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:07:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:07:38.661 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T17:07:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:07:38.688 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T17:07:38.698 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T17:07:38.745 INFO:teuthology.task.internal:Duration was 3129.431771 seconds 2024-04-25T17:07:38.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T17:07:38.779 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T17:07:38.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T17:07:38.782 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T17:07:38.785 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T17:07:38.823 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:07:38.835 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:07:38.848 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T17:07:39.307 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T17:07:39.307 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-04-25T17:07:39.308 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T17:07:39.332 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-04-25T17:07:39.332 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T17:07:39.370 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-04-25T17:07:39.370 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T17:07:39.391 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T17:07:39.391 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T17:07:39.394 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T17:07:39.413 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T17:07:39.457 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T17:07:39.457 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T17:07:39.461 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T17:07:39.486 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T17:07:39.869 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T17:07:40.307 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T17:07:40.307 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T17:07:40.347 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T17:07:40.373 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T17:07:40.414 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T17:07:40.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:07:40.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:07:40.431 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:07:40.452 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-04-25T17:07:40.452 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-04-25T17:07:40.485 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-04-25T17:07:40.525 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:07:40.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:07:40.539 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:07:40.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:07:40.553 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T17:07:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T17:07:40.579 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T17:07:40.590 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T17:07:40.592 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi098 2024-04-25T17:07:40.592 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T17:07:40.633 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi173 2024-04-25T17:07:40.634 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T17:07:40.669 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254/remote/smithi203 2024-04-25T17:07:40.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T17:07:40.705 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T17:07:40.705 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T17:07:40.708 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T17:07:40.710 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T17:07:40.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T17:07:40.765 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T17:07:40.766 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T17:07:40.794 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T17:07:40.795 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T17:07:40.797 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T17:07:40.799 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T17:07:40.808 INFO:teuthology.orchestra.run.smithi098.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 17:07 /home/ubuntu/cephtest 2024-04-25T17:07:40.811 INFO:teuthology.orchestra.run.smithi173.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 17:07 /home/ubuntu/cephtest 2024-04-25T17:07:40.813 INFO:teuthology.orchestra.run.smithi203.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 17:07 /home/ubuntu/cephtest 2024-04-25T17:07:40.825 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T17:07:40.838 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T17:07:40.862 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during scrub thrashing: 2024-04-25T17:07:40.863 INFO:teuthology.nuke:Checking targets against current locks 2024-04-25T17:07:40.887 DEBUG:teuthology.nuke:shortname: smithi098 2024-04-25T17:07:40.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T17:07:40.913 DEBUG:teuthology.nuke:shortname: smithi173 2024-04-25T17:07:40.913 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T17:07:40.918 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:09:53.127783', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAbpKmuJ8XytYJkBUjL8NiCFnnUjKlfTN87sG+UVhfJQCodHf6YHJ8hBC6Us584YzPhINaGSRUeZ8h3mB+o038Y='} 2024-04-25T17:07:40.936 DEBUG:teuthology.nuke:shortname: smithi203 2024-04-25T17:07:40.936 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T17:07:40.943 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:09:53.129061', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFLgQk0metQO+1/T7DgZU4TXXsmE+JGSAbvvJUAGX4M8t0fK7Tr5YY7HBPydPNUMbozvUCQhOEhbzEnlvBq+rg4='} 2024-04-25T17:07:40.961 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673254', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 16:09:53.130185', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVxOtrt9fVPNBp+Qd2NuaBQF7bnlNYrx4MqFBf+vottMr/B4o54q4upy4rRMciDt7RiQtEr6tXHjcsFuBpQFfM='} 2024-04-25T17:07:40.973 INFO:teuthology.orchestra.console.smithi098:Power off 2024-04-25T17:07:40.973 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T17:07:40.998 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-04-25T17:07:40.998 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:41.014 INFO:teuthology.orchestra.console.smithi173:Power off 2024-04-25T17:07:41.015 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T17:07:41.030 INFO:teuthology.orchestra.console.smithi203:Power off 2024-04-25T17:07:41.031 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-25T17:07:41.042 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-04-25T17:07:41.042 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:41.051 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-25T17:07:41.053 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-04-25T17:07:41.053 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:41.067 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-25T17:07:41.076 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-25T17:07:45.053 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:45.068 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:45.076 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:45.163 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-25T17:07:45.177 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-25T17:07:45.185 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-25T17:07:49.168 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:49.178 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:49.187 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:49.278 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-04-25T17:07:49.288 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-04-25T17:07:49.297 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-04-25T17:07:49.397 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-04-25T17:07:53.283 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:53.292 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-25T17:07:53.395 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-04-25T17:07:53.402 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-04-25T17:07:53.496 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-04-25T17:07:53.503 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-04-25T17:07:53.609 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} duration: 3129.4317705631256 failure_reason: 'Command failed (workunit test kernel_untar_build.sh) on smithi098 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=25fc6b198dc14be39cf1e4a724142f81 status: fail success: false 2024-04-25T17:07:53.610 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T17:07:53.692 INFO:teuthology.run:FAIL